jamesward
commented
2 years ago I'd love this for a project I'm working on, but I realize the Jline replacement piece would be tough. Maybe I can find some cycles to help. :)
Closed bitspittle closed 1 year ago
jamesward
commented
2 years ago I'd love this for a project I'm working on, but I realize the Jline replacement piece would be tough. Maybe I can find some cycles to help. :)
bitspittle
commented
2 years ago If you want to explore this any time, let me know and I'll give you all the cycles I can! Happy to meet first at some point and tell you all known pitfalls.
On Mon, Jul 18, 2022, 4:24 PM James Ward @.***> wrote:
I'd love this for a project I'm working on, but I realize the Jline replacement piece would be tough. Maybe I can find some cycles to help. :)
— Reply to this email directly, view it on GitHub https://github.com/varabyte/kotter/issues/63#issuecomment-1188430583, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKNONASRH6MG4RRLVA3PV4LVUXRSHANCNFSM5FMJECLA . You are receiving this because you authored the thread.Message ID: @.***>
rohanprabhu
commented
1 year ago We are working on building a CLI application and this library looks like something that would be perfect for our use case. The only issue is that our CLI has to have native targets, and hence kotlin-native support is imperative. Talking about the JLine functionality, is there an interface (or interfaces) that abstracts out the JLine functionality - such that a native implementation of the same would suffice for a kotlin-native target? This is something I'd be very interested to pick up depending on what complexity we are looking at.
bitspittle
commented
1 year ago Hey! Thanks for your interest in Kotter!
I'd be happy to set up some time to meet, to share with you over video chat the current situation and perhaps accelerate any quick experiments you want to try. I'll send a message to your github account with details after writing this.
To answer your question, Kotter has a Terminal interface which is pretty minimal, and it sounds like what you're looking for. If you can implement it effectively for native, then you won't need JLine. Check out TestTerminal for a in-memory terminal I created for tests, and the SystemTerminal implementation which is the only part of the code that references JLine.
Note that there will still be some difficulties ahead, particularly threading. I am not too familiar with how threading works in Kotlin/native, because as far as I can tell the docs kind of suck. Note here how they bounce you to the "modern memory management" docs but those don't talk about threading at all.
There are a few places where I kick off threads (usually via coroutines, but in one case from an executor).
Terminal interface exposes a Flow of raw ANSI bytes that are fired when keys are pressed. Those need to be translated into the Keys instances exposed by Kotter. This thread handles the flow collection and mapping.You'll want to make sure you understand all those cases. Getting this to work for native may be a trivial case of adding actual / expect or it may be a show stopper due to my incompatible design, all based on how native threads work. I'm simply not sure, and that's the first place I would experiment with.
Beyond that, you should clone the project and do a search for import java under kotter/kotter/src/main. That should show you some of the trouble you might be getting into :) I should have probably used kotlin Duration instead of java Duration, but it was totally experimental when I started writing Kotter, and now the API exposes java Durations in a bunch of places.
Otherwise, in theory you just need to move the following packages from kotter/kotter/src/main to commonMain:
com.varabyte
kotter
foundation.*
runtime.*
kotterx.*and then move com.varabyte.kotter.terminal implementations to jvmMain. (And you'd create your new native terminal implementation in the native main section that you create).
At this point, brace yourself for a bunch of compile errors :)
Finally, here are some alternate solutions to make sure you've considered other options:
bitspittle
commented
1 year ago I heard that Mosaic went multiplatform recently. If I can see how they did it and it doesn't seem that hard, I'll give a shot porting Kotter to multiplatform.
At one point, I did some experiments with GraalVM which seemed promising at first, and I really wanted to make that my recommended solution, but soon I got a class not found exception from the kotlin standard library (kotlin.random.Random wasn't found??!), and I didn't enjoy the runaround to try to figure out how to fix it. I found the docs for GraalVM occasionally as inscrutable as I sometimes find Gradle docs to be.
I also found a bug reported against the GraalVM Gradle plugin for someone who had a similar exception to what I have, including they did some leg work where they reported the exact line that needed to get fixed, and after that -- nothing. No response or change from the GraalVM team.
Anyway, since even my initial impressions with GraalVM were frustration, I think multiplatform (if I can get it to work) will be a much easier thing for me to recommend to kotter users.
bitspittle
commented
1 year ago First hurdle - Kotter makes use of reentrant locks. It seems like this is not officially supported in Kotlin/Native. There's atomicfu which provides reentrant locks but says that it's a heavily experimental feature and that library authors should not use them.
I may be able to reimplement what I need using expect / actual but this could get ugly...
bitspittle
commented
1 year ago Initial prototype success:
Final debug exe size: 7.6MB, release exe size: 1.4MB. That's not bad....
Still, best to temper any excitement with the following points:
1) I am implementing my own ReentrantLock and ReentrantReadWriteLock classes at this point. This scares the hell out of me. I am NOT a concurrency expert. It's possible my code will break on a really hard to reproduce edge cases later, or it will work just fine but might have relatively poor performance characteristics.
2) I am not supporting reading yet. Printing text is easy. Reading characters in a platform independent way may or may not be easy. I need to put the terminal into raw mode (so keys aren't repeated when the user types them) and I have no idea how to do that in Kotlin/Native at this point.
3) I have not found out how to handle CTRL-C yet. There's a Kotlin native method for registering a callback but it complains because I'm passing in a lambda to it, and now amount of inlining has made things happy. I'm not sure what the proper solution is yet or if there even is one.
4) I have not tested Windows yet.
5) I have not tested most of the samples yet.
6) While many user programs might not be affected, there are a handful of APIs now that are not currently backwards compatible. I might be able to smooth these over but I'm not sure.
Got farther than I expected to today, but I'm still not sure yet I'm sold on the extra maintenance burden / complexity this change might bring. If I could get GraalVM working consistently instead, that would be so tempting...
bitspittle
commented
1 year ago In case anyone is still around who is interested in helping me test this in the next few days, please ping me or give a thumbs up to this comment. It would really help the library to get other eyes on this besides mine.
Lots of improvements today. Kotter native is on track.
I am feeling a little better about my reentrant lock code, which I may still come to regret someday. But at least it seems like it's holding up well to a barrage of tests.
Reading user input is now supported :tada: See video above, which runs the examples/input example using native.
CTRL-C is now handled.
Windows still not tested. Will test in the next few days.
Most samples still not tested yet. Will run a bunch of them in the next few days.
I'm more and more OK with breaking backwards compatibility, as the migration steps are trivial. (In most cases, users will just convert lines line Duration.ofMillis(100) to 100.milliseconds, or reimport extension methods that now live in a new place).
In summary, still to do...
bitspittle
commented
1 year ago Kotlin/Native looking solid on Linux at this point.
Main thing left at this point is figuring out Windows (which it seems will need its own Terminal implementation, and then publishing artifacts on Maven.
It's my top priority for the next few days, hopefully no major snags.
jamesward
commented
1 year ago Loving this progress! Thank you @bitspittle :)
bitspittle
commented
1 year ago Thank you @jamesward!
It's a testament to Kotlin/Native how smoothly this has been going. I was sure I was going to run into those memory freeze exceptions I heard so much about, or other problems that would crop up since I didn't really consider Kotlin/Native when I first built Kotter. But the Kotlin standard library is pretty impressive at this point. And it wasn't long before I had some samples running locally in Kotlin/Native.
My only real regret with Kotter was using Java durations instead of Kotlin durations. I believe Kotlin durations were still experimental when I started writing Kotter, but still -- I should have been more confident that at that point it was probably safe to use them.
The documentation on JB's side for Kotlin/Native is a little meh (JB has a style where they dump a bunch of information on you all at once, much of it intimidating and unrelated, where what I think most people are looking for is a step-by-step book or codelab that introduces concepts in layers).
But once I got going, it was pretty neat to write C-style code with the magic of memScoped blocks. The team did some fancy stuff with Kotlin with their native APIs, which is crazy seeing how few people probably will ever truly use or appreciate them.
Anyway, let's see how Windows goes today...
bitspittle
commented
1 year ago Easy part is done (the text example just prints, it's totally not interactive):
Tomorrow will be focused on getting interactive mode / input working. Hopefully I can find a good article about modifying the terminal from C in Windows.
bitspittle
commented
1 year ago :relieved:
Aiming to drop 1.1.0-rc1 on Friday. We'll see how that goes.
P.S. Dealing with the Windows API was a PITA :) It didn't help that I was working on a slow ass virtual machine. Anyway, hopefully I can put this dark memory behind me...
bitspittle
commented
1 year ago Good news -- I have a multiplatform project with a simpler setup than Kotter (https://github.com/varabyte/truthish for the curious) which I used as a test run for figuring out publishing Kotlin / Native artifacts on maven central. Took a bit longer than I expected to set up, but as of this morning, it's working.
It shouldn't be too complicated to carry that logic over to Kotter.
At this point, Kotlin / Native support for Kotter is close. New ETA: should be up within a few days, maybe even tomorrow if things go smoothly.
The scariest thing that could hit me, I think, is a subtle race condition showing up in tests.
The final checklist:
bitspittle
commented
1 year ago Alright folks! If anyone is eager to try Kotter+Multiplatform out right now, you can grab some dev snapshots I just published.
:detective: With multiplatform increasing the surface of things that can go wrong, I can really use as many eyes as I can get on this! Especially if you use Windows, Mac Intel, and/or Mac M1. Non US-keyboards on Windows a plus. THANK YOU! :detective:
First, you'll need to declare the following repository:
repositories {
maven("https://s01.oss.sonatype.org/content/repositories/snapshots/")
}Then, declare your dependency:
JVM project
dependencies {
// IMPORTANT! This *used* to be "com.varabyte.kotter:kotter:..."
implementation("com.varabyte.kotter:kotter-jvm:1.1.0-SNAPSHOT")
}Multiplatform project
kotlin {
// declare targets, e.g. linuxX64(), mingwX64()...
sourceSets {
val commonMain by getting {
dependencies {
implementation("com.varabyte.kotter:kotter:1.1.0-SNAPSHOT")
}
}
}
}If you've never set up a Kotlin/Native project before, please see the official docs.
You can also confer with the Kotter native example.
Migration from Kotter 1.0.x
Duration -> Kotlin Duration
Duration.ofMillis(250) -> 250.millisecondsDuration.ofSeconds(5) -> 5.secondsduration.toMillis() -> duration.inWholeMillisecondsjava.net.URI -> com.varabyte.kotter.platform.net.Uri
java.util.concurrent.locks.ReentrantReadWriteLock.write -> com.varabyte.kotter.platform.concurrent.locks.ReentrantReadWriteLock.write
I still have a fair bit of busy work to do, so a stable release may take a few more days. But, at this point, the snapshots should be a really solid representation of the final release. I'm expecting mostly docs and bug fixes from here on out.
Questions? Feedback? If I'm awake, you can usually catch me on my Discord server. My timezone is US Pacific Time (UTC-8)
bitspittle
commented
1 year ago I'm really hoping I can turn this as is into 1.1.0 in a week or two. However, I'm nervous because I'm just one guy. Despite all my pounding on it, it's easy to feel paranoid about missing something.
If anyone in here wouldn't mind spending a few minutes even just trying to set up a quick dummy project, any confirmation that this is working for others would help me immensely.
Thank you all very much for your interest in Kotter. Getting Kotter+Native to work was an interesting journey. I hope the feature ultimately helps even one person get a Kotter app out when they might not have otherwise.
bitspittle
commented
1 year ago Has anyone had a chance to try out Kotter 1.1.0-rc1?
I've been using it myself non-stop and haven't had any issues, so I may just put out 1.1.0 in the next day or two.
However, if anyone seeing this message could commit giving it a try, I'd be happy to wait a few more days, to make sure I don't send out a 1.1.0 that's fundamentally busted in some way I'm just not running into in my own environment.
jamesward
commented
1 year ago This is awesome! I'm giving it a try and hitting a strange runtime exception. Code:
fun main() = session {
val rpc = BarsRPC(Config.barsUrl)
var loaded by liveVarOf(false)
var bars by liveVarOf<List<Bar>>(emptyList())
run {
section {
textLine("Connecting to: ${Config.barsUrl}")
if (loaded) {
textLine("loaded")
if (bars.isEmpty()) {
textLine("No Bars")
} else {
textLine("Bars:")
bars.forEach {
textLine(" ${it.name}")
}
}
}
}.runUntilKeyPressed(Keys.ESC) {
bars = rpc.fetchBars()
loaded = true
}
}
}Getting:
Connecting to: https://kotlinbars.jamesward.com/api/bars
Uncaught Kotlin exception: kotlin.ArithmeticException
at 0 tui.kexe 0x441a66 kfun:kotlin.Throwable#<init>(){} + 86
at 1 tui.kexe 0x43af7c kfun:kotlin.Exception#<init>(){} + 76
at 2 tui.kexe 0x43b16c kfun:kotlin.RuntimeException#<init>(){} + 76
at 3 tui.kexe 0x43bd2c kfun:kotlin.ArithmeticException#<init>(){} + 76
at 4 tui.kexe 0x46e134 ThrowArithmeticException + 116
at 5 tui.kexe 0x8e4ca0 kfun:com.varabyte.kotter.runtime.internal.text#numLines__at__kotlin.collections.List<com.varabyte.kotter.runtime.internal.TerminalCommand>(kotlin.Int){}kotlin.Int + 640
at 6 tui.kexe 0x8c90f7 kfun:com.varabyte.kotter.runtime.Section.$renderOnceAsync$lambda$3COROUTINE$17.invokeSuspend#internal + 2471
at 7 tui.kexe 0x4457e4 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916
at 8 tui.kexe 0x7009e3 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3571
at 9 tui.kexe 0x72a9e3 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$workerRunLoop$lambda$1COROUTINE$225.invokeSuspend#internal + 1027
at 10 tui.kexe 0x4457e4 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916
at 11 tui.kexe 0x7009e3 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3571
at 12 tui.kexe 0x687ab1 kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long + 1569
at 13 tui.kexe 0x725244 kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 660
at 14 tui.kexe 0x72490d kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 2125
at 15 tui.kexe 0x724b66 kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 358
at 16 tui.kexe 0x72a09a kfun:kotlinx.coroutines.MultiWorkerDispatcher.workerRunLoop#internal + 186
at 17 tui.kexe 0x72a49e kfun:kotlinx.coroutines.MultiWorkerDispatcher.<init>$lambda$0#internal + 62
at 18 tui.kexe 0x72b262 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$0$FUNCTION_REFERENCE$72.invoke#internal + 66
at 19 tui.kexe 0x72b342 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$0$FUNCTION_REFERENCE$72.$<bridge-UNN>invoke(){}#internal + 66
at 20 tui.kexe 0x454b8a WorkerLaunchpad + 170
at 21 tui.kexe 0x5cbe30 _ZN6Worker19processQueueElementEb + 4624
at 22 tui.kexe 0x5cab5e _ZN12_GLOBAL__N_113workerRoutineEPv + 142
at 23 libc.so.6 0x7f467f294fd3 0x0 + 139940757852115
at 24 libc.so.6 0x7f467f31566b 0x0 + 139940758378091 Any ideas?
Note that I'll add input prompt into this loop.
bitspittle
commented
1 year ago @jamesward Moved this specific issue to #98. Let's move the discussion over there and squash this!
bitspittle
commented
1 year ago OK folks, Kotter 1.1.0 is out. It's always a little terrifying releasing something to mavenCentral.
If you try this and find any problems with Kotlin/Native, please open a new issue. Thanks!!
Context: https://www.reddit.com/r/Kotlin/comments/q17zjq/introducing_konsole_a_kotlinidiomatic_library_for/hfggbsv/
Currently, this library is JVM only. Adding native support would allow users to use this library in a Kotlin native context, which would:
Current challenges:
This bug isn't a priority for me at the moment because it's not clear to me how many people actually want this support, but hey, if you do, please indicate so by adding a thumbs up reaction to it (and consider leaving a comment on why you don't want to just use the JVM, I'd be curious!)