Serializability of coroutine classes

[elizarov]

Currently, if you attempt to serialize coroutine state via standard Java Serialization the tree of reference leads you to the objects defined in kotlinx.coroutines like StandaloneCoroutine (which serves as a completion for launched coroutines) and its contexts (like CommonPool) which are not currently serializable. They should be made properly serializable.

See also https://github.com/Kotlin/kotlin-coroutines/issues/28

[fvasco]

Serializable object should work out of the box, like

enum class CommonPool { INSTANCE }

but I suspect that this is compiler language issue.

[elizarov]

It does, but, unfortunately, making it enum pollutes its namespace with methods like valueOf and values which we don't need here. The correct solution is to provide private fun readResolve. Ideally, this should be done by Kotlin compiler, but it is not there yet: https://youtrack.jetbrains.com/issue/KT-14528

[elizarov]

Note, that you can easily serialize coroutine state with 3rd party serialization framework like Kryo. It is just that standard JVM serialization does not currently work due to objects that do not implement Serializable.

[Restioson]

+1. I was able to achieve serialization (with the help of @elizarov, thank you so much), but only with the use of reflection, to change the value of result to COROUTINE_SUSPENDED

[wdroste]

+1

@elizarov totally understand the concept and saw the great example for ES6 generators but is there a sample project that can demonstrate the serialization that we can build upon.. so we make sure we're doing it correctly seems pretty deep.

[elizarov]

@wdroste What is your use-case for serialization? Can you explain it here, please.

[wdroste]

We have a IoT type server application w/ millions of clients. These clients are low powered in most cases and the protocol they're using requires many HTTP based request/responses to a per enterprise customer customizable business logic (scriptable workflow). Now that process can take anywhere from 30 secs to 1 min for an entire HTTP Session.. that process is mostly waiting on I/O from the clients, the actual server processing is only a 1-2 secs total. The memory required to hold the session state is rather large so we like to make a trade off. Incur the CPU cost of serialization and server side I/O persistence so we can free up some memory to continue processing.

Basically the issue is the JVM spends a lot of time looking for memory to free that it can't because the session/workflow state must be maintained, if we were able to persist that state it could free the memory and use it for the other requests, thereby increasing throughput and as a side effect we could be stateless as well since any server could load the current state and continue the workflow making loading balancing much easier.

Our workflows are python scripts built as generators such that we can provide an synchronous programing model to an asynchronous protocol, the issue is python does not support pickling/serialization of the generator. We're looking to move to something that does.

I mentioned ES6 generator because that's the style python uses and we require to convert our workflows to another language. The 'yield' statement must return an object as well as provide one. I would prefer Kotlin excellent IDE support, type safety, and compiler based null checks, but there's others that would recommend we just do this in Javascript Rhino since there's support for both 'yield' and serialization of coroutines.

[Restioson]

I wrote a Kotlin test of it using Kryo with @elizarov's help - https://gist.github.com/Restioson/fb5b92e16eaff3d9267024282cf1ed72 . The only issue is that it uses Kryo, a Java library (and a bit of java reflection which I'm sure can be converted to Kotlin). Theoretically it would work if you subbed that out with another serialization library able to serialize private fields, I imagine, or if you did it yourself with reflection.

[wdroste]

@Restioson appreciate it, I will take a look.

[pdvrieze]

I'd like to add an additional use case. This is in case of Android. In Android in particular there are cases that require repeated invocations of dialogs or independent activities. The coding of this is very suited for coroutines and it works "well" if you ignore the fact that android activities can be closed (and saved to disk) at any time. Some form of serialization would solve this problem. as a continuation can then just be stored allowing safe resumption of state.

[pdvrieze]

I've managed to make something work on Android (including serializing across activity recreation). There is one giant hack though. In a coroutine it is deceptively easy to capture an activity. This is not valid. The hack will actually serialize Service, Application and Activity (subtypes) as simple enum constants. On deserialization a context is passed which is used in it's place (with a cast to the "type").

The Kryo part lives in KryoIO.kt. It's use for android (including some stuff around wrapping startActivityForResult is in CoroutineActivity.kt

[wdroste]

I've create a project to illustrate my statements above. The ES6 generator part works great, however I'm unable to serialize the coroutine. If we can serialize the workflow we can use it for long running processes.

@elizarov could really use some help w/ this. @Restioson tried to follow your gist but it doesn't seem to work in 1.2.10 https://github.com/wdroste/kotlin-generator-serializer

[unicomp21]

Looking at aws step functions, this would be so much nicer.

[pdvrieze]

@wdroste There are some really nasty bits of internals to take care of. I've managed to make it work (https://github.com/pdvrieze/android-coroutines/tree/master/core/src/main/java/nl/adaptivity/android/kryo) although this is not for the latest coroutines library (and it has internals).

One of the problems is that the coroutines library uses sentinel objects that are just objects (rather than enum instances - for the experimental version. Enums are used in 1.3 nonexperimental) and as such don't handle serialization properly. If course there are also other issues.

[mrussek]

Any update on this? I think this would be super useful!

[elizarov]

@mrussek What's your use-case for this?

[joost-de-vries]

I think it would be very useful to implement long running business processes. The tools that are available for this are usually proprietary using some incomplete home brew expression language and flow control. And almost all of them use proprietary storage and run platform. As a result you can't use proper version control, proper debugging, proper module reuse, proper cloud deployment etc. So I being able to use coroutines to implement this would be incredibly exciting.

[elizarov]

You can do it right now. It takes so hardship to setup, but I don't see how we could make it any more simple. If you are to serialize coroutine it automtically means you have to abide by certain restrictions in your code.

[joost-de-vries]

I can understand that. Tx. Can you give some pointers? I'd like to make a small playground to see what's involved.

[elizarov]

Read the discussion in this issue. There are links to a bunch of working examples.

[pdvrieze]

@joost-de-vries You may want to look at business process managements systems for this. For now, that is your best bet (although indeed it doesn't support debugging across the process). I'm not sure that coroutines can actually fully do what you want though. A key design criterion is still in ACID properties of the steps.

Btw. serialization would probably be a lot easier if it was possible to have the compiler validate captures (or the lack of captures).

[mrussek]

Yeah, my work is currently looking at Uber's cadence for something like this, although their approach seems a bit more hacky imo.

[pdvrieze]

@mrussek I had a quick look at uber cadence. It seems a sensible system although I'm not clear why you would use it over a bpmn2 based system. In any case, key in those systems is that they are essentially distributed systems where the workflows system doesn't do the work. It may be interesting to use workflows in a single-process system, but this is something that is not quite ready for bleeding edge yet. I have a system that should be able to do it reasonably easily as I already have the testing of workflows implemented without actual actions attached. Linking them with a lambda for the behaviour (rather than workflow messaging) should be reasonably straightforward.

[Miha-x64]

One more use-case: Telegram bots. For each user, a bot stores a state of the dialogue, which is a finite state machine easy to express with a coroutine, e. g.

suspend fun talkTo(user: User, firstMessage: Message) {
    when (firstMessage.text) {
        "Buy some donuts" -> sellDonutsTo(user)
        …
    }
}
private suspend fun sellDonutsTo(user: User) {
    sendMessage(user, "How many?")
    val number = receiveNumberFrom(user, onError = "Why don't you send me a whole positive number?")
    …
}

Currently this can be done with sealed classes which requires much more patience.

[philipguin]

Another use-case: support for scripting via coroutines in a game engine that needs to serialize the scripts on save/load, or heaven forbid, in a "rollback" networking system, where suspended scripts are saved each frame and swapped between depending on network-delayed inputs from other clients.

Save/load also encompasses "zoning", where for instance, as the player runs around the game world, parts of the world are loaded and unloaded dynamically. This would include NPCs with "wander" scripts, enemies with "scan then attack" scripts, etc.

[caeus]

Another use-case: To have code describe human processes. Such as making a delivery, hiring people or any bureaucratic process a company may have implemented internally. Orchestration could be done by a coroutine. Once it needs a task from another server, a human, or anything that cannot be done in the same runtime, it's suspended, and persisted in a DB. Once the task has been completed, coroutine is loaded from DB and resumed with the value of the task.

[Miha-x64]

As an author of an external serialization tool, now I wonder how this is going to be implemented. Will this require kotlinx.serialization? How much stable serialized representation will be?

[elizarov]

@Miha-x64 This particular issue is about making it serializable with Java Serialization. However, there's a parallel thought process on how we might be able, in the future, support kotlinx.serialization for coroutine state, too.

[mrussek]

For anyone that is curious, I am working on a prototype implementation that can serialize the continuation state to JSON. It is a little hacky, but it seems like it may be a good starting point for further conversations.

[Miha-x64]

Isn't it “just works” with reflection, e.g. Gson or Moshi?

[mrussek]

My implementation uses jackson. I haven't tried with Gson or Moshi, but part of what makes it hacky is the fields in the continuation object are declared private and there is no way to access them other than via reflection. I'm not sure if Gson or Moshi do this automatically.

[pdvrieze]

I've done it based on kryo. It works, but is is far from straighforward. And in reality many coroutines capture all kinds of references to surrounding objects (that shouldn't be serialized - in my case Android Activities). There are ways to "catch" this and work around it, but it's not great.

What we could really do with is some way to require a lambda/suspending lambda not to be capturing. That would make serializability much more feasible. In some ways I'd just want a single token to pick up the process where it left off.

[elizarov]

@pdvrieze I'm thinking along the lines that if we go with kotlinx.serialzation support in the future, then we'll be explicitly annotating suspending functions and lambda as @Serializable and the compiler will guarantee that they only capture and work with serializable classes.

[pdvrieze]

@elizarov That sounds like a good solution. Of course having an annotation to prevent capturing may still be worthwhile (especially in the Android case - it is very very easy to create leaks by capturing the wrong things - Making the lambda an extension on Nothing can help a bit ;-)).

[Miha-x64]

But we typically need the host Activity of Fragment. For example, in callback-based APIs, I pass functions which require receiver to be Screen (an abstraction on top of Fragment): pureParcelFunction3(RootScreen::photoTaken).

[elizarov]

But we typically need the host Activity of Fragment. For example, in callback-based APIs, I pass functions which require receiver to be Screen (an abstraction on top of Fragment): pureParcelFunction3(RootScreen::photoTaken).

@Miha-x64 That's a larger serialization-related question. What if I serialize some state, but it also contains a reference to some "context" that I don't want to be serialized, but property replaced with a new context during deserialization. We don't have an answer to that yet, but this falls under the serialization design anyway, not coroutines design.

[mrussek]

Agreed. FWIW, I think that in some circumstances there could be support at the application framework level. Not sure what you would do for Android, but in my prototype I had a hack that serialized all Spring beans in the context using the Spring bean identifier. This allowed several Spring application servers to cooperate on executing a coroutine.

[aa0ndrey]

I think there are a lot of problems to serialize coroutine state. But for long running process/activity it could be enough to have ability to serialize state of iterators created by sequence-scope https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.sequences/-sequence-scope/ Now such iterators are created by coroutine and we stuck in the same problem: "how to serialize coroutine". But if kotlin dev team could change implementation for the sequence on something like C# do https://csharpindepth.com/Articles/IteratorBlockImplementation it would be awesome! We could wrap any continuation logic inside such serializable iterator to persist and restore it. And we could use them inside coroutines without any upgrading for coroutines.

[jgreenyer]

I've posted another use case where the possibility to serialize/clone coroutines or whole coroutine contexts would be helpful: https://discuss.kotlinlang.org/t/clone-coroutines/20009/2

Are there any more recent developments in this direction?

[qwwdfsad]

@jgreenyer nice use-case! There is no new development, unfortunately, this is out of our scope right now.

Alternatively, you can use Kryo (there is a snippet in the discussion with how to use that) or clone not the coroutines, but their state which can be expressed as a regular Kotlin [data] class

[mandolaerik]

Another use-case is computer architecture simulators. This use case is in many ways similar to @philipguin 's game engine use case; a simulation is a set of things that interact deterministically in a controlled way. An important feature of simulators like Simics is to accurately save/load simulation state. Furthermore, coroutines are often a useful way to model data communication between devices; many models written in the SystemC framework are implemented in terms of (stackful) coroutines, which makes serialization very hard. Few simulators are interoperable with JVM, so this use case brings LLVM compatibility as a requirement.

[holgerbrandl]

Same here using the latest and greatest of kryo, kotlin and coroutines. See https://github.com/holgerbrandl/kryo-kotlin-sam/blob/master/src/main/kotlin/simpleproc/SimpleProc.kt for the complete example including gradle-project context for reproducibility.

My use case would also point towards simulation and is described in https://github.com/holgerbrandl/kalasim/issues/19

I've carefully reviewed all materials from above, but so far did not find any way to use the continuation after deserialization. All I get is an NPE (included in the source pointer from above). Interestingly, the example from above allows to deserialize the object but fails later when consuming the sequence iterator again.

Any help/point here to overcome/workaround this issue would be greatly appreciated.

[pdvrieze]

@holgerbrandl It is possible to do it, but it requires quite some configuration of Kryo. For the android use case you can have a look at: https://github.com/pdvrieze/android-coroutines (it should work although it was more of an experimental piece of software).

[Miha-x64]

By the way, when designing serializability, it should be quite easy to implement cloneability, too. This is especially useful for people from Arrow which are used to rewind an existing coroutine to the previous state.

[pdvrieze]

@Miha-x64 Serialization can be used for cloning - either through a format or "directly" (and for equals and hashcode).

[Miha-x64]

@pdvrieze, sure. But “direct” cloneability is way easier to design, and it can satisfy some use-cases.

[pdvrieze]

@Miha-x64 Actually, direct cloning is not that easy due to control inversion (both serialization and deserialization are driven from the serializer side, so the format cannot easily shuffle between the two. You'd have to use (actual) threads, or use a buffer.

[holgerbrandl]

Thanks for your feedback and the codepointer @pdvrieze . I'll analyze it to see if I can adapt it for my use case.

[holgerbrandl]

@pdvrieze I've tried to work through your mentioned android-coroutines repo, extracted the non-android bits into https://github.com/holgerbrandl/kryo-kotlin-sam/blob/master/src/main/kotlin/kryo (to work out a more minimalistic POC example) and prepared an example along with it in https://github.com/holgerbrandl/kryo-kotlin-sam/blob/master/src/main/kotlin/kryo/SeqBuilderSerializationExample.kt.

When doing so, I had lifted coroutines, kotlin and kryo dependencies to the current versions.

Unfortunately, the example still throws an NPE when continuing the sequence. I suspect https://github.com/holgerbrandl/kryo-kotlin-sam/blob/eb3dd4ab23e6dd036fde095e6e3065202ac6d995/src/main/kotlin/kryo/AndroidKotlinResolver.kt#L42-L45 to to refer to no longer valid/used classes. Also CommonPool which is somehow used to configure the Kryo instance seems gone. But these aspects may not even relate to the root cause, which is still in the dark for me.

I know it is asking a lot, but is there any chance you could provide me with more guidance to make this - i.e. sequence-builder persistence & continuation using kryo - work?

[pdvrieze]

@holgerbrandl I didn't look at it, but I suspect you are correct. Basically to serialize coroutines you are going into the guts of how coroutines work (and for Android into replacing old instances with new ones - so you have a valid context rather than the original one - which isn't valid). I haven't really touched that code in a while so it is probably no longer working with the latest library versions that have different internal state.