Support multi-threaded coroutines on Kotlin/Native

[elizarov]

You can have multiple threads in Kotlin/Native. ~Each thread can have its own event loop with runBlocking and have number of coroutines running there~. Currently communication between those threads via coroutine primitives (like channels) is not supported. This issue it to track enhancement of Kotlin/Native in kotlinx.coroutines library so that all the following becomes possible:

• Launching coroutines from one thread with a dispatcher on another thread
• Await/join coroutine running on another thread
• Send/Receive elements to/from coroutines on other threads

UPDATE: Currently, coroutines are supported only on the main thread. You cannot have coroutines off the main thread due to the way the library is currently structured.

UPDATE 2: the separate library version that supports Kotlin/Native multithreading is released on a regular basis. For the details and limitations, please follow kotlin-native-sharing.md document. The latest version: 1.5.2-native-mt

[elizarov]

📣 kotlinx.coroutines version 1.3.5-native-mt-1.4-M1 for those of you using Kotlin 1.4-M1.

[ScottPierce]

@elizarov I just filed a potential bug with the native-mt version of coroutines / Flow here: https://github.com/Kotlin/kotlinx.coroutines/issues/1895

[stefannegele]

Since this seems to get stable now, are there any plans / estimations, when something like newSingleThreadContext will be implemented? Is there a convenient way to achieve such thing already?

[sergiocasero]

Hi guys, I'm trying to use the 1.3.5-native-mt with ktor, but I'm getting this error:

HttpClient: REQUEST URL failed with exception: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.util.pipeline.Pipeline.PhaseContent@a7fe18

Uncaught Kotlin exception: kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for AwaitContinuation(Shareable[used]){HttpResponseData=(statusCode=200 OK)}@27e9c78. Please read KDoc to 'handleFatalException' method and report this incident to maintainers

I'm not pretty sure if this error should be reported to ktor guys or here, my class looks like this ->

class Remote {
    val client = HttpClient()

    suspend fun getData() = client.get<Data>()

}

I suppose that this is because I'm not creating Remote instance on the bg thread, Am I right?

[kpgalligan]

@sergiocasero ktor is not compatible with the MT coroutines. We have a very hacky workaround here until they are compatible. The summary is you can use ktor as long as you don't freeze the Job in your context. https://github.com/touchlab/KaMPKit/blob/master/shared/src/iosMain/kotlin/co/touchlab/kampstarter/ktor/Platform.kt

[sergiocasero]

Thanks @kpgalligan :), But if I understand, the workaround executes the ktor calls on the main thread, right?

[kpgalligan]

@sergiocasero ktor calls are suspending. The call starts on the main thread, but then suspends and the underlying networking will happen async. When the call is complete, your code will resume on the main thread. So, it won't block on the main thread, but our calls start there. We haven't tried doing all that on a different thread, but haven't needed to (because the calls suspend).

[sergiocasero]

jmmm, I think I understand you, thanks @kpgalligan :)

[dimsuz]

@kpgalligan do you by any chance know an issue on the ktor bugtracker I can subscribe to in this regard?

[joselufo]

@dimsuz I think this is the issue created by @kpgalligan related to ktor.

https://github.com/ktorio/ktor/issues/1538

[CoreyKaylor]

Are there any plans to push a new native-mt version for coroutines 1.3.7 and kotlin 1.3.72? Unless I'm missing something it doesn't seem like there's a published version that includes the recent addition of StateFlow. I've had some curiosity and want to try it out on iOS.

[gswierczynski]

Hi,

First thank you for all your work. Great stuff.

I am new to coroutines, but I think I have a grasp of things already. If I am not using them correctly in below examples please correct me.

For below native example I used: 1.4-M1 and 1.3.5-2-native-mt-1.4-M1

Native:

fun main() = runBlocking {

    val durationInSFirst = 2
    val durationInSSecond = 1

    launch(Dispatchers.Default) {
        platform.posix.sleep(durationInSFirst.toUInt())
        println("resumed first", coroutineContext)
    }
    println("launched first (block for ${durationInSFirst}s)", coroutineContext)

    launch(Dispatchers.Default) {
        platform.posix.sleep(durationInSSecond.toUInt())
        println("resumed second", coroutineContext)
    }
    println("launched second (block for ${durationInSSecond}s)", coroutineContext)
}
private fun println(msg: String, coroutineContext: CoroutineContext) {
    println("${getTimeMillis()} $coroutineContext: $msg")
}

prints:

932193135: launched first (block for 2s) | [BlockingCoroutine{Active}@4ad05e68, ShareableEventLoop@4ad05ba8]
932193135: launched second (block for 1s) | [BlockingCoroutine{Active}@4ad05e68, ShareableEventLoop@4ad05ba8]
932195138: resumed first | [StandaloneCoroutine{Active}@4ad06198, WorkerCoroutineDispatcherImpl@4ad05d68]
932196143: resumed second | [StandaloneCoroutine{Active}@4ad07588, WorkerCoroutineDispatcherImpl@4ad05d68]

JVM:

fun main() = runBlocking {

    val durationInSFirst = 2
    val durationInSSecond = 1

    launch(Dispatchers.Default) {
        Thread.sleep(durationInSFirst * 1000L)
        println("resumed first", coroutineContext)
    }
    println("launched first (block for ${durationInSFirst}s)", coroutineContext)

    launch(Dispatchers.Default) {
        Thread.sleep(durationInSSecond * 1000L)
        println("resumed second", coroutineContext)
    }
    println("launched second (block for ${durationInSSecond}s)", coroutineContext)
}

private fun println(msg: String, coroutineContext: CoroutineContext) {
    println("${System.currentTimeMillis()} $coroutineContext: $msg")
}

prints

1592129905337: launched first (block for 2s) | [BlockingCoroutine{Active}@5eb5c224, BlockingEventLoop@53e25b76]
1592129905341: launched second (block for 1s) | [BlockingCoroutine{Active}@5eb5c224, BlockingEventLoop@53e25b76]
1592129906342: resumed second | [StandaloneCoroutine{Active}@57265a91, DefaultDispatcher]
1592129907338: resumed first | [StandaloneCoroutine{Active}@7cc3330e, DefaultDispatcher]

Please notice the difference in the order of the two last lines of the output.

Is Dispatchers.Default on Native backed by a single thread at the moment?

[LouisCAD]

@gswierczynski

Is Dispatchers.Default on Native backed by a single thread at the moment?

Yes, it is even documented in #1648

[gswierczynski]

@LouisCAD Got it, thanks Documentation: https://github.com/Kotlin/kotlinx.coroutines/blob/native-mt/kotlin-native-sharing.md

A Default dispatcher on Kotlin/Native contains a single background thread. This is the dispatcher that is used by default in GlobalScope.

This limitation may be lifted in the future with the default dispatcher becoming multi-threaded and/or its coroutines becoming isolated from each other, so please do not assume that different coroutines running in the default dispatcher can share mutable data between themselves.

[elizarov]

📣 UPDATE: Version 1.3.7-native-mt-1.4-M2 is released — the fresh update of native-mt for Kotlin 1.4-M2. For leak-free operation, this version should be used with a specially patched version of Kotlin: 1.4-M2-eap-23 (it differs from stock 1.4-M2 in Kotlin/Native only).

This version includes fixes for additional bugs discovered by native-mt early adopters (thanks a lot for all of you ❤️):

• Fixed awaitAll in multi-threaded scenarios (#2025)
• Made sure that FreezingException does not break coroutine machinery (#2064)

The code is now based on the new features available in Kotlin/Native 1.4 (namely WorkerBoundReference) and there will be no more updates from native-mt branch for Kotlin 1.3. We plan to include the code from native-mt branch into the default kotlinx.coroutines distribution as Kotlin/Native version of the library for upcoming Kotlin 1.4 release.

[CoreyKaylor]

Edit: After a few more tries, I finally found a version that worked. I figured it out once I realized StateFlow seems almost designed for something like an actor and that lead me to where I needed to go.

Decided to give version 1.3.7-native-mt-1.4-M2 a spin to see if I can make use of StateFlow on iOS, but can't seem to find the right combination (tried numerous variations) to set StateFlow without getting the InvalidMutabilityException. Any ideas on how I could make this work? The error gets thrown in the call to stateSetter. I based the organization of flow execution from the docs on flows, and specifically the section flowOn from here https://kotlinlang.org/docs/reference/coroutines/flow.html

typealias Reducer<TState> = (TState, Action) -> TState

@OptIn(ExperimentalCoroutinesApi::class)
open class KStore<TState:Any>(initialState: TState, private val reducer: Reducer<TState>) {
    private val state = MutableStateFlow(initialState)

    private fun currentState() = state.value
    private fun setState(newState: TState) {
        state.value = newState
    }

    fun dispatch(action: Action) {
        val currentGetter: () -> TState = ::currentState
        val stateSetter: (TState) -> Unit = ::setState
        val stateChanges = when(action) {
            is FlowAction -> action.toFlow()
            else -> flow { emit(action )}
        }.map {
            reducer(currentGetter(), it)
        }.flowOn(Dispatchers.Default)
        println("Got state changes")
        stateChanges.onEach {
            stateSetter(it)
        }.launchIn(UIScope())
    }

    //callback method for pushing back to swift, hopefully a better alternative can be found
    fun listen(onStateChange: (TState) -> Unit) = runBlocking {
        state.onEach {
            onStateChange(it)
        }.launchIn(UIScope())
    }
}

[Thomas-Vos]

We plan to include the code from native-mt branch into the default kotlinx.coroutines distribution as Kotlin/Native version of the library for upcoming Kotlin 1.4 release.

@elizarov does that mean that the native-mt branch will be merged into master or will it be a separate package that can only be used on Kotlin/Native? Would it still be possible with Kotlin 1.4 to use the native-mt code on JVM/Android? The native-mt branch fixes a crash on Android devices so it would be great if it could be used on JVM/Android as well. See: https://github.com/Kotlin/kotlinx.coroutines/pull/1648#issuecomment-617143634

[elizarov]

We plan to include the code from native-mt branch into the default kotlinx.coroutines distribution as Kotlin/Native version of the library for upcoming Kotlin 1.4 release.

@elizarov does that mean that the native-mt branch will be merged into master or will it be a separate package that can only be used on Kotlin/Native? Would it still be possible with Kotlin 1.4 to use the native-mt code on JVM/Android? The native-mt branch fixes a crash on Android devices so it would be great if it could be used on JVM/Android as well. See: #1648 (comment)

We don't plan to merge all the code from native-mt in the master, but gradually work on porting its pieces. Only native code will be built from native-mt branch to start with. It would be great if we can, at least, figure out the bytecode of which classes causes the crash, so that we can concentrate on moving those changes to master first.

[slipdef]

@elizarov I still don't quite understand what should we expect with Kotlin 1.4 in regards to coroutines multi-threading support. 1.4 M3 was released 2 weeks ago with all major libs (ktor, serialization, non-mt couroutines). Coroutines mt version wasn't included. Still no official announcement when native-mt M3 is going to be released. If you plan to include native-mt in master with 1.4 release then why it's considered as a second class citizen now? Also could you explain exactly in what form coroutines-mt will be released with Kotlin 1.4 because I hardly understand what does it mean in regards to 1.4 - We don't plan to merge all the code from native-mt in the master, but gradually work on porting its pieces.

[raniejade]

@elizarov Is there a native-mt build for Kotlin 1.4-M3?

[elizarov]

📣 Published version 1.3.8-native-mt-1.4.0-rc for Kotlin 1.4.0-rc

[IanArb]

@elizarov Is this published for the kotlinx-coroutines-core-common module too?

[joreilly]

@IanArb with new "hierarchical project structure" support that was added since 1.4-M2 (https://blog.jetbrains.com/kotlin/2020/06/kotlin-1-4-m2-released/) you only need dependency on org.jetbrains.kotlinx:kotlinx-coroutines-core

[IanArb]

@IanArb with new "hierarchical project structure" support that was added since 1.4-M2 (https://blog.jetbrains.com/kotlin/2020/06/kotlin-1-4-m2-released/) you only need dependency on org.jetbrains.kotlinx:kotlinx-coroutines-core

Thanks!

[joreilly]

@elizarov more of an fyi (and in case there's any known workaround) but getting following when using 1.3.8-native-mt-1.4.0-rc in a macOS project built using XCode 12 beta (not seeing issue in XCode 11.6). Also not seeing issue with the nonnative-mt version.

Undefined symbols for architecture x86_64:
  "__kernelrpc_mach_port_destroy_trap", referenced from:
      _platform_darwin__kernelrpc_mach_port_destroy_trap_wrapper850 in common(result.o)
     (maybe you meant: knifunptr_platform_darwin883__kernelrpc_mach_port_destroy_trap, _platform_darwin__kernelrpc_mach_port_destroy_trap_wrapper850 )

[SvyatoslavScherbina]

@joreilly I don't think it is related to kotlinx.coroutines. Could you create a Kotlin issue? https://kotl.in/issue We'll also need additional details about your project, let's discuss this in the new issue.

[joreilly]

@SvyatoslavScherbina thanks, have created https://youtrack.jetbrains.com/issue/KT-40782

[hbucius]

Do we have any change log and doc for 1.3.8-native-mt-1.4.0-rc build?

[qwwdfsad]

1.3.9-native-mt is here: mirror of 1.3.9 with the same changes in Native part as in previous -native-mt versions

[uOOOO]

Do you have any plan for a new native-mt version including fix #2025? I can't use ktor log level BODY and ALL with 1.3.9-native-mt.

[uOOOO]

1.3.9-native-mt-2 was released and ktor is working without any problem on native target. 👍

[werner77]

Anybody has any idea why this issue exists in the mt version? https://github.com/Kotlin/kotlinx.coroutines/issues/2335

runBlocking is not the only problem, I also witnessed code not being executed in main code. In particular with nested withContext() blocks.

[ntherning]

I filed this report https://youtrack.jetbrains.com/issue/KT-42898 which I am a bit surprised no one else is running into. Basically, what happens is that a child coroutine returning a new instance of a class causes that instance to be frozen, even when all coroutines run on the same thread. I wouldn't expect this.

Perhaps I'm doing something stupid. Would be awesome if someone on the team could have a quick look and determine whether this is just me or an actual bug.

[brettwillis]

May I have an ETA on a 1.4.x-native-mt release? This is blocking ktorio/ktor#2160, and as ktor seems to enforce strict coroutines version check on native, and I've already migrated my codebase to use shared flows, this is blocking my development.

As it's been two week since 1.4.0 I'm wondering if this will happen soon or if I should migrate back to 1.3.9 to unblock development...

Thanks 🙏

[elizarov]

UPDATE: kotlinx-coroutines version 1.4.1-native-mt released.

[jcraane]

Does this also fix this issue https://youtrack.jetbrains.com/issue/KT-38770 ?

Haven't tested it myself yet but am planning to do that tomorrow. Will update this thread with my results.

[TrevorStoneEpic]

I might be missing something, when I create a MutableStateFlow but set the value I get an InvalidMutabilityException. The same code worked in 1.3.9. Has anyone else ran into issues?

[twyatt]

@TrevorStoneEpic others are experiencing that issue as well, looks like #2138 was re-opened and is tracking it.

[qwwdfsad]

Released 1.4.2-native-mt

[jcraane]

Does this also fix this issue https://youtrack.jetbrains.com/issue/KT-38770 ?

Haven't tested it myself yet but am planning to do that tomorrow. Will update this thread with my results.

I can confirm 1.4.2-native-mt does fix the issues we had with immutability exceptions!

[andersio]

@qwwdfsad Is there any plan to make a new release soon? Would appreciate one that incorporates #2477 and #2476. 🙇

[qwwdfsad]

1.4.3-native-mt is released

[BulatMukhutdinov]

Calling suspended functions in Swift from a background thread causes an exception with Calling Kotlin suspend functions from Swift/Objective-C is currently supported only on main thread. Does it mean that we only can change threads in Kotlin native part of the code and have to call suspended functions from the main thread on Swift?

I am using 1.4.3-native-mt version.

[qwwdfsad]

I suggest all users of native-mt builds to read the following announcement: https://blog.jetbrains.com/kotlin/2021/08/try-the-new-kotlin-native-memory-manager-development-preview/

We aim to merge the support of the new MM into coroutines 1.6.0 (https://github.com/Kotlin/kotlinx.coroutines/issues/2914) and then, depending on the stability of the new MM, promote it as the default way to write concurrent code on Native platforms.

native-mt won't be supported as soon as the new MM is stable.

[ankushg]

native-mt won't be supported as soon as the new MM is stable.

Just to confirm: this means while the new MM is still in "preview", we will still be getting -native-mt builds of kotlinx.coroutines releases, correct?

[qwwdfsad]

We definitely going to release 1.6.0-native-mt, along with minor updates like 1.6.x-native-mt.

We are not sure about 1.7.0 and above though, it will greatly depend on the status of the new MM in Kotlin 1.7.0 and the general perception.

I would anyway strongly recommend migrating from native-mt to a new memory model if possible during the course of the next 6 months, as we plan to decommission native-mt eventually

[sebleclerc]

@qwwdfsad I understand that if we migrate to the new memory model, we don't have to use the native-mt release? Am I correct? Thanks 😄

[qwwdfsad]

@sebleclerc correct, you can use regular releases (starting from 1.6.0-RC) and forget about freezes and InvalidMutabilityException as a bonus

[Legion2]

Is the New memory model migration guide still up-to-date with the new 1.6.0-RC coroutines version and the stable 1.6.0 kotlin release?

[qwwdfsad]

Yes, this document is kept up-to-date