Open sandtreader opened 5 days ago
The proposal #6780 is a possible way forward on this, but it's going to take a long time. Until then, this will be unsupported on our end. In the meantime, you may be able to make your own forked Tokio that works via rubicon.
Hi Alice,
Thanks for your quick response! That RFC (https://github.com/tokio-rs/tokio/pull/6780) is interesting for dynamic library deployment in general but I'm not sure it's relevant to this problem, unless I'm missing something!
Just to reiterate, my test:
As I mentioned, this is a specific problem with (I think) some kind of static data in the multi-threaded runtime, not one with dynamic linking in general.
Thanks again!
Paul
(* actually it's not really Foreign at all, since it's calling a dylib, not a cdylib)
I understand that the RFC solves a broader problem than your problem. But it would solve your problem.
As for the single-threaded runtime, I'm pretty sure that you're just getting lucky. Both runtimes use a thread-local variable for the context, and the thread-local does not work if you use dylibs in a way that cause the application to contain several copies of Tokio.
OK, thanks - I'll have to hold out hope that that gets adopted then! :-)
I had just found this (runtime/context.rs:77) which seems the key to it, as you say:
tokio_thread_local! {
static CONTEXT: Context = const {
Context {
#[cfg(feature = "rt")]
thread_id: Cell::new(None),
// Tracks the current runtime handle to use when spawning,
// accessing drivers, etc...
#[cfg(feature = "rt")]
current: current::HandleCell::new(),
What I don't understand, though, is why when I'm calling the runtime.spawn()
directly, or use enter()
- which also fails - this isn't set in the dylib's version of the thread_local? But I'll take your word for it!
This was interesting, though: "if you use dylibs in a way that cause the application to contain several copies of Tokio". Does that mean there's a way using using dylibs where Tokio is shared?
The root cause is that you end up with several copies of the thread-local, and only one of them gets updated. I don't know if it's possible to compile the dylibs without getting multiple copies of Tokio.
Version
Platform
Debian 12
Linux j 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
Description Put simply, using Tokio multi-threaded runtime with dynamic libraries doesn't work, even if you explicitly pass it through to the library and call it directly. Single-threaded works fine.
I've created a test environment for this: https://github.com/sandtreader/rust-tokio-dylib with annotated code and a full explanation.
My guess is that the multi-threaded runtime is using some static data for its thread-pooling which is not encompassed by the runtime, and the dynamic library has a different copy of it. I've tried to remove the issue of the thread-local variables by passing the runtime directly to the library and calling it directly rather than implicitly with
tokio::spawn()
.I know this has been raised before (both here: https://github.com/tokio-rs/tokio/issues/1964, and in async-ffi: https://github.com/oxalica/async-ffi/issues/14) but I think it's important - a lot of larger systems use dynamic plugin mechanisms for flexibility and shorter build times, and it's a shame that Tokio (at least in multi-threaded form) can't be used in these. I'm certainly prepared to put effort into resolving it, although I know little about Tokio internals at the moment!
Just one more thing - yes I know the whole area of dynamic libraries in Rust has major issues around the unstable ABI. In my particular use case this isn't a major problem because it's mostly there for compositional flexibility and build times within a single workspace, but there are also possible solutions like
abi_stable
or reducing the interface to C-only. This static data problem (if that's what it is) is an orthgonal issue.Many thanks!
Paul