Support for dynamic linking

[sandtreader]

Version

└── tokio v1.41.0
    └── tokio-macros v2.4.0 (proc-macro)

Platform

Debian 12 Linux j 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

Description Put simply, using Tokio multi-threaded runtime with dynamic libraries doesn't work, even if you explicitly pass it through to the library and call it directly. Single-threaded works fine.

I've created a test environment for this: https://github.com/sandtreader/rust-tokio-dylib with annotated code and a full explanation.

My guess is that the multi-threaded runtime is using some static data for its thread-pooling which is not encompassed by the runtime, and the dynamic library has a different copy of it. I've tried to remove the issue of the thread-local variables by passing the runtime directly to the library and calling it directly rather than implicitly with tokio::spawn().

I know this has been raised before (both here: https://github.com/tokio-rs/tokio/issues/1964, and in async-ffi: https://github.com/oxalica/async-ffi/issues/14) but I think it's important - a lot of larger systems use dynamic plugin mechanisms for flexibility and shorter build times, and it's a shame that Tokio (at least in multi-threaded form) can't be used in these. I'm certainly prepared to put effort into resolving it, although I know little about Tokio internals at the moment!

Just one more thing - yes I know the whole area of dynamic libraries in Rust has major issues around the unstable ABI. In my particular use case this isn't a major problem because it's mostly there for compositional flexibility and build times within a single workspace, but there are also possible solutions like abi_stable or reducing the interface to C-only. This static data problem (if that's what it is) is an orthgonal issue.

Many thanks!

Paul

[Darksonn]

The proposal #6780 is a possible way forward on this, but it's going to take a long time. Until then, this will be unsupported on our end. In the meantime, you may be able to make your own forked Tokio that works via rubicon.

[sandtreader]

Hi Alice,

Thanks for your quick response! That RFC (https://github.com/tokio-rs/tokio/pull/6780) is interesting for dynamic library deployment in general but I'm not sure it's relevant to this problem, unless I'm missing something!

Just to reiterate, my test:

• Exists all within one workspace (same compiler, same Tokio version)
• Uses FFI * only for a single 'init' method
• Works with the single-threaded runtime

As I mentioned, this is a specific problem with (I think) some kind of static data in the multi-threaded runtime, not one with dynamic linking in general.

Thanks again!

Paul

(* actually it's not really Foreign at all, since it's calling a dylib, not a cdylib)

[Darksonn]

I understand that the RFC solves a broader problem than your problem. But it would solve your problem.

As for the single-threaded runtime, I'm pretty sure that you're just getting lucky. Both runtimes use a thread-local variable for the context, and the thread-local does not work if you use dylibs in a way that cause the application to contain several copies of Tokio.

[sandtreader]

OK, thanks - I'll have to hold out hope that that gets adopted then! :-)

I had just found this (runtime/context.rs:77) which seems the key to it, as you say:

tokio_thread_local! {
    static CONTEXT: Context = const {
        Context {
            #[cfg(feature = "rt")]
            thread_id: Cell::new(None),

            // Tracks the current runtime handle to use when spawning,
            // accessing drivers, etc...
            #[cfg(feature = "rt")]
            current: current::HandleCell::new(),

What I don't understand, though, is why when I'm calling the runtime.spawn() directly, or use enter() - which also fails - this isn't set in the dylib's version of the thread_local? But I'll take your word for it!

This was interesting, though: "if you use dylibs in a way that cause the application to contain several copies of Tokio". Does that mean there's a way using using dylibs where Tokio is shared?

[Darksonn]

The root cause is that you end up with several copies of the thread-local, and only one of them gets updated. I don't know if it's possible to compile the dylibs without getting multiple copies of Tokio.