Move to thread per core

[rklaehn]

We need to do this asap otherwise it will be a lot of effort.

There are a lot of perf gains to be had here - changing Arc to Rc and Mutex to RefCell. Even quinn is a bit faster when used in a single threaded executor.

One question is if we can make our public api non send without driving our users crazy...

• [x] iroh-bytes
• [ ] iroh-net

[dignifiedquire]

One question is if we can make our public api non send without driving our users crazy...

we can probably provide a channel based interface, to make that easy

[rklaehn]

So here is a little hack to easily and gradually move to thread per core:

Wherever you have a non-send future, you just use this, which spawns the whole thing on one of the blocking io threads. Which is not so bad, since this is usually io heavy.

This is kind of using the blocking thread pool for the single threaded futures. Later we could go to a dedicated thread pool.

async fn run_non_send<F, Fut, T>(f: F) -> T
where
    F: FnOnce() -> Fut + Send + 'static,
    Fut: Future<Output = T> + 'static,
    T: Send + 'static,
{
    tokio::task::spawn_blocking(move || {
        tokio::runtime::Handle::current().block_on(tokio::task::LocalSet::new().run_until(f()))
    })
    .await
    .unwrap()
}

[flub]

I was going to point to tokio::task::spawn_local but realised I never used it so looked up how it worked and... oh dear, you still have to combine it with what you do IIUC: https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html

[rklaehn]

Yes, it is all a bit of a mess...

[dignifiedquire]

A first start was done in #1106. The next step here will be to make use of this in iroh-net

[rklaehn]

I am happy with how this works for iroh-bytes and io in general. So checking the first box.