EmbarkStudios / cervo

Utility wrappers for tract
Apache License 2.0
40 stars 1 forks source link

Add support for threaded execution #35

Closed tgolsson closed 1 year ago

tgolsson commented 1 year ago

Is your feature request related to a problem? Please describe.

There's no dependency between individual models when using cervo-runtime, so supporting threaded execution should be fairly easy at the "top" layer.

Describe the solution you'd like

I'm not super-enthused about depending on a specific threading model; so I'd prefer something that was injected. However; task-style dispatch makes gathering results harder than something like Rayon. Not sure what SOTA is here. Something like blocking handles + oneshot channels at the collect step?

At the runtime layer; we would want something like

#[cfg(feature = "threading")]
impl Runtime {
    fn execute_threaded[_for](&mut self, dispatcher: impl Dispatcher[, time_limit: std::time::Duration]) -> ... {
        let max_parallelism = dispatcher.max_threads();

        // do some dispatch over all threads

       // gather all results
    }
}