search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.44k stars 721 forks source link

Drop `tokio` requirement for `object_store` crate #6051

Open criccomini opened 1 month ago

criccomini commented 1 month ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'm a library developer. I would like my library to have async code, but not force developers into a specific async runtime (smol, tokio, etc). My library depends on object_store, which uses reqwest, which requires tokio to run:

My library can't use object_store without forcing all my users to adopt tokio as their async runtime.

Describe the solution you'd like

I think object_store should drop reqwest as a requirement for its cloud integrations. Perhaps adopt a different HTTP client that works with other async reactors as well?

Describe alternatives you've considered

We looked at dropping our dependency on object_store, but it's such a good library.

We are currently using futures-rs's block_on, but we get:

there is no reactor running, must be called from the context of a Tokio 1.x runtime
tustvold commented 1 month ago

It would be a fairly major undertaking to decouple the various implementations from reqwest, as whilst efforts have been made to avoid it leaking overly out into the public APIs, the exposed feature set is pretty broad and there is a significant amount of client code. At the very least I would expect subtle behaviour changes between different client implementations.

I have filed https://github.com/apache/arrow-rs/issues/6055 to track making the dependency on tokio optional, and this would then allow interested parties to build third-party implementations that expose the same ObjectStore trait for their runtime of choice.

I have also filed https://github.com/apache/arrow-rs/issues/6056 to track decoupling the HTTP client, but I don't really think this is actionable at this stage. Unless we can demonstrate tangible wins for our existing users from switching HTTP client, I think we would need to preserve the ability to use the battle-tested reqwest implementation.

alamb commented 1 month ago

Thank you for filing this issue @criccomini

I think starting with https://github.com/apache/arrow-rs/issues/6055 is a great first step -- that way the crate could be used without tokio.

We could then have a subsequent discussion on how best to provideObjectStore trait impls for various backing stores without tokio (e.g. https://github.com/apache/arrow-rs/issues/6056 or improving reqwest upstream, etc)

alamb commented 1 month ago

In case anyone cares, here is a related twitter https://twitter.com/andrewlamb1111/status/1812442468853444855