Open criccomini opened 1 month ago
It would be a fairly major undertaking to decouple the various implementations from reqwest
, as whilst efforts have been made to avoid it leaking overly out into the public APIs, the exposed feature set is pretty broad and there is a significant amount of client code. At the very least I would expect subtle behaviour changes between different client implementations.
I have filed https://github.com/apache/arrow-rs/issues/6055 to track making the dependency on tokio
optional, and this would then allow interested parties to build third-party implementations that expose the same ObjectStore
trait for their runtime of choice.
I have also filed https://github.com/apache/arrow-rs/issues/6056 to track decoupling the HTTP client, but I don't really think this is actionable at this stage. Unless we can demonstrate tangible wins for our existing users from switching HTTP client, I think we would need to preserve the ability to use the battle-tested reqwest
implementation.
Thank you for filing this issue @criccomini
I think starting with https://github.com/apache/arrow-rs/issues/6055 is a great first step -- that way the crate could be used without tokio.
We could then have a subsequent discussion on how best to provideObjectStore
trait impls for various backing stores without tokio (e.g. https://github.com/apache/arrow-rs/issues/6056 or improving reqwest upstream, etc)
In case anyone cares, here is a related twitter https://twitter.com/andrewlamb1111/status/1812442468853444855
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I'm a library developer. I would like my library to have async code, but not force developers into a specific async runtime (smol, tokio, etc). My library depends on
object_store
, which usesreqwest
, which requirestokio
to run:My library can't use
object_store
without forcing all my users to adopttokio
as their async runtime.Describe the solution you'd like
I think object_store should drop
reqwest
as a requirement for its cloud integrations. Perhaps adopt a different HTTP client that works with other async reactors as well?Describe alternatives you've considered
We looked at dropping our dependency on
object_store
, but it's such a good library.We are currently using
futures-rs
'sblock_on
, but we get: