search

roeap / object-store-python

Python bindings and arrow integration for the rust object_store crate.
Apache License 2.0
57 stars 9 forks source link

Async interface #2

Closed jhamman closed 5 months ago

jhamman commented 9 months ago

The Rust crate this project is based on provides an async interface. However, this project provides a synchronous interface. Would it be possible to expose an async interface here (perhaps in addition to the existing synchronous interface)?

Context: In the Zarr-Python project, we are considering using a this library as one of the store backends. In the next version of Zarr-Python, all stores will be async-only so we would want the backend to expose as much async functionality as possible.

@kylebarron recently pointed me to https://github.com/PyO3/pyo3/issues/1632#issuecomment-1752582018 which suggests "the next version of pyo3 is likely to have big progress in async handling".

cc @martindurant

roeap commented 9 months ago

Hey @jhamman - in another project we are working on there is a need for async python bindings as well, so definitely something I am interested in.

So excited to learn that there may be something coming in pyo3 itself. I guess once a good pattern is figured out, it may be quite straight forward to implement this.

I do have to find some time for this though. Updating this project to latest dependencies (and features) of object_store has been on my TODO for a while, so hopefully will get to it soon :).