search

developmentseed / obstore

Simple, fast integration with Amazon S3, Google Cloud Storage, Azure Storage, and S3-compliant APIs like Cloudflare R2
https://developmentseed.org/obstore
MIT License
146 stars 3 forks source link

fsspec integration #63

Closed martindurant closed 1 week ago

martindurant commented 3 weeks ago

Replaces #60

martindurant commented 3 weeks ago

I don't know why the conftest is not being picked up (works locally). I could put the fixtures into a utils module ...

kylebarron commented 3 weeks ago

I'm afraid I don't know enough about pytest to understand why the test setup isn't working correctly. Does it work if you move it into conftest.py?

martindurant commented 2 weeks ago

Actually, all the tests are failing in CI, not what I see locally :|

martindurant commented 2 weeks ago

green :)

martindurant commented 2 weeks ago

(this fails because he _fetch_range fix in fsspec isn't released yet)

martindurant commented 2 weeks ago

OK, so the prefix case doesn't work ATM (I wasn't sure if you were implying it should) and "create" doesn't raise for moto. I haven't tried real s3 yet.

kylebarron commented 2 weeks ago

I don't think "create" will work until https://github.com/apache/arrow-rs/pull/6682

martindurant commented 2 weeks ago

OK, I can make that one xfail, and I suppose we wait for fsspec's release before merging here?

kylebarron commented 2 weeks ago

If this doesn't work until the next fsspec release, we should wait to merge until that is released and then probably do an fsspec version check on module import.

martindurant commented 2 weeks ago

It would work now with the subclass I had before, which might prove convenient in the long run... up to you.

kylebarron commented 2 weeks ago

I'm ok with the subclass as long as it's not exposed in a public API. It's essentially just an implementation detail for now, right? We could remove it in the future easily when the new fsspec version is released?

martindurant commented 1 week ago

Let's keep to this formalism for now, then.

kylebarron commented 1 week ago

Publishing v0.3.0-beta.6

kylebarron commented 1 week ago

We still need to add documentation about this to the website