search

developmentseed / obstore

Simple, fast integration with Amazon S3, Google Cloud Storage, Azure Storage, and S3-compliant APIs like Cloudflare R2
https://developmentseed.org/obstore
MIT License
146 stars 3 forks source link

Kyle/fsspec integration #60

Closed martindurant closed 3 weeks ago

martindurant commented 3 weeks ago

Running this test fails: 

Exception: Generic S3 error: Error performing list request: Error after 0 retries in 39.083µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://localhost:5555/test?delimiter=%2F&list-type=2)

even though that URI is gettable with requests, returning valid XML. Thoughts?

kylebarron commented 3 weeks ago

even though that URI is gettable with requests, returning valid XML. Thoughts?

I think that usually means that it's falling back to instance authentication? Maybe we should default that to off for simpler errors

martindurant commented 3 weeks ago

we should default that to off

How to do that?

kylebarron commented 3 weeks ago

I think there'll be a bit of a learning curve for us to figure out the right set of auth configuration and ensure it's documented correctly.

So far, whenever I've hit auth errors, I've been able to re-check my credentials and get it working

kylebarron commented 3 weeks ago

https://github.com/developmentseed/obstore/pull/62 has an example of working tests against moto

martindurant commented 3 weeks ago

Perfect, I'll build on that. "Unsigned" seems wrong, but oh well.

martindurant commented 3 weeks ago

Er, something has gone very wrong with my git here. I'll get back to it tomorrow with any luck.

kylebarron commented 3 weeks ago

The branch you're trying to merge into is also behind. It's fine if you switch the target branch to main

kylebarron commented 3 weeks ago

I'll close this as I believe the only PR you intend to work on is #63, but reopen if I'm wrong.