developmentseed / obstore

Simple, fast integration with Amazon S3, Google Cloud Storage, Azure Storage, and S3-compliant APIs like Cloudflare R2
https://developmentseed.org/obstore
MIT License
146 stars 3 forks source link

Add async iterator over bytes in result of `get` #11

Closed kylebarron closed 1 month ago

kylebarron commented 1 month ago

This is super cool. Really excited to test this tomorrow. Original credit goes to @timteeee in https://github.com/roeap/object-store-python/pull/29

Tomorrow I'll add a test for this.

kylebarron commented 1 month ago
import boto3
import object_store_rs as obs

store = obs.store.S3Store.from_session(boto3.Session(), "ookla-open-data")
path = "parquet/performance/type=mobile/year=2023/quarter=3/2023-07-01_performance_mobile_tiles.parquet"

resp = obs.get(store, path)
async for buf in aiter(resp):
    print(len(buf))

that's super cool