parquet: support writing parquet files to S3

apibara / dna

Apibara is the fastest platform to build production-grade indexers that connect onchain data to web2 services.

https://www.apibara.com/

Apache License 2.0

168 stars 32 forks source link

parquet: support writing parquet files to S3 #320

Closed bigherc18 closed 5 months ago

bigherc18 commented 6 months ago

Simply by setting the --output-dir to an S3 URL like s3://bucket/, the S3 credentials will be loaded using the default AWS credentials provider chain

bigherc18 commented 6 months ago

#326
#325
#321
#320 👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bigherc18 and the rest of your teammates on Graphite

fracek commented 6 months ago

Maybe the batch-related bug can be fixed in #321 since it will affect any sink that uses batching.