search

kbajalc / parquets

MIT License
43 stars 21 forks source link

Direct upload to cloud #24

Open govthamreddy opened 3 years ago

govthamreddy commented 3 years ago

Is there anyway to upload parquet file directly to cloud without creating a local file?

dobesv commented 3 years ago

You can write a parquet file one row group at a time using the ParquetTransformer, and pass those onto your client library for your cloud storage.

dobesv commented 3 years ago

See also https://stackoverflow.com/questions/37336050/pipe-a-stream-to-s3-upload which shows an example of how to pass through a writable stream to the AWS client.