Open brad-richardson opened 1 month ago
Read and write directly to S3 to avoid need for large local disks
osmpbf supports arbitrary inputs that support Read + Send https://github.com/b-r-u/osmpbf/blob/main/src/reader.rs#L17
Read + Send
The parquet rust library has an object reader - https://arrow.apache.org/rust/parquet/arrow/async_reader/struct.ParquetObjectReader.html but doesn't appear to have an object writer. Can probably implement an S3AsyncWriter conforming to https://arrow.apache.org/rust/parquet/arrow/async_writer/trait.AsyncFileWriter.html
https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3.html
This has been implemented for writing out to S3
Read and write directly to S3 to avoid need for large local disks
osmpbf supports arbitrary inputs that support
Read + Send
https://github.com/b-r-u/osmpbf/blob/main/src/reader.rs#L17The parquet rust library has an object reader - https://arrow.apache.org/rust/parquet/arrow/async_reader/struct.ParquetObjectReader.html but doesn't appear to have an object writer. Can probably implement an S3AsyncWriter conforming to https://arrow.apache.org/rust/parquet/arrow/async_writer/trait.AsyncFileWriter.html
https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3.html