Use Async Parquet Writer for `dora-record`

dora-rs / dora

DORA (Dataflow-Oriented Robotic Application) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.

https://dora-rs.ai

Apache License 2.0

1.36k stars 69 forks source link

Use Async Parquet Writer for `dora-record` #434

Closed haixuanTao closed 4 months ago

haixuanTao commented 4 months ago

This PR makes it possible to:

Concurrently write records of data
Use Apache Parquet Compression to reduce storage size to avoid slow transfer speed.