search

flavray / avro-rs

Avro client library implementation in Rust
MIT License
169 stars 95 forks source link

Question: How to deserialize to a columnar format #193

Open jorgecarleitao opened 3 years ago

jorgecarleitao commented 3 years ago

Hey, Thank you so much for this library!

I am considering deserializing avro to arrow, and I am looking for an efficient way of doing it.

In other row-based formats (CSV, JSON), we have been doing the following:

  1. separate "reading" (Read+Seek -> Vec<Row> or AsyncRead+AsyncSeek -> Future<Vec<Row>>) from "deserializing" (Vec<Row> -> Vec<Columns>)
  2. use a reading to &mut [Row], so that we can use a reuse a single Vec<Row> for multiple "batches of rows".

Could someone offer some guide for doing this?