search

delta-io / delta-kernel-rs

A native Delta implementation for integration with any query engine
Apache License 2.0
144 stars 41 forks source link

expand default parquet writer API to support multiple batches #466

Open zachschuermann opened 1 week ago

zachschuermann commented 1 week ago

from @nicklan:

I'm okay with that. Note that the trait probably wouldn't have into_engine_data, but rather something in the engine would need an into_engine_data(Vec) (or maybe iter). You need a collection of them all at once

we want to have a way to write multiple parquet files and then bundle up all the DataFileMetadata into a multi-row EngineData