kentik / ktranslate

System for pulling and pushing network data.
Apache License 2.0
57 stars 24 forks source link

No buffer mode for parquet files which can not be appended to #771

Closed i3149 closed 3 weeks ago

i3149 commented 3 weeks ago

Fixes a bug where parquet files will get corrupted because you can't just append one to another.

Adds a new type of data option which forces sinks to flush the data right away instead of buffering.

@kentik-rbarnes this is the fix you're waiting on.