netenglabs / suzieq

Using network observability to operate and design healthier networks
https://www.stardustsystems.net/
Apache License 2.0
791 stars 106 forks source link

sq-coalescer: fix multiple writes on the same parquet file #826

Closed claudiolor closed 1 year ago

claudiolor commented 1 year ago

Description

The PR fixes a bug causing the coalescer to write multiple times the same coalesced file. This was because the coalescer wrote in advance the files for all the coalescing windows detected during the sqpoller coalescing, and then wrote again the file with the diff coming from the window. Additionally, now the coalescer doesn't write anymore a coalesced file for each coalescing window even when there aren't changes, as in the reading phase the engine is able to handle missing coalescing periods, always retrieving the data according to start and end time.

Type of change

Please delete options that are not relevant.