search

destrex271 / pgwatch3_rpc_server

RPC Server Implementation for pgwatch3
BSD 3-Clause "New" or "Revised" License
3 stars 0 forks source link

Parquet Receiver in Remote Sinks. #9

Closed destrex271 closed 3 months ago

destrex271 commented 4 months ago

Addresses #5

@pashagolub can you review this PR? Would love to have your opinion on how we can handle appends better in case of Parquet files.

Right now we are storing all the measurements in a single parquet file for each database.

Once we get some new measurements, we read the existing measurements from the file, append the new measurements to the slice and write it back to the file.

All this is done because we cannot directly append to parquet file afaik.

Alternative: We create multiple files each time a MeasurementMessage comes, and merge the parquet files at a fixed interval

destrex271 commented 3 months ago

Merging this PR, will push better operation to get this working in another PR.