Add support for using parquet files - Githubissues

archiver-appliance / epicsarchiverap

This is an implementation of an archiver for EPICS control systems that aims to archive millions of PVs.

Other

38 stars 37 forks source link

Add support for using parquet files #208

Closed jacomago closed 5 months ago

jacomago commented 5 months ago

Adds support in the plainstorage plugin for storing pb messages in parquet files.

Only recompress the parquet rows when the compression mode changes.

This is unnecessary when PR apache/parquet-mr#1136 is in the parquet release.

Fix ETLTimeTest to handle src made up of ZIP files.
Fix NamedFlagETLTest to use a single configservice so as to not conflict test with each other.
Fix ETLPostProcessorTest to use a single configservice so as to not conflict test with each other.

removes need for single forks tests tag Removes need for slow tests tag

Combine parquet and pb compression mode

Adds a new wrapper class for CompressionCodec from parquet and CompressionMode in pb