Investigate seekable Zstandard to speed up measurement access

ooni / backend

Everything related to OONI backend infrastructure: ooni/api, ooni/pipeline, ooni/sysadmin, collector, bouncers and test-helpers

BSD 3-Clause "New" or "Revised" License

48 stars 28 forks source link

Investigate seekable Zstandard to speed up measurement access #782

Open FedericoCeratto opened 6 months ago

FedericoCeratto commented 6 months ago

Zstandard supports seekable compression. This could be use to fetch only the required row[s] from a JSONL file on S3 when accessing the measurement in Explorer or when reprocessing measurements in batches.

Example: https://github.com/Fuuzetsu/zstd-seekable-s3

Related to #191