search

ooni / backend

Everything related to OONI backend infrastructure: ooni/api, ooni/pipeline, ooni/sysadmin, collector, bouncers and test-helpers
BSD 3-Clause "New" or "Revised" License
48 stars 28 forks source link

Update fastpath to feed from JSONL files #778

Closed FedericoCeratto closed 3 months ago

FedericoCeratto commented 7 months ago

Decompression should be more efficient leading to faster reprocessing and CI runs.

hellais commented 3 months ago

ooni/data already uses JSONL files and the fastpath will eventually not need to do any reprocessing (see future of OONI infrastructure).

We should however evaluate if we need to use a more performant file format that isn't JSONL in ooni/data, since JSON parsing is currently the bottlneck there, see: https://github.com/ooni/data/issues/59