hellais
commented
10 months ago I came to the bottom of why the processing time for measurements in 2021 buckets was so slow. The root cause of it is https://github.com/ooni/backend/issues/763.
In the interim, the refactoring in this branch does achieve marginally better performance, yet it's currently capped by the network (I am fully saturating the 1 GBiT link), see:
There isn't much more I can do except get a machine with a faster network or reprocess the postcans to compress them from a place that is closer to s3.
codecov-commenter
Summary of changes:
oonidata_processing_logsmetrics about how long it took process a certain daily batchPrior to this refactor we were achieving a rate of 5000 measurements per second, but were only reaching this speed when processing multiple bucket_dates at the same time.
This was happening because the parallelisation was happening on a per day bucket basic. When processing a single day worth of data (which is what happens in the daily batch), we were only reaching a speed of about 500 measurements per second.
As part of this PR I have added support for parallelising on a per archived file basis, which allows us to reach a performance of 7-8k measurements per second on a per day basis (which is actually faster than what we were doing before). The splitting into batches happens based on the size of the files so we should be getting a pretty consistent performance irrespective of the size of the daily batch.
This is the first step towards improving the performance of reprocessing of legacy buckets, where each daily bucket is too small to benefit from just splitting based on day (we ideally want to process batches of days together).
I also implemented as part of this same PR some changes to the schema of HIRL to make it more useful for analysis.