search

ooni / data

OONI Data CLI and Pipeline v5
https://docs.ooni.org/data
8 stars 4 forks source link

Experiment results analysis generation #44

Closed hellais closed 9 months ago

hellais commented 9 months ago

To make it easier to review I am separating all the analysis diff into a separate PR that targets the experiment_results branch.

This way it's possible to view the diff more cleanly and make it easier to understand what is going on.

Overview of scope of this PR

Inside of this PR I implement an Experiment Results analysis approach based on the analysis tables.

The basic idea is to take the analysis keys that are generated by comparing an individual observation with ground truth data. Through a very large set of rules we are able to assign individual blocking, down and ok rules based on how confident we are in that particular signal being a sign for censorship.

We then take all the scores pertaining to a particular observation group relevant to a measurement and generate a MeasurementExperimentResult which should be backward compatible with out existing PR.

Based on this we add support for generating the experiment results based on the analysis inside of the mkanalysis command and a simple web interface for inspecting them.

In terms of performance some cursory benchmarks were run the dataset from 2023-09-01 - 2023-11-01 and it was processing data at a rate of ~7k observations per second scaling on 34 cores.

Summary of changes

codecov[bot] commented 9 months ago

Codecov Report

Attention: 148 lines in your changes are missing coverage. Please review.

Comparison is base (1f24e63) 82.14% compared to head (9b8f594) 81.19%.

:exclamation: Current head 9b8f594 differs from pull request most recent head 0a0aeaa. Consider uploading reports for the commit 0a0aeaa to get more accurate results

Files Patch % Lines
oonidata/analysis/website_experiment_results.py 77.14% 109 Missing :warning:
oonidata/dataviz/web.py 0.00% 31 Missing :warning:
oonidata/analysis/web_analysis.py 84.61% 2 Missing :warning:
oonidata/cli/command.py 80.00% 1 Missing :warning:
oonidata/db/connections.py 50.00% 1 Missing :warning:
...ata/transforms/nettests/measurement_transformer.py 80.00% 1 Missing :warning:
oonidata/workers/common.py 98.48% 1 Missing :warning:
tests/conftest.py 91.66% 1 Missing :warning:
tests/test_analysis.py 80.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## experiment-results #44 +/- ## ====================================================== - Coverage 82.14% 81.19% -0.96% ====================================================== Files 69 69 Lines 6005 6067 +62 ====================================================== - Hits 4933 4926 -7 - Misses 1072 1141 +69 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

hellais commented 9 months ago

Performed some clean up of the git commit history to make it possible to merge this without needing to rely on the github squash and merge function.

The reason to do that is that I want to preserve the history of certain changes in a dedicated commit (the ones about deleting dead code and the one about the clickhouse bug). This is so that in the future if we need to recover the dead code it's easy and so it's possible to easily see all the changes that are needed in order to address the clickhouse bug.

bassosimone commented 9 months ago

Performed some clean up of the git commit history to make it possible to merge this without needing to rely on the github squash and merge function.

The reason to do that is that I want to preserve the history of certain changes in a dedicated commit (the ones about deleting dead code and the one about the clickhouse bug). This is so that in the future if we need to recover the dead code it's easy and so it's possible to easily see all the changes that are needed in order to address the clickhouse bug.

Yeah, this seems like a good choice 💯