ooni / data

OONI Data CLI and Pipeline v5
https://docs.ooni.org/data
8 stars 4 forks source link

Body archiver #26

Closed hellais closed 1 year ago

hellais commented 1 year ago

This adds support for archiving bodies as part of a separate command.

It currently write to an on-disk SQLite3 database. In the future we should consider writing to a clickhouse table directly so that we able to parallelise this part of the workflow.

codecov-commenter commented 1 year ago

Codecov Report

Base: 83.07% // Head: 84.15% // Increases project coverage by +1.07% :tada:

Coverage data is based on head (a2b9e5b) compared to base (75dd9ef). Patch coverage: 33.33% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #26 +/- ## ========================================== + Coverage 83.07% 84.15% +1.07% ========================================== Files 57 57 Lines 4745 4745 ========================================== + Hits 3942 3993 +51 + Misses 803 752 -51 ``` | [Impacted Files](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni) | Coverage Δ | | |---|---|---| | [tests/test\_workers.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-dGVzdHMvdGVzdF93b3JrZXJzLnB5) | `97.46% <ø> (ø)` | | | [oonidata/workers/response\_archiver.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvd29ya2Vycy9yZXNwb25zZV9hcmNoaXZlci5weQ==) | `62.41% <19.64%> (-11.04%)` | :arrow_down: | | [oonidata/cli/command.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvY2xpL2NvbW1hbmQucHk=) | `89.25% <90.90%> (+3.04%)` | :arrow_up: | | [oonidata/workers/\_\_init\_\_.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvd29ya2Vycy9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | | | [oonidata/workers/observations.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvd29ya2Vycy9vYnNlcnZhdGlvbnMucHk=) | `87.02% <100.00%> (+9.12%)` | :arrow_up: | | [oonidata/workers/ground\_truths.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvd29ya2Vycy9ncm91bmRfdHJ1dGhzLnB5) | `96.25% <0.00%> (-1.25%)` | :arrow_down: | | [oonidata/workers/common.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvd29ya2Vycy9jb21tb24ucHk=) | `97.01% <0.00%> (-0.75%)` | :arrow_down: | | [oonidata/analysis/control.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvYW5hbHlzaXMvY29udHJvbC5weQ==) | `98.21% <0.00%> (+0.01%)` | :arrow_up: | | [oonidata/models/observations.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvbW9kZWxzL29ic2VydmF0aW9ucy5weQ==) | `96.00% <0.00%> (+0.10%)` | :arrow_up: | | [oonidata/analysis/websites.py](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni#diff-b29uaWRhdGEvYW5hbHlzaXMvd2Vic2l0ZXMucHk=) | `77.67% <0.00%> (+0.15%)` | :arrow_up: | | ... and [5 more](https://codecov.io/gh/ooni/data/pull/26?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni) | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ooni)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.