OONI Data, OONI Pipeline split - Githubissues

ooni / data

OONI Data CLI and Pipeline v5

https://docs.ooni.org/data

8 stars 4 forks source link

OONI Data, OONI Pipeline split #60

Closed hellais closed 4 months ago

hellais commented 5 months ago

This PR introduces a major refactoring of oonidata into two separate packages:

oonidata, which is the end user pip installable package to download and parse measurements (should have minimal dependencies and not require additional components to run, eg. clickhouse)
oonipipeline, which is the thing that actually performs the analysis and processing of data. This does require external dependencies to run.

Eventually we might want to move them into their own respective repos.

This fixes the following issues:

https://github.com/ooni/data/issues/41, we keep oonidata as the name for the CLI tool, but oonipipeline is the actual component doing analysis.
https://github.com/ooni/data/issues/46, we are currently using temporal.io. After this lands we might want to make prototypes of the other orchestration systems to compare.
https://github.com/ooni/data/issues/57, we might eventually not go for monorepo, but for the moment this is what we got.

hellais commented 5 months ago

Note: you might see many removed lines and wonder like me what caused that. The cause for that is we dropped poetry in place of hatch and remove the poetry.lock which was 3.5k lines.