hellais
commented
2 weeks ago At some point we should do some performance tuning. Leaving here a few relevant links:
- https://docs.temporal.io/develop/worker-performance
- Why did test runtime increase when max_workers was added in the CI: https://github.com/ooni/data/actions/runs/10216291832/job/28267448413 vs https://github.com/ooni/data/actions/runs/10515053692/job/29134296033
At the moment ooni/data runs on a daily scheduled based on cron.
It would be good to spend some time codifying the deployment in ansible and/or considering some more sophisticated tooling for running it and monitoring it's progress.
Airflow seems like the thing that might actually fit the bill quite nicely, so perhaps it could be considered to dust that back out of the closet.