[Bug] Popularity Data Workflow should not run monthly

cc-archive / cccatalog

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.

https://github.com/WordPress/openverse-catalog

MIT License

63 stars 60 forks source link

[Bug] Popularity Data Workflow should not run monthly #449

Closed mathemancer closed 4 years ago

mathemancer commented 4 years ago

Bug Description

The Apache Airflow DAG defined in src/cc_catalog_airflow/dags/popularity_workflow.py is currently configured to run monthly, but this is not necessary. Further, running this DAG monthly creates potential for failures when we do want to run it, since it runs for multiple days, and is not set up to run more than once concurrently.

This is a relatively simple change to the configuration of the DAG.

avats-dev commented 4 years ago

Hey @mathemancer @kgodey , I'm new here. I couldn't find src/cc_catalog_airflow/dags/popularity_workflow.py anywhere. Is this already done, as it should be closed then or it is located elsewhere?

mathemancer commented 4 years ago

@avats-dev Great catch, thanks! This ticket is out of date. We ended up decommissioning that DAG and replacing it with the following three DAGs (for which the issue described in the ticket isn't a problem):

recreate_image_popularity_calculation.py
refresh_all_image_popularity_data.py
refresh_image_view_data.py

All are in the src/cc_catalog_airflow/dags folder at the moment.