cc-archive / cccatalog

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.
https://github.com/WordPress/openverse-catalog
MIT License
63 stars 60 forks source link

[Feature] Apache Airflow DAG to run the new walters_art_museum.py script #507

Closed dravadhis closed 4 years ago

dravadhis commented 4 years ago

Problem Description

In order to get the walters_art_museum.py script (see [#506 ]) into production, we need to implement a new Apache Airflow DAG to run the script.

Solution Description

Implement such a dag. For an example, see src/cc_catalog_airflow/dags/flickr_workflow.py (note the use of the new DAG Factory). The DAG should be configured to run the main function from src/cc_catalog_airflow/dags/provider_api_scripts/walters_art_museum.py (no parameter) once per month, with a 24-hour Dag Run timeout. Leave concurrency at its default value of 1.