[Bug] DB cleaner DAG cannot clean when tags or metadata are defective

cc-archive / cccatalog

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.

https://github.com/WordPress/openverse-catalog

MIT License

62 stars 60 forks source link

Closed mathemancer closed 3 years ago

mathemancer commented 3 years ago

The new DB cleaner DAG (at src/cc_catalog_airflow/dags/cleaner_workflow.py) implemented in #517 fails whenever:

Whenever the above are true, the cleaner DAG should log the defect, then clean the rest of the row.