Surface open data portal upload failures at end of dag run

cal-itp / data-infra

Cal-ITP data infrastructure

https://docs.calitp.org/data-infra

GNU Affero General Public License v3.0

48 stars 13 forks source link

Surface open data portal upload failures at end of dag run #3234

Open charlie-costanzo opened 10 months ago

charlie-costanzo commented 10 months ago

We recently implemented exception handling in the airflow job for the open data portal upload, so that if one table fails to upload to CKAN it will continue to attempt the others.

However, we still want to surface those failures at the end of the run so that the workflow remains red and we can address failures when they come up.

tiffanychu90 commented 8 months ago

Just wanted to add a note here that stop_times was only bumped to 4 GB. Slack thread here and here. Solr was unwilling to bump too much, only 1GB each ticket, and we should anticipate it takes a couple weeks for our ticket to make our way from CDT -> Solr -> reach out to us to confirm -> finally implement.