Extract, Transform, and Load script for fetching new data from the NYC Open Data Portal's vehicle collision data and loading into the NYC Crash Mapper table on CARTO.
We have a system in place that sends emails if the ETL script hits an exception.
We confirmed it worked back in March:
But then as described in #28 and #25, we have had ongoing import failures that don't cause emails to be sent. I think what we need to do is add more conditions that send emails.
Like if we see error in the API response from CARTO, we should trigger an alert:
2020-09-01T17:23:25.713549+00:00 app[scheduler.3180]: 05:23:25 PM - INFO - {"error":["You are over platform's limits: SQL query timeout error. Refactor your query before running again or contact CARTO support for more details."],"context":"limit","detail":"datasource"}
We have a system in place that sends emails if the ETL script hits an exception.
We confirmed it worked back in March:
But then as described in #28 and #25, we have had ongoing import failures that don't cause emails to be sent. I think what we need to do is add more conditions that send emails.
Like if we see
errorin the API response from CARTO, we should trigger an alert:2020-09-01T17:23:25.713549+00:00 app[scheduler.3180]: 05:23:25 PM - INFO - {"error":["You are over platform's limits: SQL query timeout error. Refactor your query before running again or contact CARTO support for more details."],"context":"limit","detail":"datasource"}┆Issue is synchronized with this Asana task