search

GeoSensorWebLab / data-transloader

Download and convert weather data for SensorThings API
GNU General Public License v3.0
3 stars 1 forks source link

Observations from Environment Canada fail to upload #2

Closed openfirmware closed 5 years ago

openfirmware commented 5 years ago

The latest uploads of Observations for Environment Canada stations into STA are 2019-10-16. I checked the logs in Apache Airflow, and it looks like the URL for station downloads changed. As the ETL client does not follow 3xx redirects, it doesn't download anything at all and exits successfully.

The new URLs for the station observation files are now HTTPS instead of HTTP, which is a simple fix. But when this happens the "download" tool should fail to make it more obvious in Airflow that a problem has occurred.

openfirmware commented 5 years ago

Error code failure is already being raised by the transload tool; the issue is that the Bash wrapper scripts that are used by Apache Airflow did not use set -e and would allow failures and return exit code 0. That meant Airflow would not mark the DAG run as a failure. I updated the Chef cookbook for the production server to fix this.