edgi-govdata-archiving / web-monitoring-ops

Documentation and configuration files for EDGI’s deployment of Web Monitoring tools.
GNU General Public License v3.0
1 stars 1 forks source link

Add incident report for 2-month Wayback Failure #16

Closed Mr0grog closed 5 years ago

Mr0grog commented 5 years ago

It seems I left the checkout of web-monitoring-processing that loads data in production from the Wayback Machine in a bad, merge-conflict state, so it failed immediately with Python syntax errors every time it ran. No process was set up to alert for this situation. (We normally rely on Sentry for this, but it runs in-process and if Python never actually runs our code, it doesn’t get a chance to work.)

The problem persisted for nearly two months (since about the start of the shutdown).

See also this explanation from the Dev meeting the next day: https://youtu.be/vcpmvMppM-0?t=654