Add in-progress incident report for broken differ

edgi-govdata-archiving / web-monitoring-ops

Documentation and configuration files for EDGI’s deployment of Web Monitoring tools.

GNU General Public License v3.0

1 stars 1 forks source link

Add in-progress incident report for broken differ #24

Closed Mr0grog closed 5 years ago

Mr0grog commented 5 years ago

One of the diffing service pods started failing and returning a variety of errors, causing downstream errors in the DB’s auto-analysis job. @Mr0grog was offline at a camp and was unable to address it for a full day.

The problem appears to have been caused by a broken process pool, but I’m still looking into the original cause and possible remediation in code. The incident itself has been resolved, though.

/cc @danielballan

Mr0grog commented 5 years ago

Lost track of this. Not sure there’s anything useful to add, so I’m going ahead and merging.