ckelner
commented
9 years ago See official issue opened with Netflix: https://github.com/Netflix/ice/issues/171
Open ckelner opened 9 years ago
ckelner
commented
9 years ago See official issue opened with Netflix: https://github.com/Netflix/ice/issues/171
Grid Billing Dashboard (Ice) Outage
Incident Report: 2015-05-04
Grid Billing Reports (emails) experienced a “bug” where expenditure was not being displayed to end users (zero values). This issue spawned debugging that lead to the discovery that the production Grid Billing Dashboard had ceased to update as seen below in comparison to staging:
Notification
Grid has no monitoring established for determining if the data displayed in Ice is accurate or up to date. Grid was not notified directly about this outage, rather notification came from an end-user about a dependent product (Grid Billing Reports).
History
In the past we’ve seen ice-processor (service/process) crash which causes “flat lining” on the dashboard (i.e. the date stays current, but shows zero value), and that may be easier to monitor by watching the process, however in this case the process was still running, but simply no longer functioning as expected.
Monitoring
Grid no longer has any sufficient monitoring for the Grid Billing Dashboard (Ice). Grid was using Boundary, but with the lack of automatic updates to alarms and dashboards the Boundary environment has gone stale. From Boundary we were monitoring ELB node health and using the HTTP Status plugin to monitor ice-processor, the service in Ice that failed in this outage, however the monitor wouldn’t have triggered in this scenario because the service was still running and responding.
Findings
Undetermined reason for failure. It appears that the ice-processor service/application had an internal error that caused it to stop processing AWS DBR data. The ice-processor process continued running both in us-west-1 and us-west-2 but remained non functional. For further detail see the Debugging section of this document.
Action Items
Resolution
Solving the outage
To solve the immediate issue, production servers were re-deployed. By re-deploying Grid Billing Dashboard (Ice) started processing data again and the May data showed up.
Long term monitoring solution
Cameron Stokes, Patrick Flaherty, and Chris Kelner met on 2015/05/06 @ 12pm: We decided that we would implement a roll up field asked by Landon Williams and monitor that field for $0.00 sum and send an alert email if it is $0.00 sum.
Original long term solution thoughts/ideas
Grid has read and been told several times over that “sometimes data gets wonky” with Ice. There hasn’t been any technical explanation as to why and we haven’t had the time or knowledge to understand it. Several users we’ve spoken to virtually and in person have suggest restarting Ice on a semi-regular basis as well as clearing the S3 bucket of any data and letting Ice rebuild itself. Grid doesn’t think that this is a “solution” but rather a band-aid, but perhaps it is time we give in and implement some sort of data dump and restart logic.
Monitoring the data is probably the best solution we can come up with. Ice-Miner already performs part of this job for us by requesting data from ice via HTTP requests to the Ice “API” (not really an API, but we treat it as such). See: https://github.com/TheWeatherCompany/ice-miner. Specifically we query Ice for data from the 1st of the month at midnight to the current date/time as seen here: https://github.com/TheWeatherCompany/ice-miner/blob/master/ice.py#L47-L50. Here we can see that Ice-Miner “detected” the outage and its logic could be used as part of a data monitoring solution:
Debugging
Checked to make sure ice-processor was still running:
Checked to make sure disk space was ok:
Last log messages were from four days prior to the incident on us-east-1. Normally activity is present in the logs every few hours as Netflix Ice processes new copies of the AWS DBR.
Additionally there was a May 03 2015 log on us-east-1, but the file was empty:
At this point, to no longer delay fixing production, new Grid Billing Dashboard (Ice) servers were deployed and the old servers preserved for continued debugging. The new servers worked fine without issue. Process info output for ice-reader:
Process info output for ice-processor:
Log files on the us-west-2 server were different for ice-processor:
This is the same NullPointerException we saw in the us-east-1 logs. Rather than nothing occurring after the NPE as it happened in us-east-1, it appears as if ice-processor continues to run the RI polling in us-west-2. Running jstack on the ice-processor process doesn’t reveal anything significant: US-EAST-1:
US-WEST-2:
Because the decommissioned (but still running) production servers and the new production servers share an s3 bucket, once the new servers started processing the Amazon DBR, the decommissioned servers process ice-reader was able to pick up the newly processed files out of S3 and display accurate data in the UI to the end-user even though the ice-reader processes were still in a funk state. See screenshot below of us-east-1 decommissioned server which before deploying new prod servers did not show May data is now showing May data:
At this point it simply looks like the ice-processor has some sort of internal error which we have no good way to account for other than attempting to monitor the data itself, see section Long term solution thoughts/ideas.
2015/05/06: Met with Cameron Stokes, outcome of meeting below: