Add Build Containers Build to New Relic

[amirbey]

User Story

Users should be able to build sites without backed up build queues due build container outage.

Background (Optional)

On 5/16 a build delay was reported and it was concluded that 5/6 build containers were down. The queue was backed up and it appeared that build were taking a long time to run.

Acceptance Criteria

• [x] Take build containers down and see if an alert is triggered via email

After evaluating, edit this part:

Level of effort - <low/medium/high>

Low

Implementation outline (if higher than "low" effort):

• [x] Setup NR Alert if the expected # of containers aren't in started state
• [x] Add instance state to Federalist Builder healtcheck

[amirbey]

Accomplished using Federalist Builder Alerts in New Relic ... there ... the alert did not contain any recipients and this has been resolved

[amirbey]

Check instance state instead of app state

https://gsa-tts.slack.com/archives/C09CR1Q9Z/p1527099621000530

[wslack]

Ok - so we need to do more work in the app or in New Relic? I think our acceptance criteria is off now.

[amirbey]

I regressed this after talking to Adam Kendall. Federalist Builder's healthcheck makes encompasses the health of the builder. However, after talking to Adam it appears we are not doing the most accurate healthcheck possible. We are checking the app state when we should be checking the instance state. That should have given us an alert on the outage ~ 5/16

[amirbey]

I think the acceptance criteria is the same but it requires tweaking federalist builder healthcheck

[wslack]

On hold given other work.