search

ausocean / cloud

GNU General Public License v3.0
1 stars 1 forks source link

Slate on hardware failure for permanent live states #179

Closed saxon-milton closed 5 days ago

saxon-milton commented 6 days ago

closes #54

We're now handling hardware failure events when we're in the permanent live/live unhealthy and slate to live states. Under these circumstances, we should stay in slate. We can do this by entering the vidforwardPermanentInFailure state in which case we wait until admin has taken the broadcast out of this state i.e. maybe fixed the problem.

saxon-milton commented 6 days ago

Is a manual (operator) intervention required to exit the vidforwardPermanentInFailure state? If so, how will the operator know?

Yes, when we enter this state there will be a notification email. The operator would then have to investigate the issue and try to ameliorate it (or wait until hardware is healthy i.e. sufficient voltage). Once hardware is healthy, we can take it out of this state. One step beyond this would be to periodically check hardware health and capacity to stream, under which conditions we can automatically go back to whatever state we should be in.