sybenzvi
commented
2 months ago nightwatch.desi.lbl.gov is live again.
Unlike #371, no configuration update was needed, just the termination of a paused pod in Workloads>Deployments>nightwatch>prod.
Presumably what happened is that after the perlmutter engineering work on 9/11 and 9/12, the file system came back online in an order that caused the pod to get stuck in a restart loop.
nightwatch.lbl.desi.gov is spitting out a "503 Service Temporarily Unavailable" warning. The pod configuration may need to be updated, similar to issue #371. We'll try to get it restarted this morning.