At some point over the holiday weekend, the redis instance was restarted, likely by Google Cloud operations, and the instance state was cleared. This wiped out some configuration like the connection password. In result, the service for pushing results back to the Compute Studio webapp was down. I reset the password on the redis instance as a temporary measure and notified the affected users.
The redis instance is supposed to write its state to a compute engine disk. This way data isn't lost if the service is restarted. The kubernetes logs show that the disk is mounted correctly. But, for some reason, it either isn't saving to disk or picking its state back up from disk properly. I will spend more time on this underlying issue later today.
hdoupe
commented
3 years ago
At some point over the holiday weekend, the redis instance was restarted, likely by Google Cloud operations, and the instance state was cleared. This wiped out some configuration like the connection password. In result, the service for pushing results back to the Compute Studio webapp was down. I reset the password on the redis instance as a temporary measure and notified the affected users.
The redis instance is supposed to write its state to a compute engine disk. This way data isn't lost if the service is restarted. The kubernetes logs show that the disk is mounted correctly. But, for some reason, it either isn't saving to disk or picking its state back up from disk properly. I will spend more time on this underlying issue later today.