I've managed to resolve the issue, but just wanted to document it here.
We were running Bitpoke mysql v0.6.2. GKE auto-upgraded our k8s cluster from v1.24 to v1.25 while we weren't looking.
The databases went down, and then back up promptly once we had upgraded the operator to v0.6.3. However all of the databases were stuck in read-only mode.
After checking all the logs (The orchestrator had the offending log lines), I found the service for the operators orchestrator had an additional label (The label was "app: mysql") that wasn't found on the pods. So the service didn't have any endpoints defined.
Removing the additional label resolved the databases to read-write capable almost immediately.
I've managed to resolve the issue, but just wanted to document it here.
We were running Bitpoke mysql v0.6.2. GKE auto-upgraded our k8s cluster from v1.24 to v1.25 while we weren't looking. The databases went down, and then back up promptly once we had upgraded the operator to v0.6.3. However all of the databases were stuck in read-only mode.
After checking all the logs (The orchestrator had the offending log lines), I found the service for the operators orchestrator had an additional label (The label was "app: mysql") that wasn't found on the pods. So the service didn't have any endpoints defined. Removing the additional label resolved the databases to read-write capable almost immediately.