search

outbrain / orchestrator

MySQL replication topology manager/visualizer
Other
829 stars 168 forks source link

Clear failover analysis? #273

Closed mysqldesu closed 7 years ago

mysqldesu commented 7 years ago

I performed a graceful-master-takeover and everything was successful. Orchestrator continues to show the cluster in Failure Analysis. How do I make that cluster no longer appear in the the failover analysis?

I've tried using ack-cluster-recoveries and it went away temporarily but then returned. I tried to "forget" all the instances and "discover" them again but orchestrator continues to ask if I want to do a recovery.

I was able to get the recovery message to go away by flushing logs, re-creating the pseudo gtid table and moving a slave around in the topology. Is there some other option to clear it?

shlomi-noach commented 7 years ago

Orchestrator continues to show the cluster in Failure Analysis

What was the failure analysis for this cluster?

mysqldesu commented 7 years ago

Thank you for response. I thought there was only one possible failure analysis so I didn't post it here. I don't have the exact output anymore. It seems to be have been a fluke because I have done several more graceful-master-takeover and the problem has not resurfaced. I will close this issue for now. If I can re-produce the problem, I'll post the the exact failure analysis and re-open the issue.

shlomi-noach commented 7 years ago

You should still be able to see it via the web interface: Audit -> Failure detection

mysqldesu commented 7 years ago

"AllMasterSlavesStale" is the message that shows up when this happens. I wasn't able to find it under the Audit page though.