shlomi-noach
commented
6 years ago Thank you for opening this Issue. While I have no immediate insight into what went wrong:
...One of the slaves was a delayed replica with a configured 24-hour delay.
It sounds as if this should be the smoking gun. However, orchestrator specifically targets delayed replicas in a special way and relocates them offline:
From the screenshot you posted it seems there were no postponed functions, which means the delayed replica was not run asynchronously.
Which reminds me I added a newer logic to relocate replicas and wanted to retire the older logic (right now configurable by config.Config.PseudoGTIDPreferIndependentMultiMatch).
Please confirm you're using PseudoGTIDPreferIndependentMultiMatch = false (or none at all, as I suspect), which means you're using the "older" method. Nothing bad about it, just so that I know where to look.
...could have an extra parameter, topologyRecovery, which can be used for optional audit logging if topologyRecovery != nil? This would allow the same routines to be used for normal relocation purposes without special logging, but under recovery processing extra logging could be emitted. Does this sound like a reasonable idea?
Absolutely, this would be the way to go in temrs of visibility/logging.
sjmudd
bbuchalter
Background
Partial Screenshot from
/web/audit-recovery-steps/A recent failover took 8 minutes to run, which is "quite a long time". 32 slaves were involved. One of the slaves was a delayed replica with a configured 24-hour delay. Orchestrator was configured on this chain to use pseudo-gtid.
FR: Improve audit logging to provide more detail
During those 8 minutes there was no visible signs of what was going on.
/web/audit-recovery-steps/output to provide more detail than currently shown. Some of that is in the debug logging but having it in the audit logging does no harm. (we’re doing a failover after all, so logging things that are not done is also fine)FR: Come up with a better way to follow what's going on
To find the detail of what was going on you could find the active orchestrator instance and look at the log files. They are quite detailed. However, for me this is quite unworkable. The installation I manage has over 1000 hosts and with instance discovery taking place every 5 seconds there is a lot of “noise” in the logging during this 8 minute period. Better visibility or what orchestrator is doing during recovery would be good.
Auditin the GUI which takes us directly to the last recovery process so we can see the steps that have been executed so far. A simple page refresh would allow us to follow changes. So while rather "brute force" it would allow me to quickly jump to and see “what orchestrator is doing from the GUI.Audit->Latest Recovery StepsWould this be possible?
Improve logging of pseudo-GTID relocations as they may be slow.
Given the issue I saw it looks like calls to
RegroupReplicasPseudoGTIDIncludingSubReplicasOfBinlogServersand similar calls could have an extra parameter,topologyRecovery, which can be used for optional audit logging iftopologyRecovery != nil? This would allow the same routines to be used for normal relocation purposes without special logging, but under recovery processing extra logging could be emitted. Does this sound like a reasonable idea?The comments here are to reference some issues seen and some thoughts on possible workarounds or improvements. Thoughts from you on this would also be most welcome.
I m not sure if I will have time now to look at this but thought it would be good to reference the issue so it does not get forgotten.