search

autopilotpattern / mysql

Implementation of the autopilot pattern for MySQL
Mozilla Public License 2.0
172 stars 68 forks source link

Failover lock in Consul isn't being cleared after failover #67

Closed tgross closed 8 years ago

tgross commented 8 years ago

After failover, I'm seeing the following in the logs:

failover session lock ({session ID}) not removed because primary has not reported as healthy'

This means the FAILOVER_IN_PROGRESS key isn't getting removed. The primary is marked as healthy however, so this error may be the result of some improper exception handling that's masking some other issue.

tgross commented 8 years ago

It looks like the mysql-backup-running lock isn't being removed either, so that may be related.

tgross commented 8 years ago

I've mischaracterized this and it's not a bug. We clear the lock but we don't remove the key. Which is not only fine but probably preferred so that an operator can see at a glance in Consul which instance had the lock last. Closing.

tgross commented 8 years ago

Oops, still have the issue of the log events though! So lemme verify that's all cleared up here before closing.

tgross commented 8 years ago

Ok, I can confirm that when we're running https://github.com/autopilotpattern/mysql/pull/61 this is fixed. I'll mark this as open PR rather than closing.

tgross commented 8 years ago

Released in https://github.com/autopilotpattern/mysql/releases/tag/5.6r3.1.0