search

mdevilliers / redishappy

Redis Sentinel high availabillity daemon
Apache License 2.0
114 stars 23 forks source link

No ability to retry failed attempts to update Consul #38

Open dominics opened 9 years ago

dominics commented 9 years ago

Currently, redishappy can fail to update Consul because it's not running:

INFO: 2015/08/18 00:42:46 consul_flipperclient.go:38: InitialiseRunningState called : [{"externalPort":6379,"name":"redis-default","ip":"1.2.3.4","port":6379}]
ERROR: 2015/08/18 00:42:46 consul_flipperclient.go:82: Error updating consul : Put http://127.0.0.1:8500/v1/catalog/register: dial tcp 127.0.0.1:8500: connection refused

OK, I thought, I'll just add some ordering in the upstart scripts so that we start redishappy just after Consul has started. However, Consul takes a few seconds to get going, and in the meantime:

INFO: 2015/08/18 00:45:03 consul_flipperclient.go:38: InitialiseRunningState called : [{"externalPort":6379,"name":"redis-default","ip":"10.10.10.5","port":6379}]
ERROR: 2015/08/18 00:45:03 consul_flipperclient.go:82: Error updating consul : Unexpected response code: 500 (No cluster leader)

In both cases, unless the Redis topography actually changes after this point, redishappy won't make any further attempt to update Consul. I guess I'm looking for some sort of retry mechanism. Or, even better, if redishappy were to "ensure" the value in Consul by reattempting updates whenever the service-state-in-Consul was different from expected, or the state couldn't be checked.

dominics commented 9 years ago

Feel free to mark this as an enhancement :D