search

cabrinoob / rancher-registrator

A registrator-like made for Consul & Rancher
15 stars 16 forks source link

Duplicated records in Consul when upgrading services #6

Closed am813nt closed 7 years ago

am813nt commented 7 years ago

With Rancher 1.5.1 duplicated service records appear in Consul after upgrading services. In stdout: Deregistering ERROR : No need to register this service

UPDATE: it happens when upgrade can't be finished or when container can't be started due to some errors.

makeacode commented 7 years ago

I've seen this same problem and have come up with a fix that i need to submit as a pull request. The problem I saw was a race condition where the docker event fires stating a container has been stopped but by the time the code tries to get the service (including port) information from rancher it no longer exists in rancher. I've change the code so that the rancher metadata query isn't necessary and it seems to be working well.

The only way i saw to fix consul is to manually deregister the services using consul api calls...to each agent. Kind of a pain...but doable.

cabrinoob commented 7 years ago

I have to admit that I did not test it with 1.5.1. As @makeacode is saying, the pb may be resolved by changing the deregistering workflow ...

am813nt commented 7 years ago

@makeacode, can you please provide the changes i need to make to fix that issue?

makeacode commented 7 years ago

I'll try to get the PR submitted today. It may be a little rough but it does seem to work. Submitted PR #7 with my changes.

cabrinoob commented 7 years ago

Nice job! I read it fast but it seems ok. I will try to test it ASAP (monday or tuesday) before merge it.

cabrinoob commented 7 years ago

7 Merged today. Thank you @makeacode