What this PR does:
Currently on shutdown the generators set a flag to stop receiving traffic and then remove themselves from the ring. This causes distributors to continue to send to a generator that is refusing traffic for a few seconds.
This PR:
1) Correctly drops out of the ring first and then sets the flag to stop ingestion
2) Adds a sleep in between these steps. Tempo by default uses memberlist for ring propagation which often requires a few seconds to propagate state from one component to another. Without this sleep the change was ineffective. I do not like the sleep but I'm not sure there's a better solution.
To the left of the red line is what a generator rollout looks like today. To the right is what this change looks like. Note that there are still a few drops but it's significantly reduced.
Checklist
[ ] Tests updated
[ ] Documentation added
[x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
What this PR does: Currently on shutdown the generators set a flag to stop receiving traffic and then remove themselves from the ring. This causes distributors to continue to send to a generator that is refusing traffic for a few seconds.
This PR: 1) Correctly drops out of the ring first and then sets the flag to stop ingestion 2) Adds a sleep in between these steps. Tempo by default uses memberlist for ring propagation which often requires a few seconds to propagate state from one component to another. Without this sleep the change was ineffective. I do not like the sleep but I'm not sure there's a better solution.
To the left of the red line is what a generator rollout looks like today. To the right is what this change looks like. Note that there are still a few drops but it's significantly reduced.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]