superfly / corrosion

Gossip-based service discovery (and more) for large distributed systems.
https://superfly.github.io/corrosion/
Apache License 2.0
684 stars 21 forks source link

Retry buffered changes application #127

Open jeromegn opened 9 months ago

jeromegn commented 9 months ago

Sometimes we get an error applying partially buffered changes because there's a schema change that doesn't fit or some other random retryable error.

Instead of forgetting about the buffered change (Corrosion only forgets in-memory, still kept in the DB), we should re-queue them somewhere to try again later. For schema errors like constraint failed, we could retry applying buffered changes when there's a new schema applied.

For other errors, maybe just a short timer.