Right now state handoff isn't quite working right.
Whenever the application receives a SIGTERM message, it moves the state from each of the games into the state handoff CRDT. Because we're stopping multiple applications at the same time though, the dynamic supervisor is restarting their game processes. Sometimes the new restarted game processes are located on other nodes that are in the process of shutting down.
Somehow we need to implement connection draining to make sure that processes are always started on the nodes that are still going to be alive.
dnsbty
commented
3 years ago
Right now state handoff isn't quite working right.
Whenever the application receives a SIGTERM message, it moves the state from each of the games into the state handoff CRDT. Because we're stopping multiple applications at the same time though, the dynamic supervisor is restarting their game processes. Sometimes the new restarted game processes are located on other nodes that are in the process of shutting down.
Somehow we need to implement connection draining to make sure that processes are always started on the nodes that are still going to be alive.