Expire old notifiers

[xuset]

When a notifier (socket.io server) comes up, it adds itself to api as notifier and sets it's expire time via POST /notify/handler. It is important for notifiers to expire because that is how old notifiers (ones that were shutdown) get removed from the notifier list. Also notifiers should continually re-add themselves so the expire time get's updated, which is important to prevent the notifier from expiring while it is still operational.

• Periodically remove expired notifiers (should be done on the api side)
• Periodically update expire time by each notifier (should be done on the notifier side)

Optional: We cannot rely on any shutdown procedures to remove a notifier since SIGKILL could be sent at any time. However we can register a handler for SIGTERM and remove the notifier when SIGTERM is received. This will keep the notifier list up to date for most of the time and is a little cleaner than fully relying on expire times.

[georgeaf99]

I agree that this is better than what we currently do (clean up the notifier's state on restart).

Who is responsible for periodically cleaning expired notifiers? (periodic job on the API)

Also, we have to keep in mind that just because the notifier is expired doesn't mean it doesn't still have active connections.

[xuset]

Who is responsible for periodically cleaning expired notifiers? (periodic job on the API)

I'm not really sure what is the best way to handle this one. My favorite idea so far is whenever we iterate of the list of notifiers (during a transaction update broadcast or elsewhere) we just call the boto Item#delete() function on any expired notifiers. Alternatively a separate thread or cronjob could do the same.

If we set the 'refresh' rate and expire time far enough apart such that many refreshes occur within the expire delta then we can assume with reasonably certainty that any expired notify is no longer running, else it would have been sending periodic refreshes. The only counter example I can think of is if there was a network partition between the notifier and api on amazon's internal network. In this case we have much bigger problems.