Event-driven TES

[BMurri]

Problem: Polling for changes in state isn't scaling well, especially when running into API rate limits.

Solution: Subscribe to changes for all Azure services used in the implementation of TES. Use Queues. Add adapters as needed.

Describe alternatives you've considered

• Subscribing to only Batch Account events.
• Changing the polling intervals.
• Staggering polling of items not recently polled.

Additional context

• There will be sub-tasks for each Azure service written soon to describe proposed implementation in detail.
• TES deployments that do NOT run the scheduler will continue retrieving state from the database.

[BMurri]

Consider Storage Queue (see #404)

[ngambani]

@BMurri is this an active issue?

[BMurri]

Yes. This is the one I want to ship in February. This should fix both #119 and #404 as well as reduce stress on the Batch APIs