investigate scheduler approaches

[bengreenier]

We desire a way to schedule chaotic events via a hosted service, to remove the dependency on manual intervention as currently exists with https://github.com/Azure/platform-chaos-cli.

This tracks the work we need to do to identify if there is a possible existing solution out there that may be coupled with the work we've already done to provide these features.

Current possibilities include:

• Azure Scheduler
• Azure Logic apps
• Lightweight azure function (timer trigger)

[bengreenier]

depends on #16 - as we'll definitely want to call out to that api from whatever tool or solution we define for scheduling

[mindlessroman]

Starting thoughts

Azure Scheduler

• "run jobs [...] on any schedule, making it ideal for recurring actions like cleaning up logs, kicking off backups, and other maintenance tasks"
• Price: $13.99 per unit / per month
• Unlimited number of executions
• Max run frequency is once a minute
• Lots of customization how a job is run and good UI - once v recurring
• Scheduler has history/logging
• Getting Started docs
• API Docs

Logic Apps

• Out of the box connections to a lot of kinds of services outside Azure
• Pricing per Action/Connection - ($0.000025, $0.000125 respectively)
• Getting Started docs
• Example Tutorial with scheduled task - decision trees

Azure Functions

• Available free tier - up to 1million executions
• Execution time: 400,000 GB-s
• Lots of language support
• General Documentation
• Timer tutorial as an Azure Function

Azure Batch

• "You don’t need to write your own work queue, dispatcher, or monitor. Batch gives you this as a service."
• Pricing
• General Docs

Jenkins

• Not Azure-exclusive, but is supported
• Maybe more configuration-heavy and not as CLI friendly under the hood?
• Docs

[bengreenier]

thanks for including those @mindlessroman! I'd be interested to hear what you recommend we use after doing some of that research.

[mindlessroman]

Discussed this a lot more in person during standup - the current front runner is Azure Scheduler. But some more questions/thoughts were raised:

How does each platform handle providing a level of randomness? What has a good/the best UI for a partner/user to configure? Scheduler - what does the max frequency entail?

[mindlessroman]

Scheduler

Randomness:

Trying to instantiate some chaos within a scheduled window would likely require some more coding gymnastics - doesn't seem to be built in. Probably not impossible 😅 More thinking would need to be done, but one thought: using an azure function (or something similar) to generate a time in specified window, then inputting that into the scheduler.

UI

Creating a quick scheduler was fairly simple and there are some user-configurable fields.

UI for timing (one-time v recurring) is pretty self-explanatory.

History

Doesn't mention who triggered it, but does manage history by scheduled task (in this example hktestingscheduler).

Frequency

The docs mention the limit on frequency. There will be a 409 error code if one tries to manually make a frequency that's higher than once a minute.

Conclusion

May be a better choice for chaos that occurs at predictable, settable times. ie chaotic actions with 'set' timing, not chaotically timed so-to-speak. But may be able to hack a solution for more time-chaos.

[mindlessroman]

Logic Apps

Randomness

Riffing on the RSS checker example - you could potentially set up an API that generates a time in the specific window and have a logic app listen/check in every X amount of time and then have it trigger the chaotic event when the generated time and the current time are the same, +/- 1 minute. Otherwise seems to be in the same area of "nothing out of the box" for chaotically timed things. Logic App switch statement may be less ideal as a solution but is available.

UI

The UI is super visual - I believe this would translate well to a user since it's so configurable. The branching of what to do and when takes a lot of coding off the user's plate. It relies on value fields filled in by the user around conditions framed largely using plain English. (There is a JSON-style code editor that can be used as well, if the user wanted.) A path on the "true" path in the example logic apps designer. Supports using a table to fill in the key-value pairs, or the use of a JSON-styled key-value associations.

History

You can use the logging analytics. Not user specific but tracks a number of other details. The tracked properties mentioned here strikes me a useful feature.
If a user wanted to customize a bit more for logging - maybe to a SQL DB - then one could construct a stored procedure to add some.

Conclusion

Seems to be a flexible option for the chaotic timing. Good if the plan were to be more complex decision paths based on what's returned in a chaotic event. (Below, a not very complicated decision tree of nested conditions) Has a scope option which may be useful to have in the toolbox.

[mindlessroman]

At this point, there doesn't seem to be one of these that offers a single solution to the problem we're hoping to address. In that case, we'll probably need have a blended solution that includes a few of these. Logic Apps definitely wins with the UI - very configurable, layman-readable and is the leading candidate in the Azure-generated solutions for something that is timed chaotically. Scheduler would be a good candidate for the predictably timed chaos, but with slightly less layman-readable UI. The UI's still very configurable, but it lacks the same visual engagement/flow that Logic Apps has.

Azure Functions would likely need to be used in our solution for generating a "random" time in a specified window.

[mindlessroman]

We had another chat last week. Using Logic Apps may be over engineering the problem - the leading solution is to invoke the chaos via CLI and take care of the randomly generated time that way. Scheduler itself doesn't care how a job is scheduled, so we should randomize at the layer ahead of Scheduler.