Budget Matching Switch Offs

[timalces]

Aims to resolve #30

• The system will now calculate recommendations of instances to switch off, to help prevent a project going over budget this cycle
• Instances are chosen to switch off based on their weighted priority: their compute group's priority * their instance type's priority (as defined in the project's config file). Instances with a lower weighted priority are chosen for switch off first
• Includes logic to minimise the number of switch offs, and to maximise the available budget. For example, if a more expensive, higher priority instance is switched off and that leaves enough remaining budget, budget switch offs for cheaper, lower priority instances will be delayed/ cancelled (if possible)
• These calculations assume switch offs will be carried out at 11:30pm on the given date
• These switch offs are shown in the costs breakdown charts, and as part of the costs forecasts when creating a request:

• When a change request is submitted, any currently calculated switch offs are included in the resulting slack message:

• As a significant imporvement to the version in cloud-cost-visualiser, instances can be switched off multiple times during a cycle (as many times as required), if they are scheduled to be switched back on by a change request. Switching low priority instances off multiple times is favoured by the calculations (to avoid higher priority switch offs, where possible).
• If there are many instances and many planned switch ons, this means there can be a large number of budget switch offs planned for a cycle:

• These switch offs are also calculated for future months, and when viewing partials of a cycle:

• Balance end and over budget calculations now take these switch offs into consideration. including on the billing management page
• Budget switch offs are included in the Manage Events page, highlighted in red:

Carrying out switch offs & reporting

• Automated actioning of the budget matching switch offs can be carried out using the rake task rake projects:budget_switch_offs:all[slack,text] or rake projects:budget_switch_offs:by_project[project_name,slack,text]

• An entry is included in config/schedule for this to run for all active projects every night at 11:30pm (which can be set using whenever -w)

• When budget switch offs are carried out, associated ActionLogs are created

• A rake task is also added to report (without actioning) any upcoming budget switch offs, using rake projects:budget_switch_off_schedule:all[slack,text] or rake projects:budget_switch_off_schedule:by_project[project_name,slack,text].

• An cron entry is also included to run this for all active projects at 12:30pm each day

Note: This is a large and complex change, with significant impact on cost calculations. Although manually tested, there may be issues missed for specific combinations of instance type, counts, date ranges and/or budgets

• Despite noticeable improvements compared to the visualiser project, these calculations may become slow if many instances and change requests. If this becomes an issue, apart from examining logic improvements, perhaps the costs charts should load asynchronously from the main costs breakdown page (for example as Azure does in its portal)
• Similarly, during testing with an extreme number of switch offs (multiple per day, every day), displaying them all on the charts becomes impossible. If this is found to be a realistic scenario, we will need to examine alternative ways of displaying such a large number of switch offs
• This is not a full rewrite, but an adaptation and significant improvement of the functionality originally developed in visualiser. If additional development time becomes available this may benefit from further refactoring, especially to improve readability given its complexity. It would also benefit from automated testing, which would in itself be a significant piece of work to add, given the large number of possible scenarios