Starts to address issue #44. This is WIP until some testing done on staging
Compute units used by control must now come from Flight Hub
This PR adds mechanisms for requesting and sending compute units to Hub, via HTTP POST requests. Results are stored in FundsTransferRequest objects
Project budget logic is very significantly updated to be based on compute units actually received from Hub at the time, stored in Budget objects, rather than calculations made on demand
Balances are replaced by HubBalances, which are combined with the available compute units received by Hub to estimate when a project's total balance will be used up
Setup
flight_hub_url must be set in the environment's config file
Hub must contain a Department that corresponds to each control project (and the project alone)
The Hub Department's ID must be assigned to a project's flight_hub_id
Compute units must be added to the Hub Department before/at the start of the project
rails db:migrate must be run. Note: this is such a significant change that it is not possible to convert an existing DB to be logically compatible with this version. Manual (and difficult) editing or a fresh database will be required
run whenever -w or manually update the cron tab
Checking available Compute Units in Hub
Control now makes regular requests to Hub to check how many compute units each associated department has
These queries are made using HTTP GET requests, via the new FlightHubCommunicator service class
If the request is successful, and the amount has changed, a new HubBalance object is created
Slack messages are sent whenever a new Hub Balance is created, or if a query fails
These balances are used when calculating the cycle budget for a Dynamic and Continuous projects
They are also used as part of calculations when estimating when a project's total balance (compute units in hub + compute units in control) will be used up
These checks can be run using rake funds:check_balance:all or rake funds:check_balance:by_project[project_name]. This is scheduled by whenever to run for all active projects every hour.
Sending back unused compute units
For non continuous projects, at the start of each billing cycle (excluding the first), the number of remaining compute units at the end of the previous cycle is calculated
If any remaining, a request is made to send these back to the associated Hub Department. Requests are made using HTTP POST requests, via the new FlightHubCommunicator service class
All attempted requests are saved in a FundsTransferRequest object. The result, including any errors, is sent to the project's slack channel
Any compute units left at the end of the project are also sent back. This is the only time a continuous project sends back compute units
If a project has gone over budget (i.e. the remaining compute units is negative), no attempt is made to send/receive, but a warning slack message sent
Requesting cycle's compute units and creating Budgets: non continuous projects
current and historic cycles' budgets are now stored in a new Budget object, instead of calculating them when needed. These have an amount and both an effective_at and optional expiry_date, and are used when plotting costs
If no unexpired Budget can be found, the project will be treated as having a budget of 0
Before a Budget is created, the number of compute units required for this cycle is calculated and then must be obtained from the associated Hub Department
Requests are made using HTTP POST requests, via the new FlightHubCommunicator service class
All attempted requests are saved in a FundsTransferRequest object. The result, including any errors, is sent to the project's slack channel
If successful, a Budget is created for the requested amount, with an effective_at date of the start of the cycle, and an expiry_date of the start of the next billing cycle (or the project's end date, if it is earlier)
If the Hub Department does not have enough compute units, a warning slack message will be sent and any active nodes immediately switched off
Records in Hub:
Continuous Projects
For continuous projects compute units are not sent back or requested from Hub at the start of a new billing cycle
Instead, at the start of the project, all of the compute units available in hub are requested and used to create the project's Budget. This Budget is effective until the project's end date (if there is one)
If more compute units are added to the Hub department, once this is detected by Control (as part of the HubBalance recording process), they are immediately requested from Hub
A new Budget is created effective from that date, with an amount equal to the previous budget + the extra amount received
Funds Management
Checking balances, sending and receiving the necessary compute units is coordinated by a new FundsManager service class. This relies heavily on CostsPlotter when making calculations
Necessary actions can be run using rake funds:check_and_update_funds:all or rake funds:check_and_update_funds:by_project[project_name]. This is scheduled by whenever to run for all active projects every day at midnight
If a project is awaiting the generation of a new Budget, users will be prevented from viewing or taking any actions on that project. Instead they will be shown an explanation page asking them to try again in 5 minutes
This will be shown if the project doesn't yet have a Budget on its start date, its end date or at the start of a new billing cycle
Other Points of Note
These changes mean Control now has a hard dependency on Hub. Control can no longer be run in isolation, and issues with Hub or failed requests to Hub will have very significant negative impacts on control
Similarly if the Hub API changes, or its URL changes, this will necessitate changes in Control
This entire request and budget process is complex, especially having to take the 4 different possible budget policy types into account. This would greatly benefit from some automated testing (which will be a v large piece of work), and/or a reduction in the number of policy types
It is very likely that there are edge cases not considered
If the sending/receiving/budget process breaks, this will likely require manual intervention in the db to resolve. This could be especially painful operationally, given budget actions take place around midnight
This PR does not include showing FundsTransferRequests in the project audit logs
Flight Hub currently has no authentication for these compute unit requests. Once this is implemented it will need to incorporated here
As an alternative to using HTTP requests, perhaps in the future we could communicate between Control and Hub (and Flight Center and Hub) using a message queue like RabbitMQ. This would require significant work to integrate with the 3 applications, but reduce some of the dependencies/ reduce some brittleness, and simplify authentication (just need authentication for the queue)
This PR includes improvements to the get_project logic, with this now a before_action in the application's relevant controllers
Starts to address issue #44. This is WIP until some testing done on staging
FundsTransferRequestobjectsBudgetobjects, rather than calculations made on demandBalancesare replaced byHubBalances, which are combined with the available compute units received by Hub to estimate when a project's total balance will be used upSetup
flight_hub_urlmust be set in the environment's config fileflight_hub_idrails db:migratemust be run. Note: this is such a significant change that it is not possible to convert an existing DB to be logically compatible with this version. Manual (and difficult) editing or a fresh database will be requiredwhenever -wor manually update the cron tabChecking available Compute Units in Hub
FlightHubCommunicatorservice classHubBalanceobject is createdrake funds:check_balance:allorrake funds:check_balance:by_project[project_name]. This is scheduled bywheneverto run for all active projects every hour.Sending back unused compute units
FlightHubCommunicatorservice classFundsTransferRequestobject. The result, including any errors, is sent to the project's slack channelRequesting cycle's compute units and creating Budgets: non continuous projects
Budgetobject, instead of calculating them when needed. These have anamountand both aneffective_atand optionalexpiry_date, and are used when plotting costsBudgetcan be found, the project will be treated as having a budget of 0Budgetis created, the number of compute units required for this cycle is calculated and then must be obtained from the associated Hub DepartmentFlightHubCommunicatorservice classFundsTransferRequestobject. The result, including any errors, is sent to the project's slack channelBudgetis created for the requested amount, with aneffective_atdate of the start of the cycle, and anexpiry_dateof the start of the next billing cycle (or the project's end date, if it is earlier)Records in Hub:
Continuous Projects
HubBalancerecording process), they are immediately requested from HubFunds Management
FundsManagerservice class. This relies heavily onCostsPlotterwhen making calculationsrake funds:check_and_update_funds:allorrake funds:check_and_update_funds:by_project[project_name]. This is scheduled bywheneverto run for all active projects every day at midnightPending Budget Update Page
Budget, users will be prevented from viewing or taking any actions on that project. Instead they will be shown an explanation page asking them to try again in 5 minutesOther Points of Note
FundsTransferRequestsin the project audit logsget_projectlogic, with this now abefore_actionin the application's relevant controllers