Currently instance prices and sizes (mem, CPU and GPU) are queried from AWS and Azure each each night and recorded in text files (lib/platform_files), as the queries are very slow to complete. These files are read once and stored in memory (see Instance class method set_instance_details) when the application first starts. This means changes are not applied until the application restarts.
This logic is carried over from the original cloud cost visualiser, which recorded hundreds of different instance types (so the text files were very large), was only used to show forecasted costs, and had no issue being regularly restarted.
Flight Control now has much more sophisticated logic, so only instance prices and sizes we are currently using are recorded, and restarting regularly is not appropriate. We should therefore move to storing these instance details in the database.
This will involve:
Creating a migration that creates a new table, e.g. instance_type_details. It will need attributes for instance_type, region, price_per_hour, cpu, gpu, mem. currency might also be useful.
Will probably want indexes on region and instance_type, as these the attributes searches will be on
Create a new model relating to this
Records should be unique for their region and instance type combination
Updating the logic in the services AwsInstanceDetailsRecorder and AzureInstanceDetailsRecorder to record these in the db, instead of text files
For Azure, prices and sizes are queried separately, so you will either need to create and then update records, or collate the data together and then create them. It's better to record some of the data than none (e.g. if one of the two queries fails)
Records should be created if no existing matching record, or updated if details have changed
The logic in Instance for reading the current text files will need to be removed, as will the text files themselves. The two region_names files will need to remain
All uses of Instance.instance_details will need to be replaced by suitable calls to the database. At a minimum this will require updating Instance and InstanceLog
It would also be good to have a flash message/ some other alerting mechanism highlighting if costs/sizing details are missing for an instance type in use
Currently instance prices and sizes (mem, CPU and GPU) are queried from AWS and Azure each each night and recorded in text files (
lib/platform_files), as the queries are very slow to complete. These files are read once and stored in memory (seeInstanceclass methodset_instance_details) when the application first starts. This means changes are not applied until the application restarts.This logic is carried over from the original cloud cost visualiser, which recorded hundreds of different instance types (so the text files were very large), was only used to show forecasted costs, and had no issue being regularly restarted.
Flight Control now has much more sophisticated logic, so only instance prices and sizes we are currently using are recorded, and restarting regularly is not appropriate. We should therefore move to storing these instance details in the database.
This will involve:
instance_type_details. It will need attributes forinstance_type,region,price_per_hour,cpu,gpu,mem.currencymight also be useful.regionandinstance_type, as these the attributes searches will be onAwsInstanceDetailsRecorderandAzureInstanceDetailsRecorderto record these in the db, instead of text filesInstancefor reading the current text files will need to be removed, as will the text files themselves. The tworegion_namesfiles will need to remainInstance.instance_detailswill need to be replaced by suitable calls to the database. At a minimum this will require updatingInstanceandInstanceLog