CloudCost Exporter needs to implement a module within aws that performs the following:
Generates a pricing map for ec2 instances
Collects a list of running ec2 instances within an AWS profile
Implements a collector that exports the hourly cost of a machine
There was a proof of concept done many months ago: https://github.com/grafana/deployment_tools/pull/96916
Note that the project structure has been completely refactored so you can't use it as is.
However the api endpoints and clients could be useful for helping bootstrap the issue.
Pricing Map
One major deviation from other Cloud Providers is that AWS only provides pricing by instance type.
The implication is that we need to make an estimate of how much an instance costs by CPU and Memory.
One way we can do this is by looking at GCP prices and finding the ratio of CPU:Memory costs.
Unknowns:
Can we pull spot pricing information from api?
Some considerations:
Cache the pricing map for a specific period of time
Can we infer the regions the profile is running in and limit the amount of skus to parse?
Collecting a list of running ec2 instances
In order to export metrics, we need to list of each of the ec2 instances that are running within a project/profile.
We need to keep track of the following bits of metadata:
instance name
region it's running in
family of the machine(T2, M2, etc)
machine type(m7g.medium, m5.large)
Number of Cores
GiB of ram
profile/project associated with the instance
OnDemand or a spot instance
Some considerations:
Performance here will be key, the list of instances needs to be collected in under 60 seconds
We can use a cache and be okay with the current exported metrics to be from a previous pull
Parallize the requests by regions/zones if possible
Export Metrics
Use the pricing map and ec2 instances to export the following metrics:
Overview
CloudCost Exporter needs to implement a module within aws that performs the following:
There was a proof of concept done many months ago: https://github.com/grafana/deployment_tools/pull/96916 Note that the project structure has been completely refactored so you can't use it as is. However the api endpoints and clients could be useful for helping bootstrap the issue.
Pricing Map
One major deviation from other Cloud Providers is that AWS only provides pricing by instance type. The implication is that we need to make an estimate of how much an instance costs by CPU and Memory. One way we can do this is by looking at GCP prices and finding the ratio of CPU:Memory costs.
Unknowns:
Some considerations:
Collecting a list of running ec2 instances
In order to export metrics, we need to list of each of the ec2 instances that are running within a project/profile. We need to keep track of the following bits of metadata:
Some considerations:
Export Metrics
Use the pricing map and ec2 instances to export the following metrics:
cloudcost_aws_ec2_instance_cpu_usd_per_core_hourcloudcost_aws_ec2_instance_cpu_usd_per_gibyte_hourBoth metrics should have the following labels