search

alphagov / govuk-infrastructure

Terraform turnup automation for the EKS Kubernetes clusters that host GOV.UK. See https://github.com/alphagov/govuk-helm-charts for application config.
MIT License
145 stars 25 forks source link

The way we're defining tfvars appears to defeat TFC's cost estimation. #1317

Closed sengi closed 1 month ago

sengi commented 6 months ago

Most of our instance counts and sizes etc. are specified in the tfc-configuration module, which then configures the vars in the TFC workspaces that run the root modules that actually manage stuff in AWS.

This appears to defeat Terraform Cloud's cost estimation feature on the very PRs where it would be most useful.

For example this speculative plan for https://github.com/alphagov/govuk-infrastructure/pull/1316 shows this, despite actually costing thousands:

Plan finished

Resources: 0 to add, 2 to change, 0 to destroy

Cost estimation finished

Resources: 0 of 159 estimated $0.00/mo +$0.00

Too many resources to display

Due to the large number of resources (159), we cannot show them in the browser.

sengi commented 6 months ago

Maybe just a bug to raise with Hashicorp/IBM tbh. Unless there's some better way we can do this configuration (but I'm pretty sure we're already doing it the way they recommended!)

(edit: IBM sorry, I'm getting mixed up between all the recent corp OSS acquisitions 🙈)

nimalank7 commented 1 month ago

Odd as it sounds the HCP cost estimation is working correctly. According to HCP documentation these are the supported AWS resources that trigger cost estimation. Of these I focused on those which incurred costs and I searched through govuk-infrastructure to see where they were used.

Resource Runs
aws_cloudwatch_metric_alarm https://app.terraform.io/app/govuk/workspaces/rds-integration/runs/run-K2kpaZur8CotPboU
aws_db_instance https://app.terraform.io/app/govuk/workspaces/rds-integration/runs/run-K2kpaZur8CotPboU
aws_elasticache_cluster https://app.terraform.io/app/govuk/workspaces/govuk-publishing-infrastructure-integration/runs/run-AkiFHMTHiLJR5oPi
aws_kms_key ecr, cluster-infrastructure, mobile-backend

Looking at the runs you can see that cost estimation is working for these. In the above example open search isn't part of the supported AWS resources so no cost estimation is run. Another oddity is that though we spin up load balancers these won't come under cost estimation as they are managed by the AWS ALB controller whose terraform resource isn't a part of the supported list.

The reason why lots of runs don't have HCP cost estimation is because most of the resources touched aren't part of the supported AWS resources that incur costs. From this it sounds like the HCP cost estimation is of limited use. (Perhaps a better way would be to have something poll the AWS Billing API after a Terraform plan run)