Very high monthly costs (+USD 800) for extreme high hours of ECS vCPU/Memory

[TreesforFood]

Dear development team,

I installed the Superset Partner Solution. However, from the moment I installed it, the application used an extreme high number of hours of memory/vCPU (+ 37,377 versus 10,000 hours) for ECS Fargate. I connected Superset with a Postgres database on Heroku, but just for testing the connection, nothing else (Superset was not used yet in production, doing queries or creating visualization traffic). So, basically, I just installed Superset, connected to a database and left it as it is (no queries or traffic of what so ever). And that created a AWS bill of USD +800 per month.... I asked AWS but they do not responded very informative. Could it be the installation settings that is causing the issue? Any help is appreaciated, maybe other encounter the same issues with ECS and this installation.

[dreilly-icf]

We're getting started here and taking your comment seriously. Did you ever figure this out?

[TreesforFood]

We're getting started here and taking your comment seriously. Did you ever figure this out?

Thanks for responding. No. Never. I searched everywhere, but as I am unfamiliar with AWS it was a total blackbox to me. I assume it has to do with CPU usage, but I was not able to detect where I could modify it and how to modify it. Whatever I switched off, the monthly AWS bill remained extremely high. In the end, I just stopped my account, too afraid to receive another +800 USD bill. Hope you can find the solution because it worked smoothly and the installation was not that difficult (except from some tweekings that I had to find out by trial and error).

[shivam01anand]

Thanks @TreesforFood for taking the bullet and informing. It would be great to have estimates from the authors or practitioners on cost benchmarks of the solution. @dreilly-icf What was the solution that you went for?

[dreilly-icf]

Hi @shivam01anand and @TreesforFood -

We are still in the prototyping phase for our implementation here. We are also using terraform to create the stack, allowing us to pass in existing arn id's and other resources to reduce duplication.

With that being said, there are some common-sense updates we made to the cloudformation template that included extracting:

1. All environment variables to parameters,
2. EC2 size values into parameters
3. ContainerUrl to a parameter
4. CreateCloudwatchDashboard to a parameter ("no", "yes")
5. Passing SupersetEnvironment (production/development) - This may actually result in cost-savings since the SupersetNodeTask is only created for the "development" build

We are currently running the regular deployment ("development") and overall, the price with minimal/no-usage is estimated around $18/day for the stack.

There's a handful of other parameters I've yet to extract, but those would likely be the vMem, vCPU values. If we can figure out what the minimum specifications are for each container, tweaking those to small/medium/large could net gain. I can share any updates if we make any progress, but that may not be for a month or so.

[cj-aki-riisio]

I was deploying the partner solution for a side project and yes, the expenses went through the roof. I have similar estimates than @dreilly-icf, $18 per day. This was not sustainable for a side project so now I'm going with typical EC2 approach. EDIT: I actually downloaded the docker image from the partner solution registry and then applied my own configs with superset_config.py and custom logos as well with a custom Dockerfile. I then published this to ECR and modified the existing CFN template to point to the ECR instead of the public registry. Such a shame, this solution worked beautifully.

[myyk]

Wow, thank you for pointing this out! I only just launched this stack and while waiting for it to come up saw this. I tried to understand the costs and this compute is probably the bulk of it. It launches all the ECS compute with Fargate so we can use their calculator to understand the costs: https://calculator.aws/#/createCalculator/Fargate

We can find the launch configurations in https://github.com/aws-ia/cfn-ps-apache-superset/blob/main/templates/superset-new-vpc-workload.template.yaml and look at the AWS::ECS::TaskDefinition resources defined in https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html

From there I just plugged the numbers into the calculator for us-east-1. It calculated the costs for these resources as such:

(db + redis + superset + superset-init-cleanup + superset-node + superset-worker + superset-worker-beat)

$(9.01 + 9.01 + 170.13 + 18.03 + 144.17 + 170.13 + 170.13) = $690.61

I think superset-init-cleanup was off after the stack started up, so I think maybe you could reduce that number by $18.

The larger Fargate 4vCPU + 16 GiB Mem could maybe be replaced with similarly sized EC2 instances such as t4g.xlarge. That would be $98.11/mo On-Demand on the high end and easily lowered to $61.54 with 1 year EC2 Instance Savings Plan. (There are other plans to get it even lower.)

I'm not sure if we can 'simply' switch the AWS::ECS::TaskDefinitions to use those similarly sized EC2 instances, but that could save a ton and even more if you can get them right sized for your usage.