Unexpected AWS Cost Explorer Costs

[mthbrown]

Describe the bug After starting to use this mod, I noticed that I'm seeing high usage of AWS Cost Explorer. After some digging, it appears that this module makes on average, 71 requests to the Cost Explorer API, which translates to $0.7/day. For production environments, this is a no-brainer but for testing, I thought that there should have been some type of warning as I left Steampipe running for over a month and got about $24 just from running this. I was running it on an AWS Organizations management account

Steampipe version (steampipe -v) v0.13.3

Plugin version (steampipe plugin list) v0.50.1

To reproduce

Expected behavior It would be great if there was some type of warning related to this or some type of way to set this to only run once a day

Additional context

[misraved]

Thanks @mthbrown for using Steampipe 👍 . Great to see that you are using the AWS Thrifty mod.

I see that you are using the 0.50.1 version of the AWS plugin, Could you please update it to the latest? We had made certain updates to our cost tables in this release.

We will definitely pick this up and see if we can bring about more optimisations around the cost tables 👍

[cbruno10]

Hey @mthbrown , sorry you ran into some unexpected costs!

In order to better understand your environment to help us improve the query and documentation for these checks, can you please provide some more details around the following:

• Are the tracked requests all from one account (assuming yes, but wanted to confirm)?
• How are you running Steampipe + the Thrifty checks, i.e., on a local machine, in a container, in a GitHub action, etc.?
• How frequently are you running the checks each day or week? Are they running on a schedule, or being manually run?
• Has caching been disabled anywhere (it's enabled by default)?
• How many accounts do you have in your AWS organization?
• Are you running all Thrifty checks, or just particular ones?
• For the full_month_cost_changes control (which is in the cost_explorer benchmark), approximately how many rows do you get in the check results? For instance, in one of my accounts I just tested against, I get 12 rows (services), which is based on which services I've used recently. This information may help us figure out how many requests are being made to the Cost Explorer API.

If there are any additional details you feel may be helpful, please include them as well.

Thanks!

[cbruno10]

@bigdatasourav Can you please have a look at any controls in this mod that query the AWS Cost Explorer API and track how many API requests each control does? This information will be helpful in determining if our controls and queries are optimized.

[bigdatasourav]

@cbruno10 only the full_month_cost_changes control query the AWS Cost Explorer API in this mod. The API request count is dependent on the number of accounts. Per account, the API will hit once for each time the control runs.

[cbruno10]

Thanks @bigdatasourav for confirming!

@mthbrown Based on our testing, it seems like the control only makes one API call per control run, but if you're still experiencing issues, please let us know and also see the questions in https://github.com/turbot/steampipe-mod-aws-thrifty/issues/74#issuecomment-1117824838, which can help us debug further. Thanks!