Cloudwatch DescribeMetricFilters function error in aws plugin 136+

[idesofoctober]

Describe the bug When running cis 3.0 benchmarks with aws plugin 136.0 and newer (though I'm testing further now re: ensuring it is in fact happening in all the current versions) I get this error for every account for the benchmarks that run queries dependent on data hydration (I believe) using DescribeMetricFilters:

operation error CloudWatch Logs: DescribeMetricFilters, https response error StatusCode: 400, RequestID: 5ded2bb4-e1db-4135-a70b-a44dde2a693e, InvalidParameterException: Describe Metric Filters request must contain both MetricName and MetricNamespace

Steampipe version (steampipe -v) Currently seeing this on 0.23.2 but I was seeing this on 0.21.4 and 0.22.0 (the error "forced me to upgrade to 0.23.2, but still seeing it)

Plugin version (steampipe plugin list) Definitely v0.136.0, doing some further testing, I can't seem to run v0.139.0 for some reason (container keeps crashing on steampipe service start, not sure why yet). Going to test 0.136.1 and 0.137/0.138 today if I can

To reproduce steampipe check benchmark.cis_v300

Expected behavior no errors re: the DescribeMetricFilters function

[ParthaI]

Hello @idesofoctober,

I apologize for the error you are experiencing.

I attempted to run the command steampipe check benchmark.cis_v300 multiple times, but I was unable to replicate the error.

In the plugin, the API DescribeMetricFilters is only used in the aws_cloudwatch_log_metric_filter table. Generally, we do not handle the InvalidParameterException error. We currently have a few tables that handle this error for specific cases. However, we are planning to remove them if they are deemed unnecessary. Additionally, in the compliance pack, I did not find any queries that use both MetricName and MetricNamespace together in the WHERE clause.

Could you please provide more context about which specific control or query is failing for you? This will help us determine if the issue needs to be addressed on the compliance query side.

Thanks!

[idesofoctober]

Hi @ParthaI , I appreciate you digging into this, I'm actually having quite a lot of problems right now, and this is only one of them :(

benchmark cisv_300 runs the below from the compliance pack, I believe and that particular control is one of the ones that errors across all my accounts. (However I cannot reliably reproduce running locally with an arm64 build, only when running in fargate ecs with an amd64 container/infrastructure....I don't mean to add any red herrings here, but wondering if you have any insight or if that brings anything interesting front of mind re: other issues you may have seen?

Can you remind me where to find the specific query referenced below in the code? Thanks!

control "log_metric_filter_unauthorized_api" { title = "Ensure a log metric filter and alarm exist for unauthorized API calls" description = "Real-time monitoring of API calls can be achieved by directing CloudTrail Logs to CloudWatch Logs and establishing corresponding metric filters and alarms. It is recommended that a metric filter and alarm be established for unauthorized API calls." query = query.log_metric_filter_unauthorized_api

[ParthaI]

Hi @idesofoctober,

You can find the query at the line query = query.log_metric_filter_unauthorized_api. The log_metric_filter_unauthorized_api query is being used in this context. You can view the actual query by visiting the Powerpipe Hub and filtering by the name log_metric_filter_unauthorized_api. The specific query used for that control can be seen like: https://hub.powerpipe.io/mods/turbot/aws_compliance/queries/log_metric_filter_unauthorized_api#sql.

Please try running the query on your local setup to see if you encounter any issues with it.

Thanks!

[gitlankford]

Additionally, in the compliance pack, I did not find any queries that use both MetricName and MetricNamespace together in the WHERE clause.

The error is that one or the other was used but not both - so a query that had both together should work whereas one that only has metric_name would fail with that error.

[idesofoctober]

@ParthaI I think the error is coming from upstream re: the compliance mod->aws plugin when it tries to run some hydrate functions. Is that possible?

I am now running:

steampipe v0.23.2 powerpipe v0.4.0 aws plugin v 0.139.0 and the latest compliance mod

When I run

powerpipe benchmark run aws_compliance.benchmark.cis_v300 --query-timeout 600 --export=_cis_v300_06242024.csv

with the additional environment variables in my docker container (some of which may be redundant now that i've transitioned to powerpipe):

disable auto-update

ENV STEAMPIPE_UPDATE_CHECK=false

disable telemetry

ENV STEAMPIPE_TELEMETRY=none

limit parallel connections to DB

ENV STEAMPIPE_MAX_PARALLEL=1 ENV POWERPIPE_MAX_PARALLEL=1

SQL Query Timeout Update

ENV STEAMPIPE_QUERY_TIMEOUT=600

I get this error:

Operation error CloudWatch Logs: DescribeMetricFilters, https response error StatusCode: 400, RequestID: , InvalidParameterException: Describe Metric Filters request must contain both MetricName and MetricNamespace (SQLSTATE HV000)

for the below controls:

4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7. 4.8, 4.9, 4.10, 4.11, 4.12, 4.13, 4.14, 4.15

[idesofoctober]

@ParthaI this DOES NOT happen with all of our accounts FYI, is there potentially something to look at re: what is different in accounts where this error occurs vs. accounts where it does not?

[ParthaI]

Hi @idesofoctober,

I apologize for misunderstanding the context initially. We were able to reproduce the error you are experiencing.

From the plugin side, the API is behaving as expected. However, I have raised a PR based on the API behavior(Preventing API calls based on query parameters). Another PR will be raised from the compliance side to resolve the issue.

Note: The plugin behavior requires that if we pass the MetricName, it must be accompanied by the MetricNamespace.

Thank you so much for providing detailed information!

CC @khushboo9024 @misraved

[idesofoctober]

Thanks @ParthaI I was looking at the aws compliance mod github repo and didn't see any issues/prs for this, but not understanding the code well, is the v.95 release meant to address this (not sure if you are even the right person to ask?)?

[ParthaI]

Hi @idesofoctober,

The fixes have already been merged from the compliance side in the main branch. The v0.95 release contains the fix addressing this issue.

Previously, the table aws_cloudwatch_log_metric_filter was queried by providing only the MetricName as a query parameter. However, the AWS DescribeMetricFilters API requires that if MetricName is provided, it must be accompanied by MetricNamespace. We can't pass only one of these values in a single API call.

The plugin fix ensures that API calls are made correctly by manipulating the query parameters to adhere to the API's behavior.

Thanks!