search

turbot / steampipe-plugin-sdk

Steampipe Plugin SDK is a simple abstraction layer to write a Steampipe plugin. Plugins automatically work across all engine types including the Steampipe CLI, Postgres FDW, SQLite extension and the export CLI.
https://hub.steampipe.io/plugins
Apache License 2.0
32 stars 16 forks source link

AWS plugin: IgnoreConfig codes are not functioning as expected #764

Open bigdatasourav opened 6 months ago

bigdatasourav commented 6 months ago

IgnoreConfig codes are not working as expected which resulted in unnecessary logs.

Sample query:

select * from aws_ecr_image_scan_finding where repository_name = 'abc' and image_tag = '2.1'

In the table below is the list config -

List: &plugin.ListConfig{
            Hydrate: listAwsEcrImageScanFindings,
            Tags:    map[string]string{"service": "ecr", "action": "DescribeImageScanFindings"},
            IgnoreConfig: &plugin.IgnoreConfig{
                ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"RepositoryNotFoundException", "ImageNotFoundException", "ScanNotFoundException"}),
            },
            KeyColumns: []*plugin.KeyColumn{
                {Name: "repository_name", Require: plugin.Required},
                {Name: "image_tag", Require: plugin.Required},
            },
        },

log output: (Even though RepositoryNotFoundException is already placed in IgnoreConfig)

2024-03-07 13:21:30.281 UTC [ERROR] steampipe-plugin-aws.plugin: [ERROR] 1709817689522: aws_ecr_image_scan_finding.listAwsEcrImageScanFindings: api_error="operation error ECR: DescribeImageScanFindings, https response error StatusCode: 400, RequestID: 576ab62f-9090-4af0-8420-fccd9af474ae, RepositoryNotFoundException: The repository with name 'abc' does not exist in the registry with id '1323443245'"
captainfalcon23 commented 6 months ago

As I mentioned in the original issue, the issue isn’t just about logs in a log file. It’s about steampipe inefficiency of querying every account and region in an aggregation instead of smartly hydrating the tables from parents. If the context is there that an image belongs to account X in region Y, I don’t understand why steampipe can’t use that context to query the appropriate connection for further info.