search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.99k stars 1.86k forks source link

AWS Lambda - "list_coverage" filter on `lambdaFunctionRuntime` not working #4224

Closed RegisGraptin closed 1 week ago

RegisGraptin commented 1 month ago

Describe the bug

I am currently using AWS Lambda to retrieve the Lambda Function from AWS. I would like to apply a filter on it using the lambdaFunctionRuntime. However, when applying it, it seems the filter is not taken into consideration. For instance, I have the following filters (that I am using in filterCriteria for the list_coverage function):

{
    "resourceType": [
        {
            "comparison": "EQUALS",
            "value": "AWS_LAMBDA_FUNCTION"
        }
    ],
    "lambdaFunctionRuntime": [
        {
            "comparison": "EQUALS",
            "value": "PYTHON_3_9"
        },
        {
            "comparison": "EQUALS",
            "value": "PYTHON_3_8"
        }
    ]
}

However, when using it, I have a bunch of lambda resource with different runtime as node and other python version.

Expected Behavior

By applying the filter, I should only have lambda function resource with the runtime matching the configuration. If I took my previous example, in the API response, we should have PYTHON_3_9 and PYTHON_3_8.

Current Behavior

Currently, it seems the filter is not working for the list_coverage function on the lambdaFunctionRuntime.

Reproduction Steps

# I am using `boto3==1.34.111`

client = boto3.client("lambda", **params)

filters = defaultdict(list)
filters["resourceType"].append({"comparison": "EQUALS", "value": "AWS_LAMBDA_FUNCTION"})
filters["lambdaFunctionRuntime"].append({"comparison": "EQUALS", "value": "PYTHON_3_9"})
filters["lambdaFunctionRuntime"].append({"comparison": "EQUALS", "value": "PYTHON_3_8"})

for page in client.get_paginator("list_coverage").paginate(
    filterCriteria=filters,
):
    # FIXME :: Not expected filter on runtime

Possible Solution

I think it is related to the API endpoint, not taking into consideration this filter.

Additional Information/Context

No response

SDK version used

1.34.111

Environment details (OS name and version, etc.)

Ubuntu - Python 3.12.3

tim-finnigan commented 1 month ago

Thanks for reaching out. Here are the available Boto3 paginators for Lambda: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#paginators . Note there is no list_coverage paginator. If you're using a different paginator, could you provide a complete snippet showing that?

Also if you could share debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script then that could help with further investigation as well.

tim-finnigan commented 1 month ago

It looks like you're probably trying to use this inspector2 paginator: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/inspector2/paginator/ListCoverage.html . Can you confirm?

import boto3

client = boto3.client('inspector2')

paginator = client.get_paginator('list_coverage')

response_iterator = paginator.paginate(
    filterCriteria={
        'resourceType': [
            {
                'comparison': 'EQUALS',
                'value': 'AWS_LAMBDA_FUNCTION'
            },
        ],
        'lambdaFunctionRuntime': [
            {
                'comparison': 'EQUALS',
                'value': 'PYTHON_3_9'
            },
            {
                'comparison': 'EQUALS',
                'value': 'PYTHON_3_8'
            },
        ]
    }
)

for page in response_iterator:
    print(page)
RegisGraptin commented 1 month ago

Hello @tim-finnigan, thanks for your quick answer. Yes I confirm, I am using the inspector2 https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/inspector2/client/list_coverage.html

tim-finnigan commented 1 month ago

@RegisGraptin thanks for confirming. Can you share the full code snippet you're using, along with debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script to help with further investigation?

RegisGraptin commented 1 month ago

@tim-finnigan there it is

Show logs ```log 2024-08-07 07:13:01,273 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2024-08-07 07:13:01,274 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway 2024-08-07 07:13:01,275 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2024-08-07 07:13:01,276 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2024-08-07 07:13:01,276 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2024-08-07 07:13:01,276 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2024-08-07 07:13:01,277 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section 2024-08-07 07:13:01,278 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2024-08-07 07:13:01,278 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section 2024-08-07 07:13:01,278 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2024-08-07 07:13:01,278 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section 2024-08-07 07:13:01,280 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/endpoints.json 2024-08-07 07:13:01,294 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/sdk-default-configuration.json 2024-08-07 07:13:01,294 botocore.hooks [DEBUG] Event choose-service-name: calling handler 2024-08-07 07:13:01,300 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/inspector2/2020-06-08/service-2.json.gz 2024-08-07 07:13:01,309 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/inspector2/2020-06-08/endpoint-rule-set-1.json.gz 2024-08-07 07:13:01,309 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/partitions.json 2024-08-07 07:13:01,310 botocore.hooks [DEBUG] Event creating-client-class.inspector2: calling handler 2024-08-07 07:13:01,310 botocore.configprovider [DEBUG] Looking for endpoint for inspector2 via: environment_service 2024-08-07 07:13:01,310 botocore.configprovider [DEBUG] Looking for endpoint for inspector2 via: environment_global 2024-08-07 07:13:01,310 botocore.configprovider [DEBUG] Looking for endpoint for inspector2 via: config_service 2024-08-07 07:13:01,310 botocore.configprovider [DEBUG] Looking for endpoint for inspector2 via: config_global 2024-08-07 07:13:01,310 botocore.configprovider [DEBUG] No configured endpoint found. 2024-08-07 07:13:01,314 botocore.endpoint [DEBUG] Setting inspector2 timeout as (60, 60) 2024-08-07 07:13:01,314 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/_retry.json 2024-08-07 07:13:01,315 botocore.client [DEBUG] Registering retry handlers for service: inspector2 2024-08-07 07:13:01,315 botocore.hooks [DEBUG] Event choose-service-name: calling handler 2024-08-07 07:13:01,316 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/lambda/2015-03-31/service-2.json.gz 2024-08-07 07:13:01,319 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/lambda/2015-03-31/endpoint-rule-set-1.json.gz 2024-08-07 07:13:01,320 botocore.hooks [DEBUG] Event creating-client-class.lambda: calling handler 2024-08-07 07:13:01,320 botocore.configprovider [DEBUG] Looking for endpoint for lambda via: environment_service 2024-08-07 07:13:01,320 botocore.configprovider [DEBUG] Looking for endpoint for lambda via: environment_global 2024-08-07 07:13:01,320 botocore.configprovider [DEBUG] Looking for endpoint for lambda via: config_service 2024-08-07 07:13:01,320 botocore.configprovider [DEBUG] Looking for endpoint for lambda via: config_global 2024-08-07 07:13:01,320 botocore.configprovider [DEBUG] No configured endpoint found. 2024-08-07 07:13:01,321 botocore.endpoint [DEBUG] Setting lambda timeout as (60, 60) 2024-08-07 07:13:01,322 botocore.client [DEBUG] Registering retry handlers for service: lambda 2024-08-07 07:13:01,326 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/inspector2/2020-06-08/paginators-1.json 2024-08-07 07:13:01,326 botocore.loaders [DEBUG] Loading JSON file: /.../.venv/lib/python3.12/site-packages/botocore/data/inspector2/2020-06-08/paginators-1.sdk-extras.json 2024-08-07 07:13:01,327 botocore.hooks [DEBUG] Event before-parameter-build.inspector2.ListCoverage: calling handler 2024-08-07 07:13:01,327 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'ap-southeast-1', 'UseDualStack': False, 'UseFIPS': False} 2024-08-07 07:13:01,327 botocore.regions [DEBUG] Endpoint provider result: https://inspector2.ap-southeast-1.amazonaws.com 2024-08-07 07:13:01,327 botocore.hooks [DEBUG] Event before-call.inspector2.ListCoverage: calling handler 2024-08-07 07:13:01,327 botocore.hooks [DEBUG] Event before-call.inspector2.ListCoverage: calling handler 2024-08-07 07:13:01,327 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListCoverage) with params: {'url_path': '/coverage/list', 'query_string': {}, 'method': 'POST', 'headers': {'Content-Type': 'application/json', 'User-Agent': 'Boto3/1.34.111 md/Botocore#1.34.116 ua/2.0 os/linux#6.5.0-45-generic md/arch#x86_64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.116'}, 'body': b'{"filterCriteria": {"resourceType": [{"comparison": "EQUALS", "value": "AWS_LAMBDA_FUNCTION"}], "lambdaFunctionRuntime": [{"comparison": "EQUALS", "value": "NODEJS_12_X"}, {"comparison": "EQUALS", "value": "PYTHON_3_7"}]}, "maxResults": 100}', 'url': 'https://inspector2.ap-southeast-1.amazonaws.com/coverage/list', 'context': {'client_region': 'ap-southeast-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-08-07 07:13:01,328 botocore.hooks [DEBUG] Event request-created.inspector2.ListCoverage: calling handler > 2024-08-07 07:13:01,328 botocore.hooks [DEBUG] Event choose-signer.inspector2.ListCoverage: calling handler 2024-08-07 07:13:01,328 botocore.auth [DEBUG] Calculating signature using v4 auth. 2024-08-07 07:13:01,328 botocore.auth [DEBUG] CanonicalRequest: POST /coverage/list content-type:application/json host:inspector2.ap-southeast-1.amazonaws.com x-amz-date:20240807T071301Z content-type;host;x-amz-date xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2024-08-07 07:13:01,328 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20240807T071301Z 20240807/ap-southeast-1/inspector2/aws4_request xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2024-08-07 07:13:01,328 botocore.auth [DEBUG] Signature: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2024-08-07 07:13:01,328 botocore.hooks [DEBUG] Event request-created.inspector2.ListCoverage: calling handler 2024-08-07 07:13:01,328 botocore.endpoint [DEBUG] Sending http request: 2024-08-07 07:13:01,329 botocore.httpsession [DEBUG] Certificate path: /.../.venv/lib/python3.12/site-packages/certifi/cacert.pem 2024-08-07 07:13:01,329 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): inspector2.ap-southeast-1.amazonaws.com:443 2024-08-07 07:13:04,058 urllib3.connectionpool [DEBUG] https://inspector2.ap-southeast-1.amazonaws.com:443 "POST /coverage/list HTTP/1.1" 200 72303 2024-08-07 07:13:04,312 botocore.parsers [DEBUG] Response headers: {'Date': 'Wed, 07 Aug 2024 07:13:03 GMT', 'Content-Type': 'application/json', 'Content-Length': '72303', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'x-amz-apigw-id': 'xxxxxxxxxxxxxxx=', 'X-Amzn-Trace-Id': 'Root=1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxx'} 2024-08-07 07:13:04,313 botocore.parsers [DEBUG] Response body: b'{"coveredResources":[{...}] // Response with "runtime":"PYTHON_3_8", "runtime":"PYTHON_3_9" .... (Filter on NODEJS_12_X & PYTHON_3_7 in this requests) 2024-08-07 07:13:04,326 botocore.hooks [DEBUG] Event needs-retry.inspector2.ListCoverage: calling handler 2024-08-07 07:13:04,326 botocore.retryhandler [DEBUG] No retry needed. 2024-08-07 07:13:04,335 botocore.hooks [DEBUG] Event before-parameter-build.inspector2.ListCoverage: calling handler 2024-08-07 07:13:04,335 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'ap-southeast-1', 'UseDualStack': False, 'UseFIPS': False} 2024-08-07 07:13:04,335 botocore.regions [DEBUG] Endpoint provider result: https://inspector2.ap-southeast-1.amazonaws.com 2024-08-07 07:13:04,336 botocore.hooks [DEBUG] Event before-call.inspector2.ListCoverage: calling handler 2024-08-07 07:13:04,336 botocore.hooks [DEBUG] Event before-call.inspector2.ListCoverage: calling handler 2024-08-07 07:13:04,336 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListCoverage) with params: {'url_path': '/coverage/list', 'query_string': {}, 'method': 'POST', 'headers': {'Content-Type': 'application/json', 'User-Agent': 'Boto3/1.34.111 md/Botocore#1.34.116 ua/2.0 os/linux#6.5.0-45-generic md/arch#x86_64 lang/python#3.12.3 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.116'}, 'body': b'{"filterCriteria": {"resourceType": [{"comparison": "EQUALS", "value": "AWS_LAMBDA_FUNCTION"}], "lambdaFunctionRuntime": [{"comparison": "EQUALS", "value": "NODEJS_12_X"}, {"comparison": "EQUALS", "value": "PYTHON_3_7"}]}, "maxResults": 100, "nextToken": "xxxxxxxx"}', 'url': 'https://inspector2.ap-southeast-1.amazonaws.com/coverage/list', 'context': {'client_region': 'ap-southeast-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-08-07 07:13:04,336 botocore.hooks [DEBUG] Event request-created.inspector2.ListCoverage: calling handler > 2024-08-07 07:13:04,336 botocore.hooks [DEBUG] Event choose-signer.inspector2.ListCoverage: calling handler 2024-08-07 07:13:04,336 botocore.auth [DEBUG] Calculating signature using v4 auth. 2024-08-07 07:13:04,336 botocore.auth [DEBUG] CanonicalRequest: POST /coverage/list content-type:application/json host:inspector2.ap-southeast-1.amazonaws.com x-amz-date:20240807T071304Z content-type;host;x-amz-date xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2024-08-07 07:13:04,336 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20240807T071304Z 20240807/ap-southeast-1/inspector2/aws4_request xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2024-08-07 07:13:04,336 botocore.auth [DEBUG] Signature: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2024-08-07 07:13:04,336 botocore.hooks [DEBUG] Event request-created.inspector2.ListCoverage: calling handler 2024-08-07 07:13:04,336 botocore.endpoint [DEBUG] Sending http request: 2024-08-07 07:13:04,336 botocore.httpsession [DEBUG] Certificate path: /.../.venv/lib/python3.12/site-packages/certifi/cacert.pem 2024-08-07 07:13:06,052 urllib3.connectionpool [DEBUG] https://inspector2.ap-southeast-1.amazonaws.com:443 "POST /coverage/list HTTP/1.1" 200 59237 2024-08-07 07:13:06,209 botocore.parsers [DEBUG] Response headers: {'Date': 'Wed, 07 Aug 2024 07:13:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '59237', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'xxx', 'x-amz-apigw-id': 'xxxx', 'X-Amzn-Trace-Id': 'Root=xxxx'} 2024-08-07 07:13:06,209 botocore.parsers [DEBUG] Response body: b'{"coveredResources":[{...}],"nextToken":"..."}' 2024-08-07 07:13:06,215 botocore.hooks [DEBUG] Event needs-retry.inspector2.ListCoverage: calling handler 2024-08-07 07:13:06,215 botocore.retryhandler [DEBUG] No retry needed. 2024-08-07 07:13:06,215 botocore.hooks [DEBUG] Event before-parameter-build.inspector2.ListCoverage: calling handler 2024-08-07 07:13:06,215 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'ap-southeast-1', 'UseDualStack': False, 'UseFIPS': False} 2024-08-07 07:13:06,216 botocore.regions [DEBUG] Endpoint provider result: https://inspector2.ap-southeast-1.amazonaws.com ```
tim-finnigan commented 1 month ago

Thanks for following up. It looks like the lambdaFunctionRuntime filter just does not get applied, for example calling the API via a CLI command:

aws inspector2 list-coverage \
--filter-criteria '{"resourceType":[{"comparison":"EQUALS","value":"AWS_LAMBDA_FUNCTION"}],"lambdaFunctionRuntime":[{"comparison":"EQUALS","value":"PYTHON_3_9"},{"comparison":"EQUALS","value":"PYTHON_3_8"}]}' \
--query 'coveredResources[].resourceMetadata.lambdaFunction.runtime'

Results in:

[
    "PYTHON_3_9",
    "PYTHON_3_9",
    "PYTHON_3_7",
    "PYTHON_3_9",
    "PYTHON_3_9",
    "PYTHON_3_9",
    "NODEJS_16_X",
    "NODEJS_14_X",
...

I'll try reaching out to the Inspector team for more info regarding this behavior, as it is an issue with the API filters/response rather than the Boto3 SDK directly.

tim-finnigan commented 1 month ago

We heard back from the service team who acknowledged the issue and is working on a fix.

tim-finnigan commented 3 weeks ago

We were informed that the fix is currently being deployed. Once the API fix is released, then the filter should be working.

tim-finnigan commented 1 week ago

Upon testing I can confirm that this filter issue now appears resolved on the API side, so the correct results should be returning for you. If you're still seeing any incorrect results please let us know, otherwise I think this can be closed as resolved.

github-actions[bot] commented 1 week ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.