search

ekristen / aws-nuke

Remove all the resources from an AWS account
https://ekristen.github.io/aws-nuke/
MIT License
257 stars 26 forks source link

GameLift and Rekognition errors #372

Open fridim opened 1 month ago

fridim commented 1 month ago

On one of our accounts, aws-nuke fails with dial tcp and no such host errors in stderr.

command: aws-nuke nuke --profile ... -c "nuke-config.yml" --quiet --log-level error --no-dry-run --force duration: ~4min RC: -9

"stderr_lines": [
            "time=\"2024-10-09T11:53:05Z\" level=error msg=\"Listing GameLiftFleet failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:07Z\" level=error msg=\"Listing GameLiftMatchmakingRuleSet failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:07Z\" level=error msg=\"Listing TranscribeLanguageModel failed:\\n    BadRequestException: Your account isn't authorized to call this operation. Check your account permissions and try your request again.\" error=\"BadRequestException: Your account isn't authorized to call this operation. Check your account permissions and try your request again.\"",
            "time=\"2024-10-09T11:53:07Z\" level=error msg=\"Listing GameLiftQueue failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:09Z\" level=error msg=\"Listing GameLiftMatchmakingConfiguration failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:10Z\" level=error msg=\"Listing GameLiftBuild failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-north-1.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-north-1.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:12Z\" level=error msg=\"Listing GameLiftFleet failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:13Z\" level=error msg=\"Listing GameLiftMatchmakingRuleSet failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:14Z\" level=error msg=\"Listing GameLiftQueue failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:15Z\" level=error msg=\"Listing GameLiftMatchmakingConfiguration failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:16Z\" level=error msg=\"Listing GameLiftBuild failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.eu-west-3.amazonaws.com/\\\": dial tcp: lookup gamelift.eu-west-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:30Z\" level=error msg=\"Listing GameLiftFleet failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:31Z\" level=error msg=\"Listing RedshiftServerlessNamespace failed:\\n    ValidationException: The Namespaces operation isn't supported.\" error=\"ValidationException: The Namespaces operation isn't supported.\"",
            "time=\"2024-10-09T11:53:32Z\" level=error msg=\"Listing RedshiftServerlessWorkgroup failed:\\n    ValidationException: The Workgroups operation isn't supported.\" error=\"ValidationException: The Workgroups operation isn't supported.\"",
            "time=\"2024-10-09T11:53:32Z\" level=error msg=\"Listing GameLiftMatchmakingRuleSet failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:33Z\" level=error msg=\"Listing GameLiftQueue failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:33Z\" level=error msg=\"Listing BackupReportPlan failed:\\n    AccessDeniedException: This API is not available in current Region.\\n    \\tstatus code: 403, request id: 17c06bab-4a1d-433e-8362-dabe319aa47c\" error=\"AccessDeniedException: This API is not available in current Region.\\n\\tstatus code: 403, request id: 17c06bab-4a1d-433e-8362-dabe319aa47c\"",
            "time=\"2024-10-09T11:53:34Z\" level=error msg=\"Listing GameLiftMatchmakingConfiguration failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:35Z\" level=error msg=\"Listing GameLiftBuild failed:\\n    RequestError: send request failed\\n    caused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\" error=\"RequestError: send request failed\\ncaused by: Post \\\"https://gamelift.ap-northeast-3.amazonaws.com/\\\": dial tcp: lookup gamelift.ap-northeast-3.amazonaws.com on 172.30.0.10:53: no such host\"",
            "time=\"2024-10-09T11:53:36Z\" level=error msg=\"Listing RedshiftServerlessSnapshot failed:\\n    ValidationException: The ServerlessToServerlessRestore operation isn't supported.\" error=\"ValidationException: The ServerlessToServerlessRestore operation isn't supported.\"",
            "time=\"2024-10-09T11:53:59Z\" level=error msg=\"Listing RekognitionProject failed:\\n    AccessDeniedException: \" error=\"AccessDeniedException: \"",
            "time=\"2024-10-09T11:53:59Z\" level=error msg=\"Listing RekognitionDataset failed:\\n    AccessDeniedException: \" error=\"AccessDeniedException: \"",
            "time=\"2024-10-09T11:54:55Z\" level=error msg=\"Listing RekognitionProject failed:\\n    AccessDeniedException: \" error=\"AccessDeniedException: \"",
            "time=\"2024-10-09T11:54:55Z\" level=error msg=\"Listing RekognitionDataset failed:\\n    AccessDeniedException: \" error=\"AccessDeniedException: \""
        ],
        "stdout_lines": [
            "> aws-nuke - v3.26.0 - 5e33e8901f8786f2839fabbbc3bb26086b01fc2a",
            "Do you really want to nuke the account with the ID xxxxx and the alias 'xxxx'?",
            "Waiting 10s before continuing.",
            "us-east-2 - CloudFormationStack - sandbox-ml2hn-open-environment - [CreationTime: \"2024-10-02T21:34:07Z\", LastUpdatedTime: \"2024-10-02T21:34:07Z\", Name: \"sandbox-xxx-open-environment\", tag:Stack: \"project sandbox-xxxx\", tag:env_type: \"sandbox\", tag:guid: \"xxxxx\", tag:owner: \"unknown\", tag:uuid: \"xxxx\"] - would remove"
        ]
    }
}

We notice the memory usage goes up to 2Go (limit of the pod) before being killed. Could be the indication of a memory leak as none of the other account reach that amount of memory.

2024-10-09_14-53

ekristen commented 1 month ago

If you are seeing multiple errors like that, aws-nuke isn't exiting because of those, it's just indicating it can't do the query against the region, but it does sounds like you said the pod got killed due to OOM?

Any idea as to the number of actual resources found in the account?

ekristen commented 1 month ago

@fridim exclude all the resource types listed in your stderr and try again. They shouldn't have any impact. Those errors trigger when the lister for the resource is unable to list, and it then exits the specific lister, the rest of the tool continues to operator. Also make sure S3Object is excluded.

fridim commented 1 month ago

I can exclude them but what if i want aws-nuke to remove them if they exist (and work in other regions)?

ekristen commented 1 month ago

For troubleshooting purposes. I'll look into handling the other errors but they aren't interfering with the tool running.