Issues when AWS_DEFAULT_REGION is set as an env var

stv-io commented 5 months ago

Hi, I've been trying this fork after discovering it from https://github.com/rebuy-de/aws-nuke/issues/1187#issuecomment-2045269560

I had CI pipelines which I adapted to use this image (keeping in mind the breaking changes). One thing which caused me a bit of pain, was that with the following env vars set AWS_REGION and AWS_DEFAULT_REGION (to eu-west-1 in my case) the cli was exiting with a misleading and confusing error:

time="2024-04-16T14:28:47Z" level=error msg="the custom region 'eu-west-1' must be specified in the configuration 'endpoints'"
time="2024-04-16T14:28:47Z" level=fatal msg="the custom region 'eu-west-1' must be specified in the configuration 'endpoints'"

I have just gotten around to figuring this out, let me know if I can provide additional context.

I plan to come back to this issue if I find anything out

ekristen commented 5 months ago

That's odd. Can you share a redacted config and how you are running the tool? Also how are you authenticating? Env var for keys? Assume role? The more you can provide the better I'll run it locally to try and duplicate.

ekristen commented 5 months ago

This could also help with debug but maybe not if auth is just not working.

https://ekristen.github.io/aws-nuke/cli-usage/#aws-nuke-explain-account

stv-io commented 5 months ago

Thanks for the quick response. I will try the explain-account and share the output later. In the meantime, the additional info you requested.

redacted config

here goes

---
regions:
- "global"
- "eu-west-1"
- "us-east-1"

blocklist:
   # redacted
  - "111111111111"
  - "2222222222"
  - "333333333"

presets:
  sso:
    filters:
      IAMSAMLProvider:
      - type: "regex"
        value: "AWSSSO_.*_DO_NOT_DELETE"
      IAMRole:
      - type: "glob"
        value: "AWSReservedSSO_*"
      IAMRolePolicyAttachment:
      - type: "glob"
        value: "AWSReservedSSO_*"

resource-types:
  excludes:
  - CloudTrailTrail
  - OSPackage
  - Route53HostedZone

settings:
  # force-delete-lightsail-addons: true
  disable-deletion-protection:
    RDSInstance: true
    EC2Instance: true
    CloudformationStack: true
    ELBv2: true
    QLDBLedger: true    

accounts:
  99999999999:   # redacted
    filters:
      IAMRole:
      - "OrganizationAccountAccessRole"
      - "shared-services-nuke-role"
      IAMRolePolicyAttachment:
      - "OrganizationAccountAccessRole -> AdministratorAccess"
      - "shared-services-nuke-role -> AdministratorAccess"
    presets:
      - sso

Also how are you authenticating

Starting from Github Actions runner, using AWS -> OIDC trust based on repository name
cross account role, from a shared account assuming a pre-provisioned role using aws-nuke nuke -c aws-nuke-config.yaml --force --quiet --assume-role-arn ${ASSUME_ROLE_ARN}
triggering a codebuild job, which runs the container image directory from ghcr, with this buildspec file (extract below)

version: 0.2
phases:
  install:
    run-as: root
    commands: apk add --no-cache --quiet aws-cli jq curl
  pre_build:
    commands:
       # AWS_NUKE_CONFIG generated and injected via terraform
      - echo ${AWS_NUKE_CONFIG} | base64 -d > aws-nuke-config.yaml
      - sed -i "s/\${account}/${ASSUME_ROLE_ACCOUNT_ID}/g" aws-nuke-config.yaml
      - sed -i "s/\${assume_role_name}/${ASSUME_ROLE_NAME}/g" aws-nuke-config.yaml
      - cat aws-nuke-config.yaml
      - export ASSUME_ROLE_ARN="arn:aws:iam::${ASSUME_ROLE_ACCOUNT_ID}:role/${ASSUME_ROLE_NAME}"
  build:
    commands:
      - |
        if [ "$DRY_RUN" = "true" ]; then
          echo "Running aws-nuke in dry-run mode .."
          aws-nuke nuke -c aws-nuke-config.yaml --force --quiet --assume-role-arn ${ASSUME_ROLE_ARN}
....  #rest of the buildspec yaml

A whole list of ENV VARS are passed from Github Actions, including which are the AWS_REGION and the AWS_DEFAULT_REGION

I was debugging locally, by running the same container interactively, and assuming similar roles and permissions, but without the plethora of ENV VARs. I wasn't passing the regions envs, and it was working locally. In order to "fix" this, I had to:

# buildspec extract
  build:
    commands:
      - |
        unset AWS_REGION
        unset AWS_DEFAULT_REGION
        echo "Running aws-nuke in dry-run mode .."
        echo "ASSUME_ROLE_ARN - ${ASSUME_ROLE_ARN}"
        aws-nuke nuke -c aws-nuke-config.yaml --force --quiet --assume-role-arn ${ASSUME_ROLE_ARN}

.. with the above trimmed down debugging config, the build ran successfully, and it listed all the resources to be nuked, as expected.

Thanks for the work on this fork! 🙇🏼

stv-io commented 5 months ago

I isolated to be a problem with the AWS_DEFAULT_REGION variable - replicated locally as well:

# export temp credentials from SSO page
/config $ export AWS_ACCESS_KEY_ID="AS..."
/config $ export AWS_SECRET_ACCESS_KEY="17...."
/config $ export AWS_SESSION_TOKEN="IQ...=="
# no issues with `AWS_REGION` set
/config $ export AWS_REGION=eu-west-1
/config $ aws-nuke nuke -c /config/forked-nuke.yaml  --force --quiet --assume-role-arn arn:aws:iam::11111111111111:role/shared-services-nuke-role
> aws-nuke - v3.0.0-beta.42 - 3b24ac94da0eecf04997cb7bd7276fdabf171cf9
Do you really want to nuke the account with the ID 11111111111111 and the alias 'disposable-ac-3'?
Waiting 10s before continuing.
^C (cancelled because I know the outcome, which is success)
/config $ unset AWS_REGION
/config $ export AWS_DEFAULT_REGION=eu-west-1
/config $ aws-nuke nuke -c /config/forked-nuke.yaml  --force --quiet --assume-role-arn arn:aws:iam::11111111111111:role/shared-services-nuke-role
ERRO[0000] the custom region 'eu-west-1' must be specified in the configuration 'endpoints'
FATA[0000] the custom region 'eu-west-1' must be specified in the configuration 'endpoints'
/config $

ekristen commented 5 months ago

Interesting I will take a look. It's on my plate to revamp the entire aws auth. Currently it's a modified version of the AWS SDK, it would be better just to allow the AWS SDK to do it, then that way all variations are supported. High on my support list is OIDC federated auth for the tool as well.

Let me see where the DEFAULT region might be coming into play.

ekristen commented 5 months ago

Looking at the code, it looks like only the following regions are allowed for AWS_DEFAULT_REGION.

us-east-1
us-east-2
us-west-1
us-west-2
us-gov-east-1
us-gov-east-2

See https://github.com/ekristen/aws-nuke/blob/main/pkg/commands/nuke/nuke.go#L83-L98

It looks like it's about setting AWS partitions for different things like standard vs gov vs china regions. Why all the regions aren't listed I'm not sure, they aren't on the upstream either. To be honest I think we can just add all other missing regions to https://github.com/ekristen/aws-nuke/blob/main/pkg/commands/nuke/nuke.go#L86 and it'll work, but I'll need to test and look into a bit more.

For now if you set your AWS_REGION=eu-west-1 and your AWS_DEFAULT_REGION=us-east-1 then everything should work just fine. The AWS_REGION is what matters most here.

ekristen commented 5 months ago

@stv-io please checkout #143, this should fix your problem. I did some digging, it looks like it was purely to determine AWS partition which can change for certain regions.

I swapped the code out for something better. I've done limited testing, seems to work ok.

I triggered a build here https://github.com/ekristen/aws-nuke/actions/runs/8715372441, you should be able to download the build artifacts, I do realize that potentially makes it harder to test in CI, but I'm hoping you can test outside of CI.

stv-io commented 5 months ago

Running the binaries, locally, directly on my mac (intel) seems to have worked as expected 👍🏼

❯ env | grep AWS
AWS_DEFAULT_REGION=eu-west-1
AWS_PAGER=
AWS_SECRET_ACCESS_KEY=G0...
AWS_ACCESS_KEY_ID=ASIA...
AWS_SESSION_TOKEN=IQoJ...==

~/Downloads/aws-nuke-fork on ☁️  (eu-west-1)
❯ ./aws-nuke nuke -c  /path/to/.local/forked-nuke.yaml --force --quiet --assume-role-arn arn:aws:iam::111111111111:role/shared-services-nuke-role
> aws-nuke - v3.0.0-beta.42-2-g4e32f2e - 4e32f2e55b6d88ca108f8920998cfbe62de32fb8
Do you really want to nuke the account with the ID 983055175492 and the alias 'disposable-ac-3'?
Waiting 10s before continuing.
^C

~/Downloads/aws-nuke-fork on ☁️  (eu-west-1) took 2s
❯ pwd
/Users/steve/Downloads/aws-nuke-fork

~/Downloads/aws-nuke-fork on ☁️  (eu-west-1)
❯ ls
LICENSE                         aws-nuke-v3.0.0-beta.42-2-g4e32f2e-darwin-amd64.tar.gz  aws-nuke-v3.0.0-beta.42-2-g4e32f2e-linux-arm64.tar.gz
README.md                       aws-nuke-v3.0.0-beta.42-2-g4e32f2e-darwin-arm64.tar.gz  aws-nuke-v3.0.0-beta.42-2-g4e32f2e-linux-arm7.tar.gz
aws-nuke                        aws-nuke-v3.0.0-beta.42-2-g4e32f2e-linux-amd64.tar.gz   binaries.zip

Unrelated, but mentioning, in case I did something wrong, or something wrong with the build process

I did try to build the docker image locally, and run the binary from the container, but something seemed to be not right with the resulting binary:

in aws-nuke on  fix-aws-parition-detection via 🐳 desktop-linux 🐹
❯ history | grep build
  522  rg build
  524  docker build -t aws-nuke:fix-aws-parition-detection .
  525  docker run -w /config -v $(pwd):/config --entrypont sh aws-nuke:fix-aws-parition-detection

/config $ /usr/local/bin/aws-nuke -c /config/forked-nuke.yaml  --force --quiet --assume-role-arn arn:aws:iam::983055175492:role/shared-services-nuke-role
sh: /usr/local/bin/aws-nuke: not found
/config $ ls /usr/local/bin/aws-nuke
/usr/local/bin/aws-nuke
/config $ /usr/local/bin/aws-nuke
sh: /usr/local/bin/aws-nuke: not found
/config $ file /usr/local/bin/aws-nuke
sh: file: not found
/config $ /usr/local/bin/aws-nuke
sh: /usr/local/bin/aws-nuke: not found
/config $ ls -lrtha /usr/local/bin/aws-nuke
-rwxr-xr-x    1 root     root      213.8M Apr 17 07:38 /usr/local/bin/aws-nuke
/config $ chmod +x /usr/local/bin/aws-nuke
chmod: /usr/local/bin/aws-nuke: Operation not permitted
/config $ cd /usr/local/bin/
/usr/local/bin $ ls
aws-nuke
/usr/local/bin $ ls -lrtha
total 214M
drwxr-xr-x    1 root     root        4.0K Jan 26 17:53 ..
-rwxr-xr-x    1 root     root      213.8M Apr 17 07:38 aws-nuke
drwxr-xr-x    1 root     root        4.0K Apr 17 07:38 .
/usr/local/bin $ ./aws-nuke
sh: ./aws-nuke: not found

ekristen commented 5 months ago

Looks like I have a bug there around dynamically linked libraries when doing docker build directly. I'll do another PR to fix that.

The preferred way to build the docker containers is with goreleaser.

goreleaser --snapshot --clean

However, this will net you all the binaries and docker images so it can be a bit heavy.

ekristen commented 5 months ago

@stv-io also thanks for testing and using the fork I appreciate it. I'll get this merged shortly.

ekristen commented 5 months ago

:tada: This issue has been resolved in version 3.0.0-beta.43 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

ekristen commented 3 months ago

:tada: This issue has been resolved in version 3.0.0 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

ekristen / aws-nuke

Issues when AWS_DEFAULT_REGION is set as an env var #142