[Bug]: run local cannot get secrets from secrets manager

[lennertcc]

Description:

Using copilot run local --use-task-role --proxy --name app --env test runs into an error in get secrets: ValidationException: Invalid name. Must be a valid name containing alphanumeric characters, or any of the following: -/_+=.@!

Details:

copilot version: v1.33.4

Manifest (stripped down a bit):

name: app
type: Backend Service

image:
  build:
    dockerfile: app/Dockerfile.prod
    context: app
    cache_from:
      - 123123123123.dkr.ecr.eu-central-1.amazonaws.com/app-copilot/app:latest
    args:
      BUILDKIT_INLINE_CACHE: 1

  port: 8000

exec: true

env_file: copilot/env-files/test.env

environments:
  test:
    cpu: 1024 # Number of CPU units for the task.
    memory: 2048 # Amount of memory in MiB used by the task.
    count:
      range: 1-2
      cpu_percentage: 65

  prod:
    cpu: 2048 # Number of CPU units for the task.
    memory: 4096 # Amount of memory in MiB used by the task.
    count:
      range: 2-10
      cpu_percentage: 50
    env_file: copilot/env-files/production.env

secrets:
  EMAIL_HOST_USER: 
    secretsmanager: 'smtp-mailing:smtp_email_host_user::'
  EMAIL_HOST_PASSWORD: 
    secretsmanager: 'smtp-mailing:smtp_email_host_password::'

Observed result:

$ copilot run local --use-task-role --proxy --name app --env test

✘ get task: get env vars: get secrets: get secret "arn:aws:secretsmanager:eu-central-1:123123123123:secret:smtp-mailing:smtp_email_host_user::": get secret "arn:aws:secretsmanager:eu-central-1:123123123123:secret:smtp-mailing:smtp_email_hostuser::" from secrets manager: ValidationException: Invalid name. Must be a valid name containing alphanumeric characters, or any of the following: -/+=.@! status code: 400, request id: 3ed0f718-bd00-4a1f-b827-4b6850e6a2a6

Expected result:

I expect the secrets are being fetched, just like during deployment with a copilot pipeline.

Debugging:

[Lou1415926]

@lennertcc This seems to be a gap between our deploy and run local. CloudFormation (the deployment mechanism of deploy) is able to take secret names followed by a JSON key. The AWS SDK API, which is what run local get the secret values by, does not.

I'm leaning towards considering this a feature request. It is expected that there is some gap between deploy and run local, and the non-JSON-key secret ARNs work. Please let me know if you disagree!

To support this. we will need to:

1. Parse the secret name (smtp-mailing) from the resource name that you provided smtp-mailing:smtp_email_host_user::. This tells us that smtp-mailing is the name and smtp_email_host_user is the JSON key.
2. Get the secret value, which is expected to be a JSON string. Parse the JSON string and get the smtp_email_host_user key from it.

For now, you can use --env-var-override to provide the value from your local machine:

run local --env-var-override EMAIL_HOST_USER=user --env-var-override EMAIL_HOST_PASSWORD=pass

[lennertcc]

Thank you for investigating.

I think it would be nice to fix gaps in general, just because of the developer experience. But having the secrets written with their json key like this is not a must have in any way. At least in my application I can easily rewrite it to get the whole secret value and get the individual json keys in code.

So for now I will solve it on my side, avoiding the use of json keys.

Hopefully I won't run into the next gap right after that, or you will hear from me again soon ;).