Allow passing environment variables from shell to Lambda through Zappa

[majackson]

Currently, the two supported methods of supplying environment variables to a lambda function through Zappa is explicitly naming the environment variables in theenvironment_variables settings in zappa_settings.json, or placing a JSON environment variables file in an S3 bucket that the AWS credentials used with Zappa have access to.

A third way which would be useful is if variables present in the shell running Zappa could be named in zappa_settings.json, and the values of these could be passed through to lambda. This would allow environment variables to be stored only in the environment Zappa is executed in, and negate the need for an additional S3 bucket solely for storing sensitive values.

An example of how this might appear in the zappa_settings.json file could be:

{
    "production": {
        "project_name": "my-project"
        "django_settings": "my_project.settings",
        "environment_pass": ["SECRET_THING"]
        "environment_variables": {
            "PRODUCTION": "TRUE",
        }
}

In this case, when Zappa is executed in a shell with SECRET_THING set to "123" the Lambda function would be deployed configured with PRODUCTION set to "TRUE", and SECRET_THING set to "123".

This would mirror some parts of docker tooling (i.e. docker-compose) which allow for environment variables to either be explicitly set, or passed through from the parent shell.

[fishi0x01]

I think in general it would be nice to allow env vars in multiple parts of the configuration

{
    "production": {
        "project_name": "my-project"
        "django_settings": "{{ env DJANGO_SETTINGS }}",
        "environment_pass": ["{{ env SECRET }}"]
        "environment_variables": {
            "PRODUCTION": "{{ env PRODUCTION }}",
        }
}

That way the zappa_settings.json acts like a template and can be reused in multiple environments. It could also be placed under source control and shared without worries about secrets or too static configuration.

[samlobel]

I think it would be helpful to be able to use a local file instead of a remote_env file, in exactly the same way. That way you could edit it easily, and just not check the sensitive parts into source control. Referenced by local_env : "./relative/path/to/.env"

Thoughts?

[arkaitzmugica]

I think the best way to make this would be the one majackson says... doing it "the docker way".

[kontrollanten]

This would be really great! I've done a workaround by creating a zappa_settings.json.tpl with env variables names like

   "environment_variables": {
     "DATABASE_PASSWORD": "$DATABASE_PASSWORD"
   }

And then I run envsubst < zappa_settings.json.tpl > zappa_settings.json in my CI environment to create the real file from the template.

[youcandanch]

@scoates instead of trying to adjust the schema of the settings file, I submitted a PR (#1731) adding a separate command for setting environment variables on-demand. I think it'll give an additional option for people here without polluting zappa_settings, but I'd be open to giving a schema adjustment the college try as well.

[drummonds]

I was just about to do this when realised I might as well take zappa_settings.json out of git and make it the respository of all secrets. You can have a skeleton in git so that people know how to make one. Simplifies my deployment.

[aarcro]

@drummonds We keep our secrets in credstash, and use the name of the key in zappa_settings. It's a django app, so settings.py pulls them all in one time during instancing.

[drummonds]

@aarco Many thanks - that is probably better so will look at migrating to it.

[texonidas]

Was there ever any progress made on this? We've run into this issue a few times now, in that the our current place for storing sensitive information is Gitlab CI, and the inability to inject variables from there into Lambda directly is a bit frustrating. We're currently using the S3 remote solution, which is less than ideal. Given Zappa precompiles the code before deploying to Lambda (afaik), surely there is a way to inject environment variables into that process?

[davidzajac1]

I ran into an issue like this where I didn't want an API Key in the zappa_settings.json, but also didn't want the hassle of having to update files in an S3 bucket and came up with this solution using GitHub Actions and the action-set-json-field Workflow. It reads the variable from the GitHub repo secrets and manually adds it to the zappa_settings.json file within the GitHub runner before running zappa update.

name: Dev Deploy Backend

on:
  push:
    branches: dev

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - name: Checkout Branch
        uses: actions/checkout@v2.5.0
      - name: Set up Python 3.9
        uses: actions/setup-python@v4
        with:
          python-version: 3.9
      - name: Add environment variables to Zappa Config
        uses: jossef/action-set-json-field@v2.1
        with:
          file: ./backend/zappa_settings.json
          field: dev.aws_environment_variables.API_KEY
          value: ${{ secrets.API_KEY }}
      - name: Install pipenv
        run: pip install pipenv
      - name: Install Pipenv dependencies
        run: pipenv install -d
      - name: Run Zappa Update
        run: pipenv run zappa update dev

Note for some reason you have to specify the full path (./backend/zappa_settings.json) and can't just set the working-directory to "backend" and the file to "zappa_settings.json".