search

suzuki-shunsuke / tfaction

Framework for Monorepo to build high level Terraform Workflows by GitHub Actions
https://suzuki-shunsuke.github.io/tfaction/docs/
MIT License
262 stars 39 forks source link

The secrets configured in tfaction.yaml are not being set as environment variables by export-secrets. #1653

Open 46158n opened 4 months ago

46158n commented 4 months ago

tfaction version

v1.4.0

Overview

The secrets configured in tfaction.yaml are not being set as environment variables by export-secrets. However, when the same secrets configuration is set in tfaction-root.yaml, it works properly.

How to reproduce

tfaction-root.yaml

plan_workflow_name: test
target_groups:
- working_directory: terraform/
  target: terraform/dev/
  aws_region: ap-northeast-1
  s3_bucket_name_tfmigrate_history: tfaction-************
  terraform_plan_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction
  tfmigrate_plan_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction
  terraform_apply_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction
  tfmigrate_apply_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction

tfaction.yaml

secrets:
- env_name: NEW_RELIC_API_KEY
  secret_name: NEWRELIC_API_KEY_DEV

GitHub Actions Workflow

I have created a workflow based on this example as a reference. I am using secrets: inherit to pass secrets to the reusable workflow.

--- test.yaml
name: pull_request_target
on: pull_request_target

concurrency:
  group: ${{ github.workflow }}--${{ github.head_ref }} # github.ref is unavailable in case of pull_request_target
  cancel-in-progress: true

permissions: {}

jobs:
  path-filter:
    uses: ./.github/workflows/wc-path-filter.yaml
    permissions: {}
    secrets:
      gh_app_id: ${{secrets.TFACTION_GITHUB_APP_ID}}
      gh_app_private_key: ${{secrets.TFACTION_GITHUB_APP_PRIVATE_KEY}}

  enable-auto-merge:
    # This job is used for main branch's branch protection rule's status check.
    # If all dependent jobs succeed or are skipped this job succeeds.
    uses: ./.github/workflows/wc-enable-auto-merge.yaml
    needs:
      - status-check
    permissions: {}
    secrets:
      gh_app_id: ${{secrets.TFACTION_GITHUB_APP_ID}}
      gh_app_private_key: ${{secrets.TFACTION_GITHUB_APP_PRIVATE_KEY}}
    if: |
      ! failure() && ! cancelled() && github.event.pull_request.user.login == 'renovate[bot]' && contains(github.event.pull_request.body, ' **Automerge**: Enabled.')

  # This job is used for branch protection rule
  # Add this job to `Status checks that are required`
  status-check:
    runs-on:
      - self-hosted
      - dev
    needs:
      - test
    if: failure()
    steps:
      - run: exit 1

  test:
    uses: ./.github/workflows/wc-test.yaml
    needs: path-filter
    permissions:
      id-token: write
      contents: read
    with:
      ref: ${{needs.path-filter.outputs.merge_commit_sha}}
    secrets: inherit

--- wc-test.yaml
name: test
on:
  workflow_call:
    inputs:
      ref:
        required: false
        type: string

env:
  TFACTION_IS_APPLY: 'false'
  GH_COMMENT_SHA1: ${{inputs.ref}}
  TFCMT_SHA: ${{inputs.ref}}

permissions: {}

jobs:
  setup:
    uses: ./.github/workflows/wc-setup.yaml
    with:
      ref: ${{inputs.ref}}
    secrets:
      gh_app_id: ${{secrets.TFACTION_GITHUB_APP_ID}}
      gh_app_private_key: ${{secrets.TFACTION_GITHUB_APP_PRIVATE_KEY}}
    permissions:
      contents: read

  plan:
    uses: ./.github/workflows/wc-plan.yaml
    needs: setup
    # if services is empty, the build job is skipped
    if: join(fromJSON(needs.setup.outputs.targets), '') != ''
    with:
      targets: ${{needs.setup.outputs.targets}}
      ref: ${{inputs.ref}}
    secrets: inherit
    permissions:
      id-token: write
      contents: read

--- wc-plan.yaml
name: plan
on:
  workflow_call:
    inputs:
      targets:
        required: true
        type: string
      ref:
        required: true
        type: string

jobs:
  plan:
    name: "plan (${{matrix.target.target}})"
    runs-on: ${{matrix.target.runs_on}}
    permissions:
      id-token: write # For OIDC
      contents: read # To checkout private repositories
    env:
      TFACTION_TARGET: ${{matrix.target.target}}
      TFACTION_JOB_TYPE: ${{matrix.target.job_type}}
      GH_COMMENT_SHA1: ${{inputs.ref}}
      TFCMT_SHA: ${{inputs.ref}}
    strategy:
      fail-fast: true
      matrix:
        target: ${{fromJSON(inputs.targets)}}
    steps:
      - uses: actions/checkout@1d96c772d19495a3b5c517cd2bc0cb401ea0529f # v4.1.3
        with:
          ref: ${{inputs.ref}}

      - name: Generate token (aqua)
        id: aqua_installer_token
        uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2.1.0
        with:
          app_id: ${{secrets.TFACTION_GITHUB_APP_ID}}
          private_key: ${{secrets.TFACTION_GITHUB_APP_PRIVATE_KEY}}
          permissions: >-
            {}
          repositories: >-
            []

      - uses: aquaproj/aqua-installer@4551ec64e21bf0f557c2525135ff0bd2cba40ec7 # v3.0.0
        with:
          aqua_version: v2.27.3
        env:
          AQUA_GITHUB_TOKEN: ${{steps.aqua_installer_token.outputs.token}}

      - uses: suzuki-shunsuke/tfaction/export-secrets@c56c96f8587d75ec918c79256b1f9c50361287fd # v1.4.0
        with:
          # If this action doesn't export secrets as you expect, please check if secrets are passed to this action properly.
          # If you use reusable workflows, maybe secrets aren't passed to the reusable workflow.
          # If so, please pass secrets properly.
          # 
          # - https://docs.github.com/en/actions/using-workflows/reusing-workflows#passing-inputs-and-secrets-to-a-reusable-workflow
          # - https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idsecrets
          secrets: ${{toJSON(secrets)}}

      - name: debug
        run: |
          echo ${NEW_RELIC_API_KEY}

      - name: Generate token to download private Terraform Modules
        id: gh_setup_token
        uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2.1.0
        with:
          app_id: ${{ secrets.TFACTION_GITHUB_APP_ID }}
          private_key: ${{ secrets.TFACTION_GITHUB_APP_PRIVATE_KEY }}
          # If you use private registries, contents:read is required
          permissions: >-
            {
              "contents": "read"
            }
          # private repositories hosting private modules
          repositories: >-
            []

      # This is required to download private modules in `terraform init`
      - run: gh auth setup-git
        env:
          GITHUB_TOKEN: ${{steps.gh_setup_token.outputs.token}}

      - name: Generate token
        id: generate_token
        uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2.1.0
        with:
          app_id: ${{secrets.TFACTION_GITHUB_APP_ID}}
          private_key: ${{secrets.TFACTION_GITHUB_APP_PRIVATE_KEY}}
          # issues:write - Create labels
          # pull_requests:write - Post comments and set labels
          # contents:write - Push commits
          permissions: >-
            {
              "pull_requests": "write",
              "issues": "write",
              "contents": "write"
            }
          repositories: >-
            ["${{github.event.repository.name}}"]

      - uses: suzuki-shunsuke/tfaction/setup@c56c96f8587d75ec918c79256b1f9c50361287fd # v1.4.0
        with:
          github_token: ${{steps.generate_token.outputs.token}}
        env:
          GITHUB_TOKEN: ${{steps.generate_token.outputs.token}} # For GitHub Provider

      - uses: suzuki-shunsuke/tfaction/get-target-config@c56c96f8587d75ec918c79256b1f9c50361287fd # v1.4.0
        id: target-config

      - name: Generate token (pull request write)
        id: pull_request_token
        uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2.1.0
        with:
          app_id: ${{secrets.TFACTION_GITHUB_APP_ID}}
          private_key: ${{secrets.TFACTION_GITHUB_APP_PRIVATE_KEY}}
          permissions: >-
            {
              "pull_requests": "write"
            }
          repositories: >-
            ["${{github.event.repository.name}}"]

      - uses: suzuki-shunsuke/tfaction/test@c56c96f8587d75ec918c79256b1f9c50361287fd # v1.4.0
        with:
          github_token: ${{steps.generate_token.outputs.token}}
        env:
          GITHUB_TOKEN: ${{steps.generate_token.outputs.token}} # For GitHub Provider and reviewdog and github-comment

      - uses: suzuki-shunsuke/tfaction/plan@c56c96f8587d75ec918c79256b1f9c50361287fd # v1.4.0
        with:
          github_token: ${{steps.generate_token.outputs.token}}
        env:
          GITHUB_TOKEN: ${{steps.generate_token.outputs.token}} # For GitHub Provider and tfcmt and github-comment

GitHub Actions' log

2024-04-24T09:22:23.9027791Z ##[group]Run suzuki-shunsuke/tfaction/export-secrets@c56c96f8587d75ec918c79256b1f9c50361287fd
2024-04-24T09:22:23.9029200Z with:
2024-04-24T09:22:23.9100101Z   secrets: {
  "NEWRELIC_API_KEY_DEV": "***"
}
2024-04-24T09:22:23.9116554Z env:
2024-04-24T09:22:23.9117384Z   TFACTION_TARGET: terraform/dev/dev
2024-04-24T09:22:23.9118374Z   TFACTION_JOB_TYPE: terraform
2024-04-24T09:22:23.9119441Z   GH_COMMENT_SHA1: 55de72c838e65f04dc7c192c36ccf4bbb6add185
2024-04-24T09:22:23.9120663Z   TFCMT_SHA: 55de72c838e65f04dc7c192c36ccf4bbb6add185
2024-04-24T09:22:23.9121733Z ##[endgroup]
2024-04-24T09:22:24.0047333Z The list of secret names passed to the action: NEWRELIC_USER_KEY_DEV
2024-04-24T09:22:24.0276970Z ##[group]Run echo ${NEW_RELIC_API_KEY}
2024-04-24T09:22:24.0278033Z echo ${NEW_RELIC_API_KEY}
2024-04-24T09:22:24.0334719Z shell: /usr/bin/bash -e {0}
2024-04-24T09:22:24.0335625Z env:
2024-04-24T09:22:24.0336428Z   TFACTION_TARGET: terraform/dev/dev
2024-04-24T09:22:24.0337408Z   TFACTION_JOB_TYPE: terraform
2024-04-24T09:22:24.0338480Z   GH_COMMENT_SHA1: 55de72c838e65f04dc7c192c36ccf4bbb6add185
2024-04-24T09:22:24.0339714Z   TFCMT_SHA: 55de72c838e65f04dc7c192c36ccf4bbb6add185
2024-04-24T09:22:24.0340747Z ##[endgroup]

Expected behaviour

The log states: export the secret NEWRELIC_API_KEY_DEV as the environment variable NEW_RELIC_API_KEY.

Below is the configuration of secrets written in tfaction-root.yaml:

2024-04-24T09:18:02.2130049Z ##[group]Run suzuki-shunsuke/tfaction/export-secrets@c56c96f8587d75ec918c79256b1f9c50361287fd
2024-04-24T09:18:02.2131402Z with:
2024-04-24T09:18:02.2206516Z   secrets: {
  "NEWRELIC_API_KEY_DEV": "***",
}
2024-04-24T09:18:02.2223279Z env:
2024-04-24T09:18:02.2224121Z   TFACTION_TARGET: terraform/dev/dev
2024-04-24T09:18:02.2225146Z   TFACTION_JOB_TYPE: terraform
2024-04-24T09:18:02.2226270Z   GH_COMMENT_SHA1: 5b2974c9ce2d3329b198fe2d9aa5b08b4aa08496
2024-04-24T09:18:02.2227546Z   TFCMT_SHA: 5b2974c9ce2d3329b198fe2d9aa5b08b4aa08496
2024-04-24T09:18:02.2228676Z ##[endgroup]
2024-04-24T09:18:02.3153510Z The list of secret names passed to the action: NEWRELIC_API_KEY_DEV
2024-04-24T09:18:02.3184615Z export the secret NEWRELIC_API_KEY_DEV as the environment variable NEW_RELIC_API_KEY
2024-04-24T09:18:02.3332689Z ##[group]Run echo ${NEW_RELIC_API_KEY}
2024-04-24T09:18:02.3333773Z echo ${NEW_RELIC_API_KEY}
2024-04-24T09:18:02.3392619Z shell: /usr/bin/bash -e {0}
2024-04-24T09:18:02.3393569Z env:
2024-04-24T09:18:02.3394407Z   TFACTION_TARGET: terraform/dev/dev
2024-04-24T09:18:02.3395445Z   TFACTION_JOB_TYPE: terraform
2024-04-24T09:18:02.3396555Z   GH_COMMENT_SHA1: 5b2974c9ce2d3329b198fe2d9aa5b08b4aa08496
2024-04-24T09:18:02.3397855Z   TFCMT_SHA: 5b2974c9ce2d3329b198fe2d9aa5b08b4aa08496
2024-04-24T09:18:02.3399166Z   NEW_RELIC_API_KEY: ***
2024-04-24T09:18:02.3400107Z ##[endgroup]
2024-04-24T09:18:02.3491770Z ***

tfaction-root.yaml

plan_workflow_name: test
target_groups:
- working_directory: terraform/
  target: terraform/dev/
  aws_region: ap-northeast-1
  secrets:
  - env_name: NEW_RELIC_API_KEY
    secret_name: NEWRELIC_API_KEY_DEV
  s3_bucket_name_tfmigrate_history: tfaction-************
  terraform_plan_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction
  tfmigrate_plan_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction
  terraform_apply_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction
  tfmigrate_apply_config:
    aws_assume_role_arn: arn:aws:iam::************:role/tfaction

tfaction.yaml

{}

Actual behaviour

The environment variable NEW_RELIC_API_KEY is not being set.

Important Factoids

This issue seems similar to https://github.com/suzuki-shunsuke/tfaction/issues/1564, where despite the log message in export-secrets stating The list of secret names passed to the action: NEWRELIC_API_KEY_DEV, the environment variable is not being set.

After reviewing the code in the following files:

I couldn't find the part where it retrieves secrets from tfaction.yaml."

Note

No response

suzuki-shunsuke commented 4 months ago

The log looks strange.

2024-04-24T09:22:23.9027791Z ##[group]Run suzuki-shunsuke/tfaction/export-secrets@c56c96f8587d75ec918c79256b1f9c50361287fd
2024-04-24T09:22:23.9029200Z with:
2024-04-24T09:22:23.9100101Z   secrets: {
  "NEWRELIC_API_KEY_DEV": "***"
}
2024-04-24T09:22:24.0047333Z The list of secret names passed to the action: NEWRELIC_USER_KEY_DEV

I'm not sure where NEWRELIC_USER_KEY_DEV (not NEWRELIC_API_KEY_DEV) came from. 🤔

rochana-atapattu commented 1 month ago

I'm getting the same issue I have set the config in tfaction.yaml

Screenshot 2024-07-22 at 22 57 16

and I passed down the secret properly, It shows up in the logs but its not being set as an env vars.

Screenshot 2024-07-22 at 22 58 07