Circle CI context environment variables not being interpolated in 3.0.0

[wbreza]

Orb Version 3.0.0

Describe the bug

It appears that starting in 3.0.0 of the orb Circle CI context variables are no longer being interpolated in both backend-config and var parameters

To Reproduce

1. Create a new Circle CI context with the following:

   • TF_RESOURCE_GROUP_NAME =
   • TF_STORAGE_ACCOUNT_NAME =
   • TF_CONTAINER_NAME =
   • TF_KEY_NAME =
   2. Reference your context variables in your job config:

     - terraform/init:
         backend: true
         backend_config:
           "resource_group_name=${TF_RESOURCE_GROUP_NAME}, \
           storage_account_name=${TF_STORAGE_ACCOUNT_NAME}, \
           container_name=${TF_CONTAINER_NAME}, \
           key=${TF_KEY_NAME}"

2. Execute your CircleCI workflow/job

Expected behavior

Circle CI context values should be interpolated before being executed in scripts.

Actual behavior

In our case we are using the Azure Storage account as backend remote state and are getting an error: Failed to get existing workspaces: Error retrieving keys for Storage Account "${TF_STORAGE_ACCOUNT_NAME}"...

Where the environment variables are not getting interpolated before script is executed

Additional context

I believe this all stems from the refactor of when the scripts were moved out from being inline to referenced shell scripts. You are creating TF_ variables for your script execution but the input of those variables is not interpolated and the environment variable notations are being sent in verbatim.

[neerajmandal]

Same

[BSamodien]

Troubling

[punkdata]

@wbreza Thanks for opening this issue and your great feedback. We'll investigate.

[mpraveenreddie]

Any updates on this issue? @punkdata

[wbreza]

@punkdata - Any updates on this issue?

[wbreza]

@punkdata - My team is in the process of creating our own custom orb. We prefer to use external shell scripts as well and found a work around for the same string interpolation issue. Seems like the environment variable mapping in Circle CI doesn't work as expected and doesn't interpolate environment variables that exists within the parameters.

As a work around we have setup a pre-script that manually exports the environment variables and then sources them into the scripts before the next steps.

Found reference from Circle CI docs as well.

ex)

steps:
  - run:
      name: Setup Environment Variables
      command: |
        echo "export EXAMPLE_VAR_1=<< parameters.example-var-1>>" >> $BASH_ENV
        echo "export EXAMPLE_VAR_2=<< parameters.example-var-2 >>" >> $BASH_ENV
        source $BASH_ENV
  - run:
      name: Fancy things
      command: <<include(scripts/example-script.sh)>>

[punkdata]

@punkdata - My team is in the process of creating our own custom orb. We prefer to use external shell scripts as well and found a work around for the same string interpolation issue. Seems like the environment variable mapping in Circle CI doesn't work as expected and doesn't interpolate environment variables that exists within the parameters.

As a work around we have setup a pre-script that manually exports the environment variables and then sources them into the scripts before the next steps.

Found reference from Circle CI docs as well.

ex)

steps:
  - run:
      name: Setup Environment Variables
      command: |
        echo "export EXAMPLE_VAR_1=<< parameters.example-var-1>>" >> $BASH_ENV
        echo "export EXAMPLE_VAR_2=<< parameters.example-var-2 >>" >> $BASH_ENV
        source $BASH_ENV
  - run:
      name: Fancy things
      command: <<include(scripts/example-script.sh)>>

@KyleTryon @gmemstr can either of you take a look at @wbreza suggestion? I will not be able to hack on this for some time and this needs some love.

@wbreza thank you for the suggestions.

[MohamedAnouar]

I can confirm that I had the same bug when I tried using the var section. Simple example :

• terraform/apply: attach-workspace: true context: terraform workspace: prod var: IMAGE_TAG=${CIRCLE_SHA1} The variable IMAGE_TAG wasn't interpolated and the variable that was retrieved in terraform is the simple string : "${CIRCLE_SHA1}".

[andresrv94]

I'm having the same issue, I need to use a path with an environment variable and it doesn't interpolate it.

[sdc-wag]

Experiencing the same and I'll follow this issue. Hopefully a fix will be made 😊 Thanks for a nice orb either way.

[gavinclarkeuk]

Came up with a reasonable workaround for this issue, by adding a job that writes a tfvars file from the env vars, and passing that through to the terraform jobs.

jobs:
  write-tf-vars:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "db_user=\"${DB_USER}\"" >> env.tfvars
      - run: echo "db_password=\"${DB_PASSWORD}\"" >> env.tfvars
      - persist_to_workspace:
          root: .
          paths:
            - env.tfvars

workflows:
  main:
    jobs:
      - write-tf-vars
      - terraform/plan:
          checkout: true
          attach-workspace: true
          workspace-root: terraform
          path: terraform
          var_file: env.tfvars
          requires:
            - write-tf-vars

Note that because of https://github.com/CircleCI-Public/terraform-orb/issues/56 if you specify a path in the terraform workflow step then you must also set the workspace-root to ensure the tfvars file is in a subdirectory of the path.

[jmstone617]

This issue exists for project environment variables, as well

[ksmnv]

@punkdata any plans to finally address this bug?

[KyleTryon]

@ksmnv I am currently working to address all currently reported issues and feature requests. I anticipate a minor update will be released before the end of next week which should address most issues. Followed by potentially a major change if needed to better support terraform cloud. Please keep an eye out over the next two weeks.

[KyleTryon]

A development version of the orb is available for testing. If you are interested and available, we would love your early feedback.

PR: https://github.com/CircleCI-Public/terraform-orb/pull/72 Orb: https://circleci.com/developer/orbs/orb/circleci/terraform?version=dev:997fc3ea0f586a39be65e7c800118ba77669c262

@wbreza there is one change still I believe needed from the example data you shared, the newlines should stay but the backslashed should be removed.

As mentioned in the PR, this affects backend-config and vars. If you are using env vars in other places that are not yet interpolated here, please let us know.

Let us know what you think, thank you!

[ksmnv]

@KyleTryon this version does not work for us as we're passing the $CIRCLE_BRANCH value to the workspace variable. Would be great if this issue could be addressed as well.

[KyleTryon]

@KyleTryon this version does not work for us as we're passing the $CIRCLE_BRANCH value to the workspace variable. Would be great if this issue could be addressed as well.

Got you covered 👍

75

You can preview/test the update here if you wish:

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link: 
https://circleci.com/developer/orbs/orb/circleci/terraform?version=dev:9af39c2d587d9ecf97dc8a19ccc4699d64fed48f

[KyleTryon]

Terraform Orb 3.1.0 is live: https://github.com/CircleCI-Public/terraform-orb/releases/tag/v3.1.0 Please try the latest version of the orb and let us know what you think!

Version 4.0.0 progress will begin soon with improved support for Terraform cloud and some new sensible defaults.

[onenessboy]

@KyleTryon Hi I have defined follwing variables in context for azure remote backedn configuration setup but it says terraform intialized in empty directory , do I need to define any additional variables which is mandatory to intitialze Properly ? by the way we are using an self hosted vm (on azure) to run my workflows TF_CONTAINER_NAME  TF_KEY_NAME TF_RESOURCE_GROUP_NAME TF_STORAGE_ACCOUNT_NAME

[ksmnv]

@KyleTryon works like a charm, thank you!

[onenessboy]

@ksmnv can you please help with sample config.yml you have to connect to azure and read remote backend... I am helpless to know what are required environment variables that I need to set ? as of now I have only the below. Any thing missing.?. my plan step failing saying no path exist... In documentation for this orb, it will be really good, if example value added for those fields for init and plan steps TF_CONTAINER_NAME TF_KEY_NAME TF_RESOURCE_GROUP_NAME TF_STORAGE_ACCOUNT_NAME

[ksmnv]

@onenessboy I won't be much of a help unfortunately as we're using AWS instead of Azure for our infrastructure.

[onenessboy]

@onenessboy I won't be much of a help unfortunately as we're using AWS instead of Azure for our infrastructure.

No problem.

@KyleTryon can you pleas help with sample confg file which can read remote azure terraform file init,plan,apply, I am getting error saying path does not exist even though I have added TF_PARAM_PATH in environment variable Just would like to know what are env variables I need to add for this orb to work?

[tk3fftk]

Similar issue may be remaining terraform/install ..

[zallesov]

Landed here too with the similar issue. only way to supply backend_config_file or var_file is by providing a path as string. env var interpolation is not working. but parameter interpolation works thou

backend_config_file: "../config/<< parameters.environment >>/config-<< parameters.environment >>-backend.tfbackend"

bit ugly but works. thou I'd prefer to set it via env var

TF_PARAM_BACKEND_CONFIG_FILE: "../config/<< parameters.environment >>/config-<< parameters.environment >>-backend.tfbackend"

That works not.