[FEATURE] Support environment variable configuration for CLI commands

[bgrams]

Is your feature request related to a problem? Please describe.

A common pattern in CICD pipelines is to use environment variables to configure the behavior of a predefined script. Our team leverages this pattern and generally make heavy use of variable inheritance to configure individual behaviors at various layers of the pipeline config hierarchy (e.g. template, pipeline, base job, job).

We would like to see first-class support for this pattern in the terramate CLI in order to simplify our own scripts. There is already precedent for this with the TM_DISABLE_SAFEGUARDS variable and the code change is trivial.

Describe the solution you'd like

Support additional environment variables to configure common terramate CLI flags found here. Examples would be TM_CHDIR for --chdir or TM_TAGS for --tags etc.

We would like to also see a similar implementation for subcommand flags (e.g. TM_RUN_PARALLEL=5 for terramate run etc.) but understand that this may come in a different form. For example, Terraform uses TF_CLI_ARGS_<subcommand>=<flags>, which I personally don't love but it works. However this method doesn't work well with inheritance given that it is a all-or-nothing definition.

I'm also not sure how this would work for shared subcommands such as terramate run and terramate script run since ideally we'd want the vars to be shared, but perhaps run args should be a special case.

Command-line flags should take precedence over env vars. It appears this is already the default behavior for the cli parser.

Describe alternatives you've considered

We build the terramate flags directly in the common scripts, which works fine but is messy. Basic example below (Gitlab CI). We'd like to keep the interface that is used by the Project Pipeline but get rid of custom logic in the Shared Template.

# Shared template
.base:
  before_script:
  - export TM_CLI_ARGS="${TM_TAGS:+--tags $TM_TAGS} <etc>"
  - export TM_RUN_ARGS="${TM_RUN_PARALLEL:+--parallel $TM_RUN_PARALLEL} <etc>"

.init:
  extends: .base
  script:
  - terramate $TM_CLI_ARGS run $TM_RUN_ARGS terraform init
...

# Project pipeline
include:
- project: shared-templates
  file: terramate.yml

# Run all terramate run commands with parallelism=5
variables:
  TM_RUN_PARALLEL: '5'

# Base jobs inherited by init, plan, apply etc. for consistency
.dev:
  variables:
    TM_TAGS: dev

.stg:
  variables:
    TM_TAGS: stg

init-dev:
  extends: [.init, .dev]

init-stg:
  extends: [.init, .stg]

...

Additional context I'd be open to making this contribution

[bgrams]

Seems related to #951 but I understand these to be distinct.

[bgrams]

@mariux given these new labels, should I open a PR?

As mentioned my only discussion point is what we want subcommand vars to look like, but I don't think we need to solve for the general case rn given that this feature is mostly useful for automation and many commands are not used in that context. So in that case the proposal is to only add support for TM_<VAR> and TM_RUN_<VAR> args, the latter of which would apply to both terramate run and terramate script run commands.

I'd like to confirm before review given that this is a user-facing change. Ref - https://github.com/terramate-io/terramate/commit/d7cc474ec55a926ff094836f9aa9731c0dac48fe.

Thoughts?

+@i4ki @soerenmartius

[i4ki]

Hey @bgrams

Yes, we will go forward with this feature and we would be very happy if you could contribute. It makes sense to only handle the cases useful for automation for now, and then later we can improve if needed. I think having a TM_ARG_ prefix is better to not pollute the TM_ namespace but I will confirm this with the team today. Please open a PR, when you have time, and we discuss the feature there.

Thank you very much!

[i4ki]

Hey, the team confirmed it would be better to have TM_ARG_ as prefix.

[uberjew666]

Support additional environment variables to configure common terramate CLI flags found here. Examples would be TM_CHDIR for --chdir or TM_TAGS for --tags etc.

This is exactly what I came to request. It's a pain without these in our Gitlab CI pipeline.