pretf init fails if using *.tfvars.py

[ryanm101]

The following works fine if i move 'environement' into common.auto.tfvars.py

pretf init || exit 1
[pretf] create: s3.tf.json could not be processed
[pretf] create: lambda.tf.json could not be processed
[pretf] create: s3.tf.py cannot access var.environment because it has no value

backend.tf.py

import os
from pretf.aws import provider_aws
from pretf.aws import terraform_backend_s3
from pretf import log

# Get Env as we dont know it yet
BACKEND = os.getenv("TFENV", "dev")
print(f"Backend: {BACKEND}")

# Override - We cant get passed the backend create prompt automatically.
#            So we replace the function temporarily.
log_accept_orig = log.accept

def log_accept_override(x):
    return True

def pretf_blocks(var):
    yield provider_aws(region=var.aws_region, version=var.aws_provider_version)
    log.accept = log_accept_override
    yield terraform_backend_s3(
        bucket=f"{var.service}-tfstate{var.postfix[BACKEND]}",
        dynamodb_table=f"{var.service}-tfstate{var.postfix[BACKEND]}",
        key="terraform.tfstate",
        region=var.aws_region,
        encrypt=True,
    )
    log.accept = log_accept_orig

common.auto.tfvars.py

def pretf_variables():
    yield {
        "bucket_name": "mybucket"
    }
    # Common
    yield {
        "postfix": {"dev": "-dev", "prd": ""},
    }

dev.tfvars.py

def pretf_variables():
    yield {
        "environment": "dev",
    }

s3.tf.py

from pretf.api import block
def pretf_blocks(var):
    bucket = yield block("resource", "aws_s3_bucket", "bucket", {
        "bucket": f"{var.bucket_name}{var.postfix[var.environment]}",
        "versioning": {"enabled": False},
        "server_side_encryption_configuration": {
          "rule": {
            "apply_server_side_encryption_by_default": {
              "sse_algorithm": "AES256"
            }
          }
        }
    })

variables.tf.py

from pretf.api import block
strCommonVars = ["postfix", "environment"]
strBucketVars = ["bucket_name"]

def pretf_blocks():
    for strVar in strCommonVars:
        yield block("variable", strVar, {})
    for strVar in strBucketVars:
        yield block("variable", strVar, {})

[ryanm101]

Work around: common.auto.tfvars.json

import os
def pretf_variables():
    BACKEND = os.getenv('TFENV', "dev")

    # ENV Specific
    print(f"Loading Vars: {BACKEND}")
    if BACKEND == "dev":
        yield {
            "environment": "dev",
        }
    elif BACKEND == "prd":
        yield {
            "environment": "prd",
        }

[raymondbutcher]

I tried this and it worked as expected. It doesn't load values from dev.tfvars.py (or dev.tfvars.json) automatically. Try running it with -var-file=dev.tfvars.json or renaming dev.tfvars.py to dev.auto.tfvars.py or terraform.tfvars.py so it gets loaded automatically.

The variable loading logic in Pretf should match what is described here, in particular the variable definition precedence section at the bottom shows how Terraform works. I'm pretty sure that Pretf is working correctly and in line with how Terraform itself handles variables.

I'm closing this because I think it's working correctly. Feel free to reopen if you think I've got it wrong.

[ryanm101]

@raymondbutcher you cant use -var-file= with init

The issue is that pretf builds all the json 1st even though in the init stage the only relevant part is the backend.

[raymondbutcher]

Ah, interesting.

I ran terraform init -var-file=dev.tfvars.json and it worked. Terraform seems to ignore this argument, at least right now, but I wouldn't blame them if they made it more strict in the future and it stopped working.

I use Pretf to dynamically generate the backend configuration that terraform init uses. I use variables to generate it.

It's getting late here so I'll think about this more another time.

[ryanm101]

No worries, I'm on a plane anyway. :-)

[ryanm101]

Added my backend.tf.py to the original description (forgot to include it which is likely what caused the initial confusion)