Zarf should manage IaC as a first class citizen

[runyontr]

Is your feature request related to a problem? Please describe. When deploying a full infra stack the wrapper around terraform creates confusion and doesn't integrate with the Kubernetes management zarf provides

Describe the solution you'd like

• IaC treated natively
• Integration between zarf state of IaC and zarf state in Kubernetes and state used for IaC tools
• Zarf deployment of Kubernetes would auto-zarf-init the cluster, enabling ease of use of zarf to manage the Kubernetes environment, or maybe install the zarf controller?
• Allow outputs from terrafrom be inputs to zarf packages

Describe alternatives you've considered Zarf not integrating natively and having to tie the pieces together manually or with separate zarf packages, e.g. via a Plugin system.

Additional context

[RothAndrew]

Recommend we break this down more into some specific use cases that have been identified as difficult without some of the capabilities that zarf brings to the table, as well as identifying the right persona. Are we going after someone who is trying to do IaC in an air gap or in a cloud environment? Depending on which, this could go in very different directions.

What problems are people facing now, using existing tools that are out there, that zarf may solve?

Additionally, I feel it is important as we work this to ensure we preserve the separation of concerns between different layers of the stack. Keeping the infra layer, the platform layer, and the app layer separate is, IMO, a best practice that should be maintained. They each have different forces at play, different upgrade cycles, different permissions boundaries, etc

[Racer159]

Agree on having a use case to test against, and I think @jeff-mccoy was going to work a bit on what a plugin system might look like once the library PR is complete and I think @UncleGedd had some thoughts as well. We could probably pull something together from them to bounce things off of.

[UncleGedd]

What problems are people facing now, using existing tools that are out there, that zarf may solve?

Context from what I'm working on (note we are in a connected, yet regulated, environment): I believe a potential use case for Zarf is in our platform delivery model (the platform is Big Bang + third party apps + Terrform/Terragrunt). One of the potential delivery models is to hand the platform IaC over to app teams and have them deploy the platform themselves. Instead of handing over the entire codebase, I think it could be better to give app teams an immutable, compiled Zarf package.

In this case, Zarf would need to run the Terraform (via Terragrunt) and then deploy the cluster. To achieve adoption though, Zarf needs a value add thats better than simply running the Terraform commands (or writing a script to run the Terraform commands).

Integration between zarf state of IaC and zarf state in Kubernetes and state used for IaC tools

Now we're talking! If you can give me a single view of my entire infrastructure's state, as opposed to having to query AWS + K8s depending on what I'm doing, that could be a major selling point.

or maybe install the zarf controller

Yes, handling drift is a problem (especially with Terraform). If we could have a unified view of the entire infra AND have Zarf detect/correct drift, that could be a big win. Although, this would need to be better than just installing the Flux tf-controller

Other thoughts:

• Need to think about how plan commands will work.
• Need to think about error handling, TF errors out all the time and the stack trace is a mess

Hopefully this provides enough context to start building a use case to test against (again, this is from a connected, cloud perspective). Happy to discuss further

[runyontr]

Interim work to allow for passing variables to Terraform files are being provided in #1171 .

Variables could be passed into a terraform module from a Zarf component via a file:

provider "aws" {
  region = "###ZARF_VAR_AWS_REGION###"
}

terraform {
  required_version = ">= 0.13"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.37.0"
    }
  }
}

While this doesn't allow for leveraging existing terraform code to be Zarfed without some modifications to allow certain deploy time configuration, it does enable it.

[jeff-mccoy]

While this doesn't allow for leveraging existing terraform code to be Zarfed without some modifications to allow certain deploy time configuration, it does enable it.

Haha "zarfed" is a new one. But valid point, this will only work one way, though you could convert it back to regular terraform with a deploy if needed.

[runyontr]

Summary of Updates based on discussions with the team

Long Term Solution

There's resistance to just adding a new noun to Zarf to enable it due to:

• Added supply chain risk with bloat of packages
• Added maintenance demands based on commonly updates.

It might be worth it in the future, but the team wasn't ready to commit to this immediately until the feasibility on #1136 was looked at to debate how that was implemented. There have been some blockers in some tests on that, so the timeline on plugins being available is unknown, and we don't want to block this capability on that.

Short Term plan

As a short term plan to better enable use of Terraform within Zarf, without adding it as a noun, the team is looking to improve the usability around scripts via #1144.

What are the gaps in Scripts that are needed:

• Scripts need different binaries for different OS/Arches. The Matrix (issue needs to be created) would provide the ability to create comparable components for terraform to work on different OS/Arches easily
  • SHASUM of different artifacts based on this OS/Arch instead of just one artifact.
• Need ability to cleanup on failure. When terraform doesn’t deploy well, added hooks to do onFailure scripts would be beneficial
• onDestroy is needed to allow terraform to specify the tf destoy command for cleaning up the environment
• preflight checks for
• extracting single file from an artifact, e.g. how terraform binary is hosted inside a tarball for their releases.

not sure how to migrate existing scripts components over. The proposed plan would be to get this PR approved and merged and then zarf.yaml files looking to manage the IaC deployment here with actions instead of scripts. I think the resulting zarf.yaml would look like this:

kind: ZarfPackageConfig
metadata:
  name: "terraform"
  description: "Run terraform/terragrunt code"

components:
  - name: download-terraform
    required: true
    actions:
      onDeploy:
        after:
          - cmd: rm -f terraform
          - cmd: ./zarf tools archiver decompress tmp/terraform_1.3.1_###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###.zip .
          - cmd: ./zarf tools archiver decompress tmp/terraform-provider-aws_4.33.0_###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###.zip ./tf-plugins/registry.terraform.io/hashicorp/aws/4.33.0/###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###
          - cmd: rm -rf tmp
      # mac m1 terraform aws provider binary
    matrix:
      localOS: [darwin, linux, windows]
      localArch: [amd64, arm64]
    files:
      - source: https://releases.hashicorp.com/terraform-provider-aws/4.33.0/terraform-provider-aws_4.33.0_###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###.zip
        target: tmp/terraform-provider-aws_4.33.0_###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###.zip
      # mac m1 terraform binary
      - source: https://releases.hashicorp.com/terraform/1.3.1/terraform_1.3.1_###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###.zip
        target: tmp/terraform_1.3.1_###ZARF_LOCAL_OS###_###ZARF_LOCAL_ARCH###.zip
  - name: download-tf-module
    required: true
    files:
      # terraform code
      - source: main.tf
        target: main.tf

  - name: terraform-init
    required: true
    actions:
      create:
        action:
        - script: terraform init -plugin-dir=tf-plugins
        - script: mv backend.example backend.tf
        - script: export BUCKET_ID=`(terraform output -raw tfstate_bucket_id)` # this might get fixed with component variables
        - script: sed -i '' 's|bucket.*|bucket         = \"'$BUCKET_ID'\"|' backend.tf
        - script: terraform init -migrate-state -force-copy
      deploy:
        action:
        - script: terraform apply -auto-approve
      destroy:
        action:
        - script: terraform destroy -auto-approve
     files:
      # terraform code
      - source: main.tf
        target: main.tf
      - source: provider.tf
        target: provider.tf
      - source: backend.example
        target: backend.example

After this work is completed, there needs to be work and discovery on:

• how to manage terraform state

• 1137 needs to be completed.

[ntwkninja]

related issue #1141

[jeff-mccoy]

1144 automatically handles migration, deprecation warnings and backwards compatibility in its current state. The resultant zarf.yaml from zarf package create will produce both the scripts andactionssections along with a warning that thescripts` are deprecated.

[jeff-mccoy]

In addition to variable injection provided by #1171, #1144 also includes deploy-time variable injections as environment variables in the form of ZARF_VAR_NAME and TF_VAR_NAME to match the expected env variable format of terraform.