Best Practice Question: How to handle "third-party" providers?

[jhoblitt]

I've been wrestling with the best way to handle the sl1pm4t fork of the kubernetes provider and have been trying to avoid using TF_PLUGIN_CACHE_DIR so that behavior is, ideally, the same for a dev running terragrunt locally and under ci/cd.

My current approach is to use a before_hook to trigger a makefile in the same dir as the terraform.tfvars for the deployment. Eg.,

terragrunt = {
  include {
    path = "${find_in_parent_folders()}"
  }

  terraform {
    source = "git::https://github.com/....git//?ref=master"

    before_hook "tf_plugins" {
      commands = ["init", "init-from-module"]

      run_on_error = false

      execute = [
        "bash", "-lc", "cd ${get_tfvars_dir()}; make"
      ]
    }
  } # terraform
}

UNAME := $(shell uname -s | tr A-Z a-z)
DL_DIR = downloads
ARCH = amd64
TF_PLUG_DIR := terraform.d/plugins/$(UNAME)_$(ARCH)

# the filename in the archive as an extra _x4 suffix
TF_K8S_ZIP_VER = v1.3.0-custom
TF_K8S_BIN_VER = v1.3.0-custom_x4
TF_K8S_ZIP_FILE := terraform-provider-kubernetes_$(TF_K8S_ZIP_VER)_$(UNAME)_$(ARCH).zip
TF_K8S_BIN_FILE := terraform-provider-kubernetes_$(TF_K8S_BIN_VER)
TF_K8S_ZIP_DL := $(DL_DIR)/$(TF_K8S_ZIP_FILE)
TF_K8S_BIN := $(TF_PLUG_DIR)/$(TF_K8S_BIN_FILE)
TF_K8S_URL := https://github.com/sl1pm4t/terraform-provider-kubernetes/releases/download/$(TF_K8S_ZIP_VER)/$(TF_K8S_ZIP_FILE)

.PHONY: all
all: $(TF_K8S_BIN)

$(TF_K8S_BIN): | $(TF_K8S_ZIP_DL)
    unzip -j $(TF_K8S_ZIP_DL) $(TF_K8S_BIN_FILE) -d $(TF_PLUG_DIR)

$(TF_K8S_ZIP_DL): | $(DL_DIR) $(TF_PLUG_DIR)
    wget -nc $(TF_K8S_URL) -O $@

$(DL_DIR) $(TF_PLUG_DIR):
    mkdir -p $@

.PHONY: clean
clean:
    -rm -rf $(TF_K8S_BIN)

Is there a better pattern for handling this scenario?

[brikis98]

This is probably the best approach available now. Adding some sort of first-class support to Terragrunt for 3rd party providers is an interesting thought. Ideas/PRs are welcome.

[dynnamitt]

Did anyone add code/PRs (or even blogs/readme's/gist) since you discussed this last year?

[neelakansha85]

Just checking in as well, as to if there any documentation/blog for how to use third party providers such as kubernetes or helm using terragrunt?

[brikis98]

Nothing has changed AFFAIK since the question was asked. We are still open to ideas/PRs!

[barryib]

Hashicorp is working on a provider registry (which is similar to the module registry) to distribute providers easily. Developers will have the ability to publish their "custom" providers.

https://www.terraform.io/docs/registry/providers/docs.html

I think it will probably the way to go for "third-party" providers.

[yorinasub17]

Nice find! I agree that the official solution is the way to go for third-party providers, and in regards to the download feature, probably better to wait for that.

With that said, I think there are still a few more things that terragrunt can do to improve the QoL around providers. It is rare for you to need to use different provider versions across an entire terragrunt project. This leads to a problem where you end up downloading the same provider version over and over again in a single terragrunt run, especially when doing a xxx-all command. For sufficiently large terragrunt projects, this can lead to a long init cycle in the initial run.

I've always thought that it would be nice to have a way for terragrunt to lookup and prefetch all the providers needed in one go, and store them in the plugin cache specified. In the future, this could also do the same thing for modules as well so that you don't end up installing the same module over and over again across your environments.