Wait for tiller to be to be running

[botzill]

Hi.

I think a good option would be to wait for the tiller to be up and running. I'm using current module like:

module "tiller" {
  source = "git::https://github.com/lsst-sqre/terraform-tinfoil-tiller.git//?ref=master"

  namespace = "kube-system"
  service_account = "tiller"
  tiller_image = "gcr.io/kubernetes-helm/tiller:v2.12.2"
}

provider "helm" "k8s_helm" {
  version = "~> 0.7.0"

  install_tiller = false
  debug = true
  service_account = "${module.tiller.service_account}"
  namespace = "${module.tiller.namespace}"

  kubernetes {
    host = "${digitalocean_kubernetes_cluster.k8s.endpoint}"

    client_certificate = "${base64decode(digitalocean_kubernetes_cluster.k8s.kube_config.0.client_certificate)}"
    client_key = "${base64decode(digitalocean_kubernetes_cluster.k8s.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(digitalocean_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate)}"
  }
}

and I can't set depends_on because this works only on resources. But inside module we can add this options.

Thx.

[eimarfandino]

+1

[isen-ng]

@botzill how do you even do that in the module? I can't figure out how to wait for the deployment to be ready before proceeding.

the deployment is clearly successful before the next step starts, but the next step still does not complete because the the deployment isn't ready.

module.k8s_helm_init.kubernetes_deployment.tiller_deploy: Creation complete after 0s (ID: kube-system/tiller-deploy)

module.k8s_sealed_secrets.helm_release.sealed_secrets: Creating...
 ...

Error: Error applying plan:

1 error(s) occurred:

* module.k8s_sealed_secrets.helm_release.sealed_secrets: 1 error(s) occurred:

* helm_release.sealed_secrets: error creating tunnel: "could not find a ready tiller pod"

[isen-ng]

In the end, i stopped using remote tiller altogether.

I installed this: https://github.com/rimusz/helm-tiller

and set my helm provider to:

provider "helm" {
  version = "~> 0.9"

  # use local tiller
  # helm plugin install https://github.com/rimusz/helm-tiller
  # helm tiller start-ci
  host = "127.0.0.1:44134"
  install_tiller = false

  kubernetes {
   ...
  }
}

No need to create service accounts no need to deploy tiller. (btw, this is the way to do in helm v3 too)

[botzill]

Hi @isen-ng. In the end I'm using terragrunt and it's module dependencies, which works when I set

dependencies {
    paths = ["../tiller"]
  }

Now, you second solution about plugin, I see that it may be a solution but if I want to use a tool like https://keel.sh/ then I can't, right?

[isen-ng]

@botzill I'm not entirely sure how keel.sh works, but I'm using weave-flux as my CD tool (deployed using helm), and it continues to work

edit: weave-flux needs tiller_deploy

[Jancis]

I'd gladly use the terraform-tinfoil-tiller module, but i have the same issue. Is there any way to fix this in code?

[jhoblitt]

What version of the kubernetes providers are you folks using? The kubernetes_deployment resource should wait for the deploy to be up but was broken until fairly recently.

[jhoblitt]

FYI that helm v3 had an alpha release a few weeks ago, so hopefully this module won't be necessary at all soon.

[isen-ng]

In the end, I ended up using tillerless helm (https://rimusz.net/tillerless-helm) for my terraform code, and still install remote tiller (because weaveflux depends on it until helm v3 arrives proper).

I have no more issues because the terraform code no longer needs to "wait" until tiller is deployed before proceeding with other helm blocks.

edit: This is old (but working) code as I plan to migrate to using tinfoil-tiller because helm.kubernetes provider does not support the exec block...

main.tf

data "external" "local_tiller" {
  program = ["sh", "${path.module}/files/local-tiller.sh"]
}

provider "helm" {
  version = "~> 0.9.0"

  # use local tiller
  host = "${data.external.local_tiller.result["helm_host"]}"

  # need to install tiller remotely for flux-helm-operator to work
  # however, terraform will still use local tiller
  service_account = "${var.tiller_service_account}"
  namespace = "${var.tiller_namespace}"
  tiller_image = "gcr.io/kubernetes-helm/tiller:v2.12.3"
  install_tiller = true

  ...

  kubernetes {
    ...
  }
}

local-tiller.sh

#!/usr/bin/env bash

set -e

helm plugin install https://github.com/rimusz/helm-tiller > /dev/null 2>&1 || true
helm tiller stop > /dev/null 2>&1 || true
helm tiller start-ci > /dev/null 2>&1

HELM_HOST=$(helm tiller env | cut -d "=" -f 2)

jq -n --arg helm_host "${HELM_HOST}" '{"helm_host":$helm_host}'