MP fails to deploy when running `make tsb`

[nacx]

When running the make tsb command to deploy everything at once, the make k8s stage completes properly but the MP creation fails.

The following error is seen in the logs int he middle of the process:

│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│
│   with module.cert-manager.helm_release.cert_manager[0],
│   on ../../modules/addons/cert-manager/main.tf line 22, in resource "helm_release" "cert_manager":
│   22: resource "helm_release" "cert_manager" {
│
╵
╷
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│
│   with module.es.helm_release.elasticsearch,
│   on ../../modules/addons/elastic/main.tf line 16, in resource "helm_release" "elasticsearch":
│   16: resource "helm_release" "elasticsearch" {
│
╵
╷
│ Error: cert-manager/istiod-cacerts failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp [::1]:80: connect: connection refused
│
│   with module.tsb_mp.kubectl_manifest.manifests_certs[1],
│   on ../../modules/tsb/mp/main.tf line 33, in resource "kubectl_manifest" "manifests_certs":
│   33: resource "kubectl_manifest" "manifests_certs" {
│
╵
╷
│ Error: cert-manager/tsb-server-cert failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp [::1]:80: connect: connection refused
│
│   with module.tsb_mp.kubectl_manifest.manifests_certs[0],
│   on ../../modules/tsb/mp/main.tf line 33, in resource "kubectl_manifest" "manifests_certs":
│   33: resource "kubectl_manifest" "manifests_certs" {
│
╵
╷
│ Error: Post "http://localhost/api/v1/namespaces": dial tcp [::1]:80: connect: connection refused
│
│   with module.tsb_mp.kubernetes_namespace.tsb,
│   on ../../modules/tsb/mp/main.tf line 38, in resource "kubernetes_namespace" "tsb":
│   38: resource "kubernetes_namespace" "tsb" {
│
╵
╷
│ Error: Get "http://localhost/api/v1/namespaces/cert-manager/secrets/selfsigned-ca": dial tcp [::1]:80: connect: connection refused
│
│   with module.tsb_mp.data.kubernetes_secret.selfsigned_ca,
│   on ../../modules/tsb/mp/main.tf line 46, in data "kubernetes_secret" "selfsigned_ca":
│   46: data "kubernetes_secret" "selfsigned_ca" {
│
╵
╷
│ Error: Get "http://localhost/api/v1/namespaces/cert-manager/secrets/tsb-server-cert": dial tcp [::1]:80: connect: connection refused
│
│   with module.tsb_mp.data.kubernetes_secret.tsb_server_cert,
│   on ../../modules/tsb/mp/main.tf line 54, in data "kubernetes_secret" "tsb_server_cert":
│   54: data "kubernetes_secret" "tsb_server_cert" {
│
╵
╷
│ Error: Get "http://localhost/api/v1/namespaces/cert-manager/secrets/istiod-cacerts": dial tcp [::1]:80: connect: connection refused
│
│   with module.tsb_mp.data.kubernetes_secret.istiod_cacerts,
│   on ../../modules/tsb/mp/main.tf line 62, in data "kubernetes_secret" "istiod_cacerts":
│   62: data "kubernetes_secret" "istiod_cacerts" {
│

Which leads to the final error when creating the MP.

│ Error: Unsupported attribute
│
│   on main.tf line 40, in module "tsb_cp":
│   40:   tsb_cacert                 = data.terraform_remote_state.tsb_mp.outputs.tsb_cacert
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "tsb_cacert".
╵
╷
│ Error: Unsupported attribute
│
│   on main.tf line 41, in module "tsb_cp":
│   41:   istiod_cacerts_tls_crt     = data.terraform_remote_state.tsb_mp.outputs.istiod_cacerts_tls_crt
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "istiod_cacerts_tls_crt".
╵
╷
│ Error: Unsupported attribute
│
│   on main.tf line 42, in module "tsb_cp":
│   42:   istiod_cacerts_tls_key     = data.terraform_remote_state.tsb_mp.outputs.istiod_cacerts_tls_key
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "istiod_cacerts_tls_key".
╵
╷
│ Error: Unsupported attribute
│
│   on main.tf line 46, in module "tsb_cp":
│   46:   es_host                    = coalesce(data.terraform_remote_state.tsb_mp.outputs.es_ip, data.terraform_remote_state.tsb_mp.outputs.es_hostname)
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "es_ip".
╵
╷
│ Error: Unsupported attribute
│
│   on main.tf line 46, in module "tsb_cp":
│   46:   es_host                    = coalesce(data.terraform_remote_state.tsb_mp.outputs.es_ip, data.terraform_remote_state.tsb_mp.outputs.es_hostname)
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "es_hostname".
╵
╷
│ Error: Unsupported attribute
│
│   on main.tf line 48, in module "tsb_cp":
│   48:   es_password                = data.terraform_remote_state.tsb_mp.outputs.es_password
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "es_password".
╵
╷
│ Error: Unsupported attribute
│
│   on main.tf line 49, in module "tsb_cp":
│   49:   es_cacert                  = data.terraform_remote_state.tsb_mp.outputs.es_cacert
│     ├────────────────
│     │ data.terraform_remote_state.tsb_mp.outputs is object with 3 attributes
│
│ This object does not have an attribute named "es_cacert".
╵

If the make targets are run individually instead (make k8s, make tsb_mp, make tsb_cp), then the deployment succeeds.

[smarunich]

what is your terraform.tfvars file - I have not hit this issue with the latest main? btw what is your terraform version? and did you do make destroy that did clean up terraform cache and lock files...?

p.s. submitted the bug template, let me know your thoughts about: https://github.com/smarunich/tetrate-service-bridge-sandbox/pull/137

[nacx]

Yes, I started from a completely clean checkout. My Terraform version is:

$ terraform version
Terraform v1.3.2
on darwin_arm64

Your version of Terraform is out of date! The latest version
is 1.3.3. You can update by downloading from https://www.terraform.io/downloads.html

And my tfvars:

{
    "name_prefix": "nacx-test",
    "tsb_fqdn": "nacx-test.tetrate.private",
    "tsb_version": "1.5.3",
    "tsb_image_sync_username": "ignasi-barrera",
    "tsb_image_sync_apikey": "<API key>",
    "jumpbox_machine_type": "n2-standard-2",
    "tsb_password": "admin",
    "tsb_mp": {
        "cloud": "gcp",
        "cluster_id": 0
    },
    "tsb_org": "tetrate",
    "mp_as_tier1_cluster": false,
    "aws_k8s_regions": [
    ],
    "azure_k8s_regions": [
    ],
    "gcp_k8s_regions": [
        "europe-southwest1"
    ]
}

Environment:

$ sw_vers
ProductName:    macOS
ProductVersion: 12.6
BuildVersion:   21G115

$ uname -a
Darwin mbp 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:20:05 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T8101 arm64

[smarunich]

ack, will validate later today, meanwhile just have completed a clean run of the below:

❯ terraform version
Terraform v1.3.3
on darwin_amd64

tfvars

{
    "name_prefix": "r153d7",
    "tsb_fqdn": "r153d7.gcp.cx.tetrate.info",
    "tsb_version": "1.5.3",
    "tsb_image_sync_username": "sergey-marunich",
    "tsb_image_sync_apikey": "<>",
    "tsb_password": "<>",
    "tsb_mp": {
        "cloud": "gcp",
        "cluster_id": 0
    },
    "tsb_org": "tetrate",
    "aws_k8s_regions": [
    ],
    "azure_k8s_regions": [
    ],
    "gcp_k8s_regions": [
        "us-west1",
        "us-west1"
    ]
}

Environment

❯ sw_vers
ProductName:    macOS
ProductVersion: 12.5
BuildVersion:   21G72
❯ uname -a
Darwin aurora.local 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:25 PDT 2022; root:xnu-8020.140.41~1/RELEASE_X86_64 x86_64

[nacx]

Ok, I've upgraded terraform to fully match your environment and it worked now... I don't think, though this is related to the terraform version at all, and it's likely that I was under the impression all internal status was cleaned up while it wasn't. I'll close for now and reopen if I can reproduce it. Thx!