Vault instance failing to create

[chrislovecnm]

Summary

I am having an issue where the vault instance will not create. The operator is up and running, but when the helm provider tries to create the vault instance it fails

Steps to reproduce the behavior

1. use an existing cluster
2. tf init, plan, apply

Expected behavior

The vault instance is created

Actual behavior

Warning: Helm release "vault-instance" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.

  with module.vault.helm_release.vault-instance[0],
  on modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
  11: resource "helm_release" "vault-instance" {

Error: Vault.vault.banzaicloud.com "vault" is invalid: spec.ingress.annotations: Invalid value: "null": spec.ingress.annotations in body must be of type object: "null"

  with module.vault.helm_release.vault-instance[0],
  on modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
  11: resource "helm_release" "vault-instance" {

Terraform version

The output of terraform version is:

Terraform v1.1.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.75.1
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.5.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.9.0
+ provider registry.terraform.io/hashicorp/local v2.2.2
+ provider registry.terraform.io/hashicorp/null v3.1.1
+ provider registry.terraform.io/hashicorp/random v3.1.2
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Module version

master

Operating system

Linux inside of the container

[chrislovecnm]

I downgraded to using v1.18.11 with a lower version of Terraform and I am getting the same result:

module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Still creating... [40s elapsed]
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Still creating... [50s elapsed]
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Still creating... [1m0s elapsed]
module.eks-jx.module.nginx.helm_release.nginx-ingress[0]: Creation complete after 1m7s [id=nginx-ingress]

Warning: Helm release "vault-instance" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.

  on .terraform/modules/eks-jx/modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
  11: resource "helm_release" "vault-instance" {

Error: Vault.vault.banzaicloud.com "vault" is invalid: spec.ingress.annotations: Invalid value: "null": spec.ingress.annotations in body must be of type object: "null"

  on .terraform/modules/eks-jx/modules/vault/charts.tf line 11, in resource "helm_release" "vault-instance":
  11: resource "helm_release" "vault-instance" {

Here is my main.tf

module "eks-jx" {
  source = "jenkins-x/eks-jx/aws"
  region       = var.region
  use_vault    = var.use_vault
  use_asm      = var.use_asm
  cluster_name = var.cluster_name
  is_jx2       = var.is_jx2
  create_eks   = var.create_eks
  create_vpc   = var.create_vpc
  create_nginx = var.create_nginx
  jx_git_url   = var.jx_git_url
  apex_domain  = var.apex_domain
  tls_email    = var.tls_email
  use_kms_s3   = var.use_kms_s3
  registry     = var.registry

  nginx_chart_version = var.nginx_chart_version
  cluster_version     = var.cluster_version
  enable_backup       = var.enable_backup
  jx_bot_username     = var.jx_bot_username
  jx_bot_token        = var.jx_bot_token
  enable_external_dns = var.enable_external_dns

  jx_git_operator_values = var.jx_git_operator_values
  production_letsencrypt = var.production_letsencrypt

}

My vars file

cluster_name="foo"
cluster_version="1.19"
region="us-west-2"
create_nginx="true"
jx_git_url="https://gitlab.com"
jx_bot_username="chrislovecnm"
enable_backup="false"
apex_domain="api-jx3.foo.com"
tls_email="john@foo.com"
enable_external_dns=true
production_letsencrypt="true"
use_kms_s3="true"
registry="foo.dkr.ecr.us-east-2.amazonaws.com"
jx_git_operator_values = [
    "gitKind: gitlab",
    "environmentGitOwner: foo"
]
nginx_chart_version="4.0.19"
create_eks=false
create_vpc=false
use_vault=true
cluster_in_private_subnet=true

[ankitm123]

I wonder if it's because you are using 1.19 version of kubernetes, is it possible to use 1.21 and see if it works?

[chrislovecnm]

Yes I can do that

[chrislovecnm]

Yep, it works with 1.21. Do you have a support matrix listed?

[ankitm123]

Yep, it works with 1.21. Do you have a support matrix listed?

Atm we dont have one (we support 1.20+ afaict), but we dont support 1.22 yet (but very close to supporting it - a few helm charts need an upgrade)

We will be adding a few e2e tests to help us create a matrix soon.

[chrislovecnm]

I can close this, but should we have a support matrix first. I think there was a change to the crd api between 1.19 and 1.21 … if I recall

[ankitm123]

I can close this, but should we have a support matrix first

Agreed, I am fixing our internal infrastructure this week, and then we plan to add support for 1.22. Once that work is done, I am going to focus on adding kind tests to run tests on different cluster versions on every PR to jx3-version repo.