projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
6.04k stars 1.35k forks source link

[v3.28.2 -> v3.29.1] tigera-operator: Pods fail to start due to missing default Tier resource #9520

Open SB-MFJ opened 3 days ago

SB-MFJ commented 3 days ago

Expected Behavior

Updating to 3.29.x should create the default Tier resource and update CRD's.

Current Behavior

attempting to update the tigera-operator installation from 3.28.2 to 3.29.1 pods spin up with

Failed to initialize datastore error=resource does not exist: Tier(default) with error: the server could not find the requested resource (post Tiers.crd.projectcalico.org)

I tried looking for what the default level 1,000,000 to manually create it but was not able to find it and I'm not seeing that CRD under crd.projectcalico.org after updating to 3.29.1.

Steps to Reproduce (for bugs)

We manage things in terragrunt/tofu, this is all the installation values we specify:

# terragrunt.hcl
# ...
inputs = {
  openid_url = dependency.k8_cluster.outputs.openid_url
  openid_arn = dependency.k8_cluster.outputs.openid_arn

  chart_version = "3.28.2"
}
# main.tf
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

locals {
  namespace_tigera = "tigera-operator"
}

resource "kubernetes_namespace" "namespace_tigera" {
  metadata {
    labels = {
      "kubernetes.io/metadata.name" = local.namespace_tigera
      name = local.namespace_tigera
      "pod-security.kubernetes.io/enforce" = "privileged"
    }

    name = local.namespace_tigera
  }
}

resource "helm_release" "tigera-operator" {
  name       = "tigera-operator"
  namespace  = kubernetes_namespace.namespace_tigera.metadata[0].name
  repository = "https://projectcalico.docs.tigera.io/charts"
  chart      = "tigera-operator"
  version    = var.chart_version

  # https://projectcalico.docs.tigera.io/reference/installation/api#operator.tigera.io/v1.InstallationSpec
  values = [<<-VALUES
  installation:
    kubernetesProvider: EKS
    calicoNetwork:
      bgp: Disabled
      linuxDataplane: Iptables
    cni:
      ipam:
        type: AmazonVPC
      type: AmazonVPC
    flexVolumePath: None
    nodeUpdateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    variant: Calico
  VALUES
  ]
}

Context

Cannot update, need calico for outside k8s comms.

Your Environment

terragrunt version v0.69.0
OpenTofu v1.8.5
on darwin_arm64
+ provider registry.opentofu.org/hashicorp/aws v5.77.0
+ provider registry.opentofu.org/hashicorp/helm v2.16.1
+ provider registry.opentofu.org/hashicorp/kubernetes v2.33.0