v0.4.3 vault-secrets-operator crashes when deploying from a manually rendered Helm chart.

[anilpally]

Describe the bug v0.4.3 vault-secrets-operator crashes, also verbs are missing for hcpauth, hcpvaultsecretsapps clusterrole, i expect these to be created with deployment/ CRDs

To Reproduce Steps to reproduce the behavior:

1. Deploy 0.4.3 vault-secrets-operator

See error (vault-secrets-operator logs, application logs, etc.)

E0129 18:15:53.073016 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch v1beta1.HCPVaultSecretsApp: failed to list v1beta1.HCPVaultSecretsApp: hcpvaultsecretsapps.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpvaultsecretsapps" in API group "secrets.hashicorp.com" at the cluster scope W0129 18:16:00.727099 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list v1beta1.HCPVaultSecretsApp: hcpvaultsecretsapps.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpvaultsecretsapps" in API group "secrets.hashicorp.com" at the cluster scope E0129 18:16:00.727258 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch v1beta1.HCPVaultSecretsApp: failed to list v1beta1.HCPVaultSecretsApp: hcpvaultsecretsapps.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpvaultsecretsapps" in API group "secrets.hashicorp.com" at the cluster scope W0129 18:16:04.331736 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list v1beta1.HCPAuth: hcpauths.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpauths" in API group "secrets.hashicorp.com" at the cluster scope E0129 18:16:04.331906 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch v1beta1.HCPAuth: failed to list v1beta1.HCPAuth: hcpauths.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpauths" in API group "secrets.hashicorp.com" at the cluster scope

Expected behavior stable deployment not crashing often, with clusterrole updated for hcpauth/hcpvaultsecretsapps.

Environment ocp 4.14

• vault-secrets-operator version: 0.4.3

Additional context Add any other context about the problem here.

[anilpally]

[athangal@marv2257 ~]$ oc logs vault-secrets-operator-controller-manager-7c6fb6cd5d-khtgr| grep ERROR 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "error": "failed to wait for hcpauth caches to sync: timed out waiting for cache to be synced for Kind v1beta1.HCPAuth"} 2024-01-29T18:46:08Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for v1beta1.HCPVaultSecretsApp Informer to sync"} 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultAuth", "error": "failed to wait for vaultauth caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultAuth"} 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "error": "failed to wait for hcpvaultsecretsapp caches to sync: timed out waiting for cache to be synced for Kind v1beta1.HCPVaultSecretsApp"} 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultpkisecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultPKISecret", "error": "failed to wait for vaultpkisecret caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultPKISecret"} 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultconnection", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultConnection", "error": "failed to wait for vaultconnection caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultConnection"} 2024-01-29T18:46:08Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for v1beta1.HCPAuth Informer to sync"} 2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultauth caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultAuth"} 2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for hcpvaultsecretsapp caches to sync: timed out waiting for cache to be synced for Kind v1beta1.HCPVaultSecretsApp"} 2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultpkisecret caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultPKISecret"} 2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultconnection caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultConnection"} 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "error": "failed to wait for vaultstaticsecret caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultStaticSecret"} 2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultstaticsecret caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultStaticSecret"} 2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "error": "failed to wait for vaultdynamicsecret caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultDynamicSecret"} 2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultdynamicsecret caches to sync: timed out waiting for cache to be synced for Kind v1beta1.VaultDynamicSecret"} 2024-01-29T18:46:08Z ERROR setup problem running manager {"error": "failed to wait for hcpauth caches to sync: timed out waiting for cache to be synced for Kind v1beta1.HCPAuth"}

[benashz]

HI @anilpally - it looks like something may have gone wrong during the installation. Can you provide more details on how you are installing VSO. Are you using Helm, Kustomize or OLM. Also, was this an upgrade or a fresh install?

In case you are using Helm and this was an upgrade, please see https://developer.hashicorp.com/vault/docs/platform/k8s/vso/installation#updating-crds-when-using-helm

[anilpally]

hi @benashz

we convert helm chart into templates, helm template -f vaules.yaml --include-crds --output-dir /tmp/vault-secrets-operator

Argocd apply manifest under /tmp/vault-secrets-operator

$ ls vault-secrets-operator/templates/ job.yaml metrics-service.yaml secrets.hashicorp.com_vaultconnections.yaml leader-election-rbac.yaml proxy-rbac.yaml secrets.hashicorp.com_vaultdynamicsecrets.yaml manager-config.yaml secrets.hashicorp.com_hcpauths.yaml secrets.hashicorp.com_vaultpkisecrets.yaml manager-rbac.yaml secrets.hashicorp.com_hcpvaultsecretsapps.yaml secrets.hashicorp.com_vaultstaticsecrets.yaml metrics-reader-rbac.yaml secrets.hashicorp.com_vaultauths.yaml serviceaccount.yaml

$ pwd /vault-secrets-operator-config $ ls templates/ default-vault-connection.yaml deployment.yaml namespace.yaml ocp-vault-connection.yaml secret_dockerextnexusread.yaml secret_vault-ca.yaml

[anilpally]

@benashz can you let us know in which order we should apply, so i can annotate them in the order arogocd applies.

[anilpally]

any update?

[benashz]

@anilpally, it looks you are using a non standard installation method by rendering the Helm chart to k8s manifests. In theory that might work, but it is not supported. We currently only support installing VSO from the Helm chart (using helm), the OLM package, or Kustomize.