search

redhat-cop / vault-config-operator

An operator to support Haschicorp Vault configuration workflows from within Kubernetes
Apache License 2.0
112 stars 49 forks source link

GitHubSecretEngineConfig referencing vaultSecret fails with panic error #205

Open rafaeltuelho opened 1 year ago

rafaeltuelho commented 1 year ago

Here is my CRD:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: GitHubSecretEngineConfig
metadata:
  name: github2
  namespace: vault-admin
spec:
  authentication: 
    path: kubernetes
    role: policy-admin
  sSHKeyReference:
    vaultSecret:
      path: "kv/data/secrets/janusidp/github-plugin2"
  path: github
  applicationID: 111111

I see this error on vault-admin operator pod:

2023-11-06T16:22:19Z    INFO    Observed a panic in reconciler: interface conversion: interface {} is nil, not string   {"controller": "githubsecretengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "GitHubSecretEngineConfig", "GitHubSecretEngineConfig": {"name":"github2","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "github2", "reconcileID": "a4b2534d-2c5c-4441-8bfe-e688c0619278"}
panic: interface conversion: interface {} is nil, not string [recovered]
    panic: interface conversion: interface {} is nil, not string

goroutine 946 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x18a3560, 0xc003778de0})
    /opt/hostedtoolcache/go/1.19.13/x64/src/runtime/panic.go:884 +0x212
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*GitHubSecretEngineConfig).setInternalCredentials(0xc00067ec40, {0x1d80ca8, 0xc0059c52f0})
    /home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/githubsecretengineconfig_types.go:156 +0x5dd
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*GitHubSecretEngineConfig).PrepareInternalValues(0x1d80ca8?, {0x1d80ca8?, 0xc0059c52f0?}, {0x0?, 0x0?})
    /home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/githubsecretengineconfig_types.go:116 +0x25
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic(0xc003729bb0, {0x1d80ca8, 0xc0059c52f0}, {0x1d94100?, 0xc00067ec40?})
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:93 +0xb6
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile(0xc003729bb0, {0x1d80ca8, 0xc0059c52f0}, {0x1d94100, 0xc00067ec40})
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65 +0x239
github.com/redhat-cop/vault-config-operator/controllers.(*GitHubSecretEngineConfigReconciler).Reconcile(0xc000200380, {0x1d80ca8, 0xc0022a1bf0}, {{{0xc0009689a0?, 0x10?}, {0xc000968996?, 0x40f6e7?}}})
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/githubsecretengineconfig_controller.go:85 +0x3f8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d80ca8?, {0x1d80ca8?, 0xc0022a1bf0?}, {{{0xc0009689a0?, 0x1806740?}, {0xc000968996?, 0x10?}}})

And here is the secret path on vault:

 vault kv get -mount=kv /secrets/janusidp/github-plugin2
============= Secret Path =============
kv/data/secrets/janusidp/github-plugin2

======= Metadata =======
Key                Value
---                -----
created_time       2023-11-06T16:57:17.834324875Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
key    -----DATA-----

It seems the errors happens at this line: https://github.com/redhat-cop/vault-config-operator/blob/a0f03904af92e0a7be7a8dcee6e715189df3bea2/api/v1alpha1/githubsecretengineconfig_types.go#L156

raffaelespazzoli commented 1 year ago

it looks like the path you are passing is wrong (the data segment seems to be at the wrong place) and at the same time, the operator does not handle that situation gracefully. Thanks for reporting this.

rafaeltuelho commented 1 year ago

So, I enabled my KV using v2, so that's why it adds /data to the mount point kv. Does the Operator expect kv v1 (without versioning)?

rafaeltuelho commented 12 months ago

I ran into another situation where the Operator crashes with panic error. This happens when it tries to read a secret that does not exist in the vault yet.

2023-11-30T21:19:41Z    DEBUG   reconcile   {"controller": "secretenginemount", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "SecretEngineMount", "SecretEngineMount": {"name":"github","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "github", "reconcileID": "42c56db7-0989-4a87-ac90-60da4b9ec9a8", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "SecretEngineMount", "namespace": "vault-admin", "name": "github"}}
2023-11-30T21:19:41Z    ERROR   unable to retrieve vault secret {"controller": "quaysecretengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "QuaySecretEngineConfig", "QuaySecretEngineConfig": {"name":"quay","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "quay", "reconcileID": "1bfca365-fab9-4923-aa8e-7faca0920e8f", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "QuaySecretEngineConfig", "namespace": "vault-admin", "name": "quay"}, "error": "secret not found"}
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*QuaySecretEngineConfig).setInternalCredentials
    /home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/quaysecretengineconfig_types.go:138
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*QuaySecretEngineConfig).PrepareInternalValues
    /home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/quaysecretengineconfig_types.go:81
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:93
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65
github.com/redhat-cop/vault-config-operator/controllers.(*QuaySecretEngineConfigReconciler).Reconcile
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/quaysecretengineconfig_controller.go:76
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
2023-11-30T21:19:41Z    ERROR   unable to prepare internal values   {"controller": "quaysecretengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "QuaySecretEngineConfig", "QuaySecretEngineConfig": {"name":"quay","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "quay", "reconcileID": "1bfca365-fab9-4923-aa8e-7faca0920e8f", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "QuaySecretEngineConfig", "namespace": "vault-admin", "name": "quay"}, "error": "secret not found"}
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:95
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65
github.com/redhat-cop/vault-config-operator/controllers.(*QuaySecretEngineConfigReconciler).Reconcile
    /home/runner/work/vault-config-operator/vault-config-operator/controllers/quaysecretengineconfig_controller.go:76
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226

I believe in such cases, the Controller should handle the error, log it, and move on.

raffaelespazzoli commented 12 months ago

the first one is a bug, we should in fact check for the key key to exist and not panic. The second does not seem to be an error. the operator does what you said: reports an error and move on. In both cases the errors occur because the secrets are not populated (yet).

rafaeltuelho commented 12 months ago

Yeah, in the second scenario I ran into the Operator POD crashed and stayed unrecovered. Yes, the secret was not populated yet in Vault. So, I think the operator should not crash and just ignore and keep trying to check the secret in vault.