search

ForgeRock / secret-agent

Generate random Kubernetes secrets and optionally store them in a Cloud Secret Manager
Apache License 2.0
17 stars 20 forks source link

Error while deploying Forgerock Secret Agent v1.1.3 #209

Closed paritoshdubey closed 3 years ago

paritoshdubey commented 3 years ago

We are getting below errors:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "SecretAgentConfiguration" in version "secret-agent.secrets.forgerock.io/v1alpha1"

error retrieving resource lock secret-agent-system/f8e4a0d9.secrets.forgerock.io: leases.coordination.k8s.io "f8e4a0d9.secrets.forgerock.io" is forbidden: User "system:serviceaccount:secret-agent-system:secret-agent-manager-service-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "secret-agent-system"

wstrange commented 3 years ago

Can you provide more information for debugging. Kubernetes provider / version. Secret agent version, and how it was installed. Any pod logs in the secret agent namespace.

paritoshdubey commented 3 years ago

Secret Agent version is v1.. our GKE version 1.201.3. We have deployed Secret agent using manifest https://github.com/ForgeRock/secret-agent/releases/latest/download/secret-agent.yaml even then it was giving error "error retrieving resource lock secret-agent-system/f8e4a0d9.secrets.forgerock.io: leases.coordination.k8s.io "f8e4a0d9.secrets.forgerock.io" is forbidden: User "system:serviceaccount:secret-agent-system:secret-agent-manager-service-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "secret-agent-system"" in pod logs of secret agent pod. then we tried deploy SAC then it was giving error "Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "SecretAgentConfiguration" in version "secret-agent.secrets.forgerock.io/v1alpha1"".

paritoshdubey commented 3 years ago

just to correct we get above error with v0.1.0. And we try to deploy using SA_VERSION=v0.1.0 kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/download/${SA_VERSION}/secret-agent.yaml

maxres-fr commented 3 years ago

I've been trying to reproduce this issue. Internally we run 1.20 at the moment with SA 1.1.3, and also in a our QA pipelines.

I would advise against using v0.1.0 especially on 1.20.

That being said, I've attempted to recreate. The only issue I found was when upgrading from v0.1.0 there was an issue with crd, and likely because the older validating webhook was still running and was causing a validation error. Removing the crd and re-adding the newer one resolved the issue.

So can you provide the output during an install? Is this error happening on install or during attempting to create a SAC? Do you have any "add ons", non default admission controllers? "error retrieving resource lock secret-agent-system/f8e4a0d9.secrets.forgerock.io: leases.coordination.k8s.io "f8e4a0d9.secrets.forgerock.io" is forbidden: User "system:serviceaccount:secret-agent-system:secret-agent-manager-service-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "secret-agent-system"" in pod logs of secret agent pod. This looks like a HA install with multiple SA replicas. Did the deployment eventually become healthy? Is the user running the install have full permissions to do so?

❯ k apply -f https://github.com/ForgeRock/secret-agent/releases/latest/download/secret-agent.yaml
namespace/secret-agent-system created
customresourcedefinition.apiextensions.k8s.io/secretagentconfigurations.secret-agent.secrets.forgerock.io created
serviceaccount/secret-agent-controller-manager created
role.rbac.authorization.k8s.io/secret-agent-leader-election-role created
clusterrole.rbac.authorization.k8s.io/secret-agent-manager-role created
clusterrole.rbac.authorization.k8s.io/secret-agent-metrics-reader created
clusterrole.rbac.authorization.k8s.io/secret-agent-proxy-role created
rolebinding.rbac.authorization.k8s.io/secret-agent-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-proxy-rolebinding created
configmap/secret-agent-manager-config created
service/secret-agent-controller-manager-metrics-service created
service/secret-agent-webhook-service created
deployment.apps/secret-agent-controller-manager created
mutatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-validating-webhook-configuration created

❯ k apply -f tests/secret_agent.yaml
secretagentconfiguration.secret-agent.secrets.forgerock.io/forgerock-sac created

❯ k get sac -w
NAME            STATUS      NUMSECRETS   NUMK8SSECRETS
forgerock-sac   Completed   5            5

❯ k delete sac forgerock-sac
secretagentconfiguration.secret-agent.secrets.forgerock.io "forgerock-sac" deleted

❯ export SA_VERSION=v0.1.0; kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/download/${SA_VERSION}/secret-agent.yaml
namespace/secret-agent-system created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/secretagentconfigurations.secret-agent.secrets.forgerock.io created
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
mutatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-mutating-webhook-configuration created
serviceaccount/secret-agent-manager-service-account created
role.rbac.authorization.k8s.io/secret-agent-leader-election-role created
clusterrole.rbac.authorization.k8s.io/secret-agent-manager-role created
rolebinding.rbac.authorization.k8s.io/secret-agent-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-manager-rolebinding created
service/secret-agent-webhook-service created
deployment.apps/secret-agent-controller-manager created
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-validating-webhook-configuration created

❯ k apply -f  tests/secret_agent.yaml
secretagentconfiguration.secret-agent.secrets.forgerock.io/forgerock-sac created

❯ kl -n secret-agent-system secret-agent-controller-manager-5cf67cbdbd-k2v2c -f
{"level":"info","ts":1630688805.9826488,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1630688805.9831314,"logger":"setup","msg":"Starting webhook related patches"}
{"level":"info","ts":1630688808.7611217,"logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"secret-agent.secrets.forgerock.io/v1alpha1, Kind=SecretAgentConfiguration","path":"/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1630688808.7611883,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1630688808.7612016,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"secret-agent.secrets.forgerock.io/v1alpha1, Kind=SecretAgentConfiguration","path":"/validate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1630688808.7612083,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1630688808.7612622,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1630688808.7617204,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0903 17:06:48.761617       1 leaderelection.go:242] attempting to acquire leader lease  secret-agent-system/f8e4a0d9.secrets.forgerock.io...
{"level":"info","ts":1630688808.8617992,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1630688808.8722568,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1630688808.8724134,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1630688808.8725376,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
I0903 17:07:04.934206       1 leaderelection.go:252] successfully acquired lease secret-agent-system/f8e4a0d9.secrets.forgerock.io
{"level":"info","ts":1630688824.9345767,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secretagentconfiguration","source":"kind source: /, Kind="}
{"level":"info","ts":1630688825.0349123,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secretagentconfiguration","source":"kind source: /, Kind="}
{"level":"info","ts":1630688825.035076,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"secretagentconfiguration"}
{"level":"info","ts":1630688825.0350933,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"secretagentconfiguration","worker count":1}
{"level":"info","ts":1630688825.0361521,"logger":"controllers.SecretAgentConfiguration","msg":"Reconcile loop complete","secretagentconfiguration":"test-ds-backup-restore/forgerock-sac","secretAgentConfiguration":"forgerock-sac"}
{"level":"info","ts":1630688825.0368829,"logger":"controllers.SecretAgentConfiguration","msg":"Reconcile loop complete","secretagentconfiguration":"test-ds-backup-restore/forgerock-sac","secretAgentConfiguration":"forgerock-sac"}
^C

❯ k get secrets
NAME                  TYPE                                  DATA   AGE
default-token-k7vjm   kubernetes.io/service-account-token   3      3d17h
ds                    Opaque                                6      24s
ds-env-secrets        Opaque                                3      18s
ds-passwords          Opaque                                2      24s
platform-ca           Opaque                                2      25s
truststore-pem        Opaque                                1      24s
maxres-fr commented 3 years ago

Also have you modified RBAC policies at all?

paritoshdubey commented 3 years ago

Hi Max,

Yes issue is with v0.1.0 in gke 1.20. I tried with v1.1.3 in gke 1.20 it worked without issue. Thanks for the help