Internal error occurred: failed calling webhook "msecretagentconfiguration.kb.io"

[dai-mk]

Hello everyone,

I followed the README and installed secret-agent in version 1.1.6 and ran kubectl apply -f config/samples/secret-agent_v1alpha1_secretagentconfiguration.yaml but I get this error. Can you guys please point me in the direction how I can fix this?

Error from server (InternalError): error when creating "config/samples/secret-agent_v1alpha1_secretagentconfiguration.yaml": Internal error occurred: failed calling webhook "msecretagentconfiguration.kb.io": Post "https://secret-agent-webhook-service.secret-agent-system.svc:443/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration?timeout=10s": context deadline exceeded

From my side I don't understand this error. Secret Agent seems to proberly run:

kubectl get all -n secret-agent-system
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/secret-agent-controller-manager-7b78bd9989-ttdr2   1/1     Running   0          105m

NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/secret-agent-webhook-service   ClusterIP   172.20.230.169   <none>        443/TCP   105m

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/secret-agent-controller-manager   1/1     1            1           105m

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/secret-agent-controller-manager-7b78bd9989   1         1         1       105m

Log also looks unsuspicious to me:

kubectl -n secret-agent-system logs secret-agent-controller-manager-7b78bd9989-ttdr2 --
{"level":"info","ts":1649757867.3604777,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1649757867.361149,"logger":"setup","msg":"Starting webhook related patches"}
{"level":"info","ts":1649757868.1816747,"logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"secret-agent.secrets.forgerock.io/v1alpha1, Kind=SecretAgentConfiguration","path":"/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1649757868.1817229,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1649757868.1817412,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"secret-agent.secrets.forgerock.io/v1alpha1, Kind=SecretAgentConfiguration","path":"/validate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1649757868.1817522,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1649757868.1817966,"logger":"setup","msg":"starting manager"}
I0412 10:04:28.182131       1 leaderelection.go:242] attempting to acquire leader lease  secret-agent-system/f8e4a0d9.secrets.forgerock.io...
{"level":"info","ts":1649757868.182451,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1649757868.1825638,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1649757868.1829543,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1649757868.1830277,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1649757868.183187,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
I0412 10:04:28.192565       1 leaderelection.go:252] successfully acquired lease secret-agent-system/f8e4a0d9.secrets.forgerock.io
{"level":"info","ts":1649757868.1928117,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secretagentconfiguration","source":"kind source: /, Kind="}
{"level":"info","ts":1649757868.2930584,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secretagentconfiguration","source":"kind source: /, Kind="}
{"level":"info","ts":1649757868.3936,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"secretagentconfiguration"}
{"level":"info","ts":1649757868.3936434,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"secretagentconfiguration","worker count":1}

[dai-mk]

@jrcast @maxres-fr @lee-baines @snarlysodboxer Is this repository maintained?

[lee-baines]

Hi @dai-mk. Yes, although we're not currently developing any new features for the secret-agent.
A couple of questions:

1. Which environment are you deploying to? e.g. AWS/GKE/minikube
2. What events do you see when you run kubectl describe sac forgerock-sac

[dai-mk]

Hi @lee-baines

1. EKS on AWS
2. I get Error from server (NotFound): secretagentconfigurations.secret-agent.secrets.forgerock.io "forgerock-sac" not found

[lee-baines]

Sorry that wouldn't work because the sac wasn't created. The sac that we use for our ForgeRock deployments can be found here: https://github.com/ForgeRock/forgeops/blob/master/kustomize/base/secrets/secret_agent_config.yaml. Can you please try and kubectl apply that?

[dai-mk]

@lee-baines

When I try that I get this error:

$ kubectl apply -f secret_agent_config.yaml 
Error from server (InternalError): error when creating "secret_agent_config.yaml": Internal error occurred: failed calling webhook "msecretagentconfiguration.kb.io": Post "https://secret-agent-webhook-service.secret-agent-system.svc:443/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[lee-baines]

Thanks, can you try a couple of things please. Can you check the secret-agent logs at the time of attempting to apply the sac? Also, in secret-agent-system namespace, run kubectl get services.

[dai-mk]

Manager:

I0422 21:44:17.702190       1 request.go:655] Throttling request took 1.047643378s, request: GET:https://172.20.0.1:443/apis/storage.k8s.io/v1?timeout=32s
{"level":"info","ts":1650663858.8644586,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1650663858.8653393,"logger":"setup","msg":"Starting webhook related patches"}
{"level":"info","ts":1650663862.0041342,"logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"secret-agent.secrets.forgerock.io/v1alpha1, Kind=SecretAgentConfiguration","path":"/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1650663862.0042436,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1650663862.0043132,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"secret-agent.secrets.forgerock.io/v1alpha1, Kind=SecretAgentConfiguration","path":"/validate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1650663862.0043674,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-secret-agent-secrets-forgerock-io-v1alpha1-secretagentconfiguration"}
{"level":"info","ts":1650663862.004456,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1650663862.0046754,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1650663862.0048833,"logger":"controller-runtime.manager.controller.secretagentconfiguration","msg":"Starting EventSource","reconciler group":"secret-agent.secrets.forgerock.io","reconciler kind":"SecretAgentConfiguration","source":"kind source: /, Kind="}
{"level":"info","ts":1650663862.0050583,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1650663862.0147226,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1650663862.014914,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1650663862.015036,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1650663862.1062105,"logger":"controller-runtime.manager.controller.secretagentconfiguration","msg":"Starting EventSource","reconciler group":"secret-agent.secrets.forgerock.io","reconciler kind":"SecretAgentConfiguration","source":"kind source: /, Kind="}
{"level":"info","ts":1650663862.2068095,"logger":"controller-runtime.manager.controller.secretagentconfiguration","msg":"Starting Controller","reconciler group":"secret-agent.secrets.forgerock.io","reconciler kind":"SecretAgentConfiguration"}
{"level":"info","ts":1650663862.206861,"logger":"controller-runtime.manager.controller.secretagentconfiguration","msg":"Starting workers","reconciler group":"secret-agent.secrets.forgerock.io","reconciler kind":"SecretAgentConfiguration","worker count":1}

kube-rbac-proxy

I0422 21:44:17.472945       1 main.go:190] Valid token audiences: 
I0422 21:44:17.473029       1 main.go:262] Generating self signed cert as no cert is provided
I0422 21:44:17.933761       1 main.go:311] Starting TCP socket on 0.0.0.0:8443
I0422 21:44:17.935092       1 main.go:318] Listening securely on 0.0.0.0:8443

The services:

$ kubectl -n secret-agent-system get services
NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
secret-agent-controller-manager-metrics-service   ClusterIP   172.20.143.47    <none>        8443/TCP   2d16h
secret-agent-webhook-service                      ClusterIP   172.20.230.169   <none>        443/TCP    13d

[lee-baines]

I can't reproduce it. I just provisioned an AWS cluster and it worked fine: It may be an issue with how your cluster is setup.

Can you try it on minikube?

[dai-mk]

Minikube with CDK works fine. I'm following the ForgeRock ForgeOps CDM readme and have create an empty cluster (but not with eksctl or cluster-up.sh) Is secret-agent needing something special?

I see at perviously this same error message was solved in another issue, but doesn't say how: https://github.com/ForgeRock/secret-agent/issues/142

[lee-baines]

Have you tried running a secret-agent delete then secret-agent install? secret-agent doesn't require anything special hence it works on a simple minikube cluster