search

ForgeRock / secret-agent

Generate random Kubernetes secrets and optionally store them in a Cloud Secret Manager
Apache License 2.0
17 stars 20 forks source link

Debug option seems to do nothing #203

Closed mstrent closed 3 years ago

mstrent commented 3 years ago

I've been troubleshooting an issue trying to get on-prem ForgeOps / Secret Agent connected to Azure Key Vault. In the course of that I tried adding "--debug=true" and "--debug" to the command line options for the manager container. No change in verbosity was observed in the logs.

BTW the issue seemed to be caused by our Man in the Middle firewall which intercepts HTTPS sessions and presents certificates signed by our internal CA. And of course your Secret Agent image doesn't trust our CA, so it couldn't verify the certificate chain it saw when trying to hit the Azure Key Vault host. But the Secret Agent log messages were pretty ambiguous, and --debug didn't seem to help at all. This could be improved. :)

{"level":"error","ts":1628724324.470808,"logger":"controllers.SecretAgentConfiguration","msg":"skipping key","secretagentconfiguration":"forgerock-sac","namespace":"forgerock","secret_name":"platform-ca","data_key":"ca","secret_type":"ca","error":"**failed api call to secret manager: keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded","errorVerbose":"keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded\nfailed api call to secret manager**\ngithub.com/ForgeRock/secret-agent/pkg/generator.(*keyGenConfig).secretManagerHasData\n\t/workspace/pkg/generator/generator.go:291\ngithub.com/ForgeRock/secret-agent/pkg/generator.(*GenConfig).GenKeys\n\t/workspace/pkg/generator/generator.go:100\ngithub.com/ForgeRock/secret-agent/controllers.(*SecretAgentConfigurationReconciler).Reconcile\n\t/workspace/controllers/secretagentconfiguration_controller.go:149\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133\nk8s

maxres-fr commented 3 years ago

What version are you using?

There should have been other log messages before this happened. are you not seeing "level": "debug"

mstrent commented 3 years ago

Using the latest version as of a few days ago. Looks like that's v1.1.2.

Nope, zero log lines seen with "level: "debug".

maxres-fr commented 3 years ago

Fixed in PR#206

❯ export IMG=k3d-fr-max.localhost:5000/controller:$(openssl rand -hex 5); make docker-build docker-push install deploy;
/home/max/go/bin/controller-gen "crd:crdVersions=v1" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
# Remove "caBuncle: Cg==" from the webhook config. controller-gen generates the manifests with a placeholder
awk '!/caBundle:/' config/webhook/manifests.yaml > t && mv t config/webhook/manifests.yaml
/home/max/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
/usr/lib/go-1.16/bin/go fmt ./...
/usr/lib/go-1.16/bin/go version
go version go1.16.7 linux/amd64
/usr/lib/go-1.16/bin/go vet ./...
mkdir -p /home/max/projects/secret-agent/testbin
test -f /home/max/projects/secret-agent/testbin/setup-envtest.sh || curl -sSLo /home/max/projects/secret-agent/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.8.3/hack/setup-envtest.sh
source /home/max/projects/secret-agent/testbin/setup-envtest.sh; fetch_envtest_tools /home/max/projects/secret-agent/testbin;
Using cached envtest tools from /home/max/projects/secret-agent/testbin
source /home/max/projects/secret-agent/testbin/setup-envtest.sh; setup_envtest_env /home/max/projects/secret-agent/testbin; /usr/lib/go-1.16/bin/go test ./... -tags=intregration -coverprofile cover.out
setting up env vars
?       github.com/ForgeRock/secret-agent       [no test files]
ok      github.com/ForgeRock/secret-agent/api/v1alpha1  6.528s  coverage: 30.3% of statements
ok      github.com/ForgeRock/secret-agent/controllers   6.850s  coverage: 0.0% of statements
ok      github.com/ForgeRock/secret-agent/pkg/generator 6.263s  coverage: 49.2% of statements
ok      github.com/ForgeRock/secret-agent/pkg/k8ssecrets        0.014s  coverage: 61.5% of statements
?       github.com/ForgeRock/secret-agent/pkg/secretsmanager    [no test files]
docker build . -t k3d-fr-max.localhost:5000/controller:a2f668944e
[+] Building 33.3s (19/19) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                      0.1s
 => => transferring dockerfile: 38B                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                         0.1s
 => => transferring context: 35B                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/openjdk:11-jre-slim                                                                                    1.2s
 => [internal] load metadata for docker.io/library/golang:1.16.5-alpine                                                                                   1.2s
 => [auth] library/golang:pull token for registry-1.docker.io                                                                                             0.0s
 => [auth] library/openjdk:pull token for registry-1.docker.io                                                                                            0.0s
 => [builder 1/6] FROM docker.io/library/golang:1.16.5-alpine@sha256:45f32e963bb3cc408cfcd01a8e76b2872fb238f602ec5481cd75393da29369c0                     0.0s
 => [internal] load build context                                                                                                                         0.2s
 => => transferring context: 549.48kB                                                                                                                     0.1s
 => [release 1/5] FROM docker.io/library/openjdk:11-jre-slim@sha256:b4f3ffe87eee841f553b9491eaa365e33c6e9bdf0656499733abb5107810aa11                      0.0s
 => CACHED [builder 2/6] WORKDIR /workspace                                                                                                               0.0s
 => CACHED [builder 3/6] COPY go.mod go.sum ./                                                                                                            0.0s
 => CACHED [builder 4/6] RUN go mod download                                                                                                              0.0s
 => [builder 5/6] COPY . .                                                                                                                                4.7s
 => [builder 6/6] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -ldflags "-s -w" -a -o manager main.go                               26.0s
 => CACHED [release 2/5] RUN apt-get update &&     DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y curl lsof net-tools &&       0.0s
 => CACHED [release 3/5] RUN addgroup --gid 11111 secret-agent &&     adduser --shell /bin/bash --home /home/secret-agent --uid 11111 --disabled-passwor  0.0s
 => CACHED [release 4/5] WORKDIR /opt/gen                                                                                                                 0.0s
 => [release 5/5] COPY --from=builder --chown=secret-agent:root /workspace/manager /                                                                      0.5s
 => exporting to image                                                                                                                                    0.3s
 => => exporting layers                                                                                                                                   0.2s
 => => writing image sha256:ffc0afb5d557cc5551b1395770c1c6a0f2e32157eda8c2b0451d970dc9a93a1f                                                              0.0s
 => => naming to k3d-fr-max.localhost:5000/controller:a2f668944e                                                                                          0.0s
docker push k3d-fr-max.localhost:5000/controller:a2f668944e
The push refers to repository [k3d-fr-max.localhost:5000/controller]
a60255e6f88a: Pushed
9b2e8a11a8a7: Layer already exists
20ef3ae236b5: Layer already exists
67f21353b8cd: Layer already exists
a292e95a5177: Layer already exists
a065e8f806e2: Layer already exists
8d5bc6a4ce7b: Layer already exists
f68ef921efae: Layer already exists
a2f668944e: digest: sha256:680ad5f4552a0c2f7115f304d2d6032afee37a09c6a8eb55ca9c71e9718f4c8a size: 1998
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/secretagentconfigurations.secret-agent.secrets.forgerock.io configured
cd config/manager && kustomize edit set image controller=k3d-fr-max.localhost:5000/controller:a2f668944e
kustomize build config/default | kubectl apply -f -
namespace/secret-agent-system unchanged
customresourcedefinition.apiextensions.k8s.io/secretagentconfigurations.secret-agent.secrets.forgerock.io configured
serviceaccount/secret-agent-controller-manager unchanged
role.rbac.authorization.k8s.io/secret-agent-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/secret-agent-manager-role configured
clusterrole.rbac.authorization.k8s.io/secret-agent-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/secret-agent-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/secret-agent-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/secret-agent-proxy-rolebinding unchanged
configmap/secret-agent-manager-config unchanged
service/secret-agent-controller-manager-metrics-service unchanged
service/secret-agent-webhook-service unchanged
deployment.apps/secret-agent-controller-manager configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-mutating-webhook-configuration configured
validatingwebhookconfiguration.admissionregistration.k8s.io/secret-agent-validating-webhook-configuration configured

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-204●]
❯ kubectl patch deploy -n secret-agent-system --type=json secret-agent-controller-manager --patch '[{"op": "add", "path": "/spec/template/spec/containers/0/args/8","value": "--debug"}]'
deployment.apps/secret-agent-controller-manager patched

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-204●]
❯ kgp
No resources found in default namespace.

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-204●]
❯ kgp -n secret-agent-system -w
NAME                                               READY   STATUS    RESTARTS   AGE
secret-agent-controller-manager-68875dd84d-4fvsp   2/2     Running   0          4m21s
secret-agent-controller-manager-5d76d7f489-5s294   1/2     Running   0          6s
secret-agent-controller-manager-5d76d7f489-5s294   2/2     Running   0          14s
secret-agent-controller-manager-68875dd84d-4fvsp   2/2     Terminating   0          4m29s
secret-agent-controller-manager-68875dd84d-4fvsp   0/2     Terminating   0          4m30s
^[[1;5D^C%
(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-204●]
❯ k apply -f config/samples/secret-agent_v1alpha1_secretagentconfiguration.yaml                                                                            <<<
secretagentconfiguration.secret-agent.secrets.forgerock.io/forgerock-sac configured

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-204●]
❯ kl secret-agent-controller-manager-5d76d7f489-5s294 -n secret-agent-system -c manager | grep -i debug
2021-08-18T18:10:34.685Z        DEBUG   controllers.SecretAgentConfiguration    ** Reconcile loop start **      {"secretagentconfiguration": "forgerock-sac", "namespace": "default"}
2021-08-18T18:10:34.685Z        DEBUG   controllers.SecretAgentConfiguration    secret found to have data, skipping     {"secretagentconfiguration": "forgerock-sac", "namespace": "default", "secret_name": "platform-ca"}
2021-08-18T18:10:34.685Z        DEBUG   controllers.SecretAgentConfiguration    secret found to have data, skipping     {"secretagentconfiguration": "forgerock-sac", "namespace": "default", "secret_name": "truststore"}
2021-08-18T18:10:34.685Z        DEBUG   controllers.SecretAgentConfiguration    secret found to have data, skipping     {"secretagentconfiguration": "forgerock-sac", "namespace": "default", "secret_name": "ds-passwords"}
2021-08-18T18:10:34.685Z        DEBUG   controllers.SecretAgentConfiguration    secret found to have data, skipping     {"secretagentconfiguration": "forgerockk
mstrent commented 3 years ago

Thank you @maxres-fr !