Unable to take backup of secrets to google secret manager

[paritoshdubey]

We have deployed forgerock secret agent v1.1.3 in gke cluster(1.20 version) and we want to use workload identity for secret agent' secrets backup to google secret manager. after doing all the configuration, we are not able to backup secret to GSM and no secret are created in GSM. Please let us know how can we resolve the issue. Also I dont see any logs for for pod taking backup of secret to google secret manager(GSM)

[wstrange]

Just to confirm, you are NOT setting spec.appConfig.credentialsSecretName, correct? Workload identity wont be used if this is set.

I'd be inclined to try using a service account json file first, before proceeding to workload identity. This will let you validate that everything else is working before debugging workload identity.

[paritoshdubey]

We tried using workload identity. Now it is giving permission error. Problem is we cant apply secret managaer admin role due to company policy. Is there any other available IAM role in gcp that we can use it to work

[wstrange]

The service account needs sufficient IAM privileges to read/write/list secrets in GCP. There isn't any alternative if you want it to backup secrets to GCP.

[paritoshdubey]

I tried with secret manager admin role. When I run the deployment, it gives error as below:

{"data_key":"cacerts", "level":"error", "logger":"controllers.SecretAgentConfiguration", "msg":"secret ref data not found", "namespace":"development", "secret_name":"truststore-pem", "secret_ref":"platform-ca/ca.pem", "secret_type":"truststore", "secretagentconfiguration":"forgerock-sac", "stacktrace":"github.com/go-logr/zapr.(zapLogger).Error /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132 github.com/ForgeRock/secret-agent/pkg/generator.(keyGenConfig).configureDependencies /workspace/pkg/generator/generator.go:259 github.com/ForgeRock/secret-agent/pkg/generator.(GenConfig).GenKeys /workspace/pkg/generator/generator.go:109 github.com/ForgeRock/secret-agent/controllers.(SecretAgentConfigurationReconciler).Reconcile /workspace/controllers/secretagentconfiguration_controller.go:151 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 k8s.io/apimachinery/pkg/util/wait.BackoffUntil /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 k8s.io/apimachinery/pkg/util/wait.JitterUntil /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 k8s.io/apimachinery/pkg/util/wait.UntilWithContext /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99", "ts":1.637161229932592E9}

I believe it is looking for secret in GSM but it should be creating secrets in GSM on its own. CAn you please help us what are we missing here

[paritoshdubey]

It also gives error attempting to create secret in [global] constraint/resourcelocation org policy. is it possible to run in single region

[wstrange]

That error "secret_ref":"platform-ca/ca.pem "secret ref data not found" ", looks like SA is trying to reference that secret, and it does not exist yet. Can you ensure secrets are ordered in SA in the order they need to get created in I'm not sure that is a GCP issue at this point.

I'd suggest paring down the secret agent file to just a single password secret, and then try to see if it gets created in GCP. This should make it easier to see if its a GCP secret problem or a problem with generating the secret itself

[jrcast]

If you have cloud backup enabled, the secret-agent looks for secrets in GSM first. If such secret is not in GSM, secret-agent generates one and pushes that new secret to GSM. Depending on the order of secrets references you might see a couple reconcile retries. If this is getting printed as errors, it should be a notice. Having said that, if you see the sample config some secrets are dependent on a secret named platform-config which contains a CA named ca in it. Do you have this secret present in your secretAgentConfiguration?

If so, you can put it a the top of the file to avoid running multiple retries. The secret-agent will eventually solve all the references as long as they are there in the secretAgentConfiguration. Regardless, we should just be printing these retries as notices instead of errors.

[paritoshdubey]

@wstrange this error is not only for platform-ca. This error is coming for all also we are able to create secrets as k8s objects. We are getting this error because of organization policy restricting it for creating secrets globally. We have org policy that allow us to create resources only in 4 regions. How to enable this feature for selected regions

[wstrange]

We have org policy that allow us to create resources only in 4 regions. How to enable this feature for selected regions

Secret agent is using the official Google Go library for access to the api: https://github.com/googleapis/google-cloud-go

I'm not sure there is anything we can do in the code itself. This appears to be an issue with configuring GCP roles and privileges to accomplish what you are attempting.

What I would suggest is creating a small sample Job that uses the GCP CLI tool: https://cloud.google.com/sdk/gcloud/reference/secrets . The script can create a secret and read it back.

Use that test Job (with workload identity or a json credential file) to debug the permissions/roles required for proper operation. Once that is working, you can try the same set of roles with secret agent.

[wstrange]

@paritoshdubey If it helps, here is a sample K8S Job that you can adapt to test your roles: https://github.com/ForgeRock/ds-operator/tree/master/tests/gcs

That example uses gsutil, but you can also try that with gcloud to test secret creation