SOPS in-cluster decryption with GCP-KMS from a non-GKE Kubernetes cluster (without workload identity)

[anthony-pastor]

Hi, We wanted to use SOPS in-cluster decryption with GCP-KMS from a non-GKE Kubernetes cluster (without workload identity). Unfortunately the documentation currently doesn't provide guides to implement this very specific setup.

As GCP-KMS uses Application Default Credentials we found a way to have a GCP-KMS encrypted file beeing in-cluster decrypted with these steps:

• Create a Google Service Account.

• Adding IAM Permissions bindings to allow this GSA to decrypt GCP-KMS Keyrings/Keys we were interested in.

• Create a JSON Key File for this GSA.

• Create a Kubernetes Secret in the flux-system with this GSA KeyFile, ex: kubectl -n flux-system create secret generic fluxcd-leo-google-sa-secret --from-file=fluxcd_gsa_keyfile.json --type=Opaque

• Edited the kustomize-controller deployment in the flux-system namespace to add:

• env:

      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /tmp/flux/fluxcd_gsa_keyfile.json

• volumeMounts:

        - mountPath: /tmp/flux/fluxcd_gsa_keyfile.json
          name: fluxcd-leo-google-sa-secret
          subPath: fluxcd_gsa_keyfile.json

• volume:

      - name: fluxcd-leo-google-sa-secret
        secret:
          secretName: fluxcd-leo-google-sa-secret

It would be helpful to add all these informations in the documentation. Thanks.

[DARK-art108]

Can I work on this issue?

[DARK-art108]

I want to know more in which place in documentation these all Information need to be added!

[stefanprodan]

@DARK-art108 here is the SOPS doc: https://github.com/fluxcd/website/blob/main/content/en/docs/guides/mozilla-sops.md#google-cloud

PS. Users shouldn't edit the kustomize-controller deployment manifest, instead they should create a kustomize patch before bootstrap, here are the docs for that: https://fluxcd.io/docs/installation/#customize-flux-manifests

[DARK-art108]

Under what header I should start adding these details in SOPS docs! Please Explain!

[tvories]

Has anyone had any updated success with this? I am still unable to get this working, and I have followed @anthony-pastor guide.

Namely, how can I edit the kustomize-controller in a git-ops way, rather than editing the current deployment?

[tvories]

OK, I was able to get this working in my deployment. First, you have to update the flux Kustomization to patch the credential. This assumes you have already created a service account with the proper permissions:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: manage-flux
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./manifests/install
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: flux

  patches:
    # GCP KMS: https://github.com/fluxcd/kustomize-controller/blob/5056fbf6ac5c48de434680042c5979fa79a5de04/docs/spec/v1beta2/kustomization.md#gcp-kms
    - target:
        kind: Deployment
        name: kustomize-controller
      patch: |
        - op: add
          path: /spec/template/spec/containers/0/env/-
          value:
            name: GOOGLE_APPLICATION_CREDENTIALS
            value: /var/gcp/gcp-flux-sa-credential.json
        - op: add
          path: /spec/template/spec/containers/0/volumeMounts/-
          value:
            mountPath: /var/gcp/gcp-flux-sa-credential.json
            name: fluxcd-google-sa-secret
            subPath: gcp-flux-sa-credential.json
            readOnly: true
        - op: add
          path: /spec/template/spec/volumes/-
          value:
            name: fluxcd-google-sa-secret
            secret:
              secretName: fluxcd-google-sa-secret

That will patch the kustomization-controller deployment to mount the gcp credential to /var/gcp/gcp-flux-sa-credential.json and set the environment variable.