Support arm64

[imjasonh]

TL;DR

The CSI driver provider for GCP is built as a single-platform Docker image supporting only linux/amd64.

With GKE now supporting Arm nodes, I'd like to be able to use this provider on Arm nodes, but the CSI driver image doesn't support it.

The secret store CSI driver itself supports arm64: https://github.com/kubernetes-sigs/secrets-store-csi-driver/issues/525

Design

Proposal

One option would be to use buildx, and use the builtin BUILDARCH instead of hard-coding amd64 here.

More info about multi-platform buildx builds here:

• https://medium.com/@tonistiigi/faster-multi-platform-builds-dockerfile-cross-compilation-guide-part-1-ec087c719eaf
• https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/

Another option would be to build using ko, since your build is just a go build and then putting that binary on top of a distroless base image.

I've tried this out, and it seems to work:

echo "defaultBaseImage: gcr.io/distroless/static-debian11" > .ko.yaml
export KO_DOCKER_REPO=gcr.io/jason-chainguard-public  # Use your own GCR repo

Then this:

ko build --platform=linux/amd64,linux/arm64

...produces this image:

gcr.io/jason-chainguard-public/secrets-store-csi-driver-provider-gcp-a743edd397a7a5976ec91daa5f2e0751@sha256:38b65465687abb5902904cf769aee3445d685c326ae706338546b9241b634378

...which supports linux/arm64.

I've added a toleration to allow the DaemonSet to run on arm64 nodes, and tested this on a GKE Arm cluster, and things seem to work just fine as far as I can tell.

Alternatives considered

I don't know of any other way to support this besides building an arm64-compatible image and tolerating arm64 nodes.

Resources

• more docs on ko: https://ko.build
• docs on ko's multi-platform support: https://ko.build/features/multi-platform/

Additional information

ko also brings some security benefits:

• generates and attaches SBOMs by default
• supports other platforms easily, including Windows with some work

If this is interesting I'd be happy to work with the maintainers to make this a reality. In the meantime I can work around this by using my own image.

[imjasonh]

A slight correction: With some work I'm able to get the DaemonSet running on both amd64 and arm64 nodes:

$ cat arm-patch.yaml
spec:
  template:
    spec:
      containers:
      - name: provider
        image: gcr.io/jason-chainguard-public/github.com/googlecloudplatform/secrets-store-csi-driver-provider-gcp@sha256:f9cf11be3f324fbace857eebc7ef49ee638d253f1c4fae099a038fd330b98163
      tolerations:
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule
$ kubectl patch daemonset csi-secrets-store-provider-gcp \                                                            
  -n kube-system \
  --patch-file=arm-patch.yaml

And replicas come up on each node, become Running, report as healthy, etc.

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
csi-secrets-store-provider-gcp   2         2         2       2            2           kubernetes.io/os=linux   93m

But when I create my arm workload, I get the dreaded:

  Warning  FailedMount  76s (x35 over 56m)  kubelet            MountVolume.SetUp failed for volume "melange-key" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

The logs for replicas on both nodes look the same (no errors):

{"ts":1668113118801.2368,"caller":"secrets-store-csi-driver-provider-gcp/main.go:83","msg":"starting secrets-store-csi-driver-provider-gcp/dev","v":0}
2022/11/10 20:45:18 reader is not registered
{"ts":1668113118899.942,"caller":"secrets-store-csi-driver-provider-gcp/main.go:222","msg":"health server listening","v":0,"addr":":8095"}

And the pod running on amd64 gets access to the secret with no issues. It's only the arm64 node that doesn't see the registered CSI driver.

Any help debugging this would be really great, I feel like I'm really close to getting this working with relatively few changes needed.

[imjasonh]

The trick seems to be to tolerate arm nodes on the csi-secrets-store daemonset as well:

$ cat arm-patch-2.yaml
spec:
  template:
    spec:
      tolerations:
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule
$ kubectl patch daemonset csi-secrets-store \                                                            
  -n kube-system \
  --patch-file=arm-patch-2.yaml

With this and the above, and a GCP provider image that supports arm64 (https://github.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp/pull/193), I'm able to mount secrets on Arm nodes 🎉

[imjasonh]

Hey @tam7t , me again 👋

What do you think about exploring building the provider with ko? Since the previous proposal, ko was accepted as a CNCF sandbox project, and still brings the security benefits described above (SBOMs, Windows support (#84), an even smaller default base image than you're using today).

If that sounds interesting I'd be happy to send a PR to make it so, and we can see what it looks like.

[imjasonh]

This is done, just not released. We're using it just fine with an image we build ourselves from source at head.

Closing since it's done; not sure if there's anywhere tracking a release to get this out officially though.