Provide arm64 arch docker images

[tiagomeireles]

Is your feature request related to a problem? Please describe.

I am unable to run pinniped on arm64 arch. Pinniped doesn't currently provide arm64 binaries and images.

The images provided at projects.registry.vmware.com/pinniped/pinniped-server are single arch/platform.

$ docker buildx imagetools inspect projects.registry.vmware.com/pinniped/pinniped-server:v0.22.0
Name:      projects.registry.vmware.com/pinniped/pinniped-server:v0.22.0
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:481b94f4468425542f111143ebb69cd2057b0003e7bee75047892638cf88e135

Describe the solution you'd like

• Build and publish arm64 binaries.
• Build and publish multi-arch container images that includes arm64.

Describe alternatives you've considered

Building pinniped from source for arm64.

Are you considering submitting a PR for this feature?

• How will this project improvement be tested?
• How does this change the current architecture?
• How will this change be backwards compatible?
• How will this feature be documented?

Additional context

Others are running on arm64. Approximately a year ago someone was getting pinniped to work on a rasphberry pi. ref

[cfryanr]

Thanks for opening an issue @tiagomeireles!

Can you share more about your use case? Curious which platform you are using, which Kubernetes distribution, and what you're trying to accomplish.

Although we haven't tried this ourselves, it should be fairly easy to make a copy of the Dockerfile, adjust it for arm64 compilation flags, docker build the Dockerfile, push the resulting container image to your favorite registry, and then use that container image when deploying Pinniped. Curious if that works for you. If your client machine is also arm64, you could compile the CLI with the appropriate arm64 flags too.

[thebearingedge]

I use an Ubuntu Linux VM on Apple silicon and do local cluster work in Docker (KinD/Talos/K3D). It would be really nice to be able to use matching images and cli. Happy to work on that!

Edit: Oh, I see that Bitnami is publishing a multi-arch image.

[cfryanr]

Some thoughts on what we would need to change to support this for the server:

• [x] Update Dockerfiles and pod tolerations, see https://github.com/vmware-tanzu/pinniped/pull/1699
• [x] Figure out how we could update CI to always build multi arch images - make temporary proof of concept CI pipeline
  • [x] This will also involve figuring out how to retain the confirm-version task on the image building job in CI, which currently relies on the image builder task outputting an extracted rootfs
  • [x] Maybe it is not worth building a multi arch fips image in CI. It's not hard to implement, but it takes about 22 minutes to build (increased from about 4 minutes for single arch), so it may not be worth slowing down the pipeline by that extent. I don't think we would stand to benefit too much by running the integration tests on the fips build on an arm64 cluster.
• [x] Find a way to run the integration tests on an arm64 cluster, e.g. by creating an arm64 kind cluster in CI and running the tests there as a new CI job - do this in the temporary proof of concept pipeline
  • [x] Allow the creation of arm64 VMs with arm64 kind clusters in CI
  • [x] Need an arm64 or multi arch build of the dex container
  • [x] Need an arm64 or multi arch build of the ldap (openldap server) container, also required some changes in the ldap deployment manifest which were included in https://github.com/vmware-tanzu/pinniped/pull/1699
    • [ ] Get the bugs fixed in bitnami/openldap and upgrade to a container image which includes the fix (not a blocker, but would be nice to have)
    • https://github.com/bitnami/containers/issues/50908
    • and https://github.com/bitnami/containers/issues/49575
  • [x] Need an arm64 or multi arch build of the proxy (squid web proxy) container
  • [x] Need an arm64 or multi arch build of the cert issuer Job's (cfssl CLI) container
• [x] Make sure that the CI code which pushes the container image while creating the GitHub release pushes it as an OCI multi arch image - found that it does not copy the image as a multi-arch image, so created a new CI task script which can d it correctly
• [x] Actually add the CI pipeline changes (prototyped above) for building multi-arch images and running integration tests on arm64 clusters to the PR pipelines and the main pipeline - all commits in the CI repo from early October are in support of this effort. For my teammates, these are the all the related CI changes: https://github.com/vmware-tanzu/pinniped-ci/compare/c6d7ad44b1253820282b90476f94a84a65967c45...2c6c23dd491741eb3a31d00a7f5027bbcd90e05b

Some thoughts on what we would need to change to support this for the CLI:

• [x] Change CI to build windows, linux, and Darwin CLI binaries for arm64, in addition to the binaries that it already builds
• [x] Confirm that CI will attach these additional binaries to the GitHub release when creating a new release (it should, it uses a glob pattern to pick up all the built CLI binaries)
• [x] Check if we need to change how CI updates brew after each new release
  • Here is our brew formula: https://github.com/vmware-tanzu/homebrew-pinniped/blob/main/pinniped-cli.rb. It does not mention any particular platform, and it always builds the code.
  • CI only updates the version number and git sha when we create a new release (e.g. this commit). So it seems that there is nothing to change here.
• [x] Create a PR to update https://pinniped.dev/docs/howto/install-cli/ to list ARM support - done in https://github.com/vmware-tanzu/pinniped/pull/1702

Some thoughts on other things that we should do:

• [x] Make sure that you can build and run the integration tests locally on a Mac with Apple Silicon (arm64 CPU)

Notes:

• Homebrew is not supported on arm64 linux machines at this time: https://docs.brew.sh/Homebrew-on-Linux#arm-unsupported. On Windows, "The Homebrew package manager may be used on Linux and Windows Subsystem for Linux (WSL) 2", which implies that it won't work on arm64 windows either. There is no other Windows support except for the WSL support (can't use brew on actual Windows OS).

[cfryanr]

The next release will include amd64+arm64 multi-arch container images for the server, and will also include arm64 for the linux/darwin/windows CLI. All the necessary changes have already been made, so we just need to create a release. 🎉