Support for docker platform architecture when pulling multi-platform images

[dewe]

Running sinker push automatically pulls the image version matching the local machine. This is fine as long as the local machine (or CI build server) runs on the same architecture as the servers running the workload.

But if I happen to run sinker push on my M1 Mac, I'll silently push a bunch of linux/arm64 images to the repo. This can result in the runtime environment pulling bad images, and workloads unable to start.

It would be good to be able to specify the expected platform when pulling and pushing with sinker:

sinker push --platform linux/amd64

[jeremy-cxf]

This should be fairly easy to implement considering ImagePullOptions in the Docker client API exposes Platform. From a quick glance, you could look at extending ImagePullOptions/ImagePushOptions opts to include platform in the interim.

[deeTEEcee]

what's the workaround for this right now if im on the wrong platform?

[dewe]

In CI, we're running sinker in a docker container on the correct architecture.

Locally we added a bit of a safety net in the the Makefile, since the consequences could be pretty bad.

ifeq ($(UNAME), arm64)
    $(error Warning: Don't use sinker push on ARM architecture)
endif
    sinker push 

[kr3cj]

A coworker at our shop found a workaround (similar to @dewe's). We are using colima instead of docker on M1s and just tell colima to startup under a different architecture:

colima delete && colima start --arch x86_64
# ensure docker is running as expected
docker info | grep -i arch
# sync images as usual
sinker push --dryrun

If you want to configure that architecture in a colima config file instead of a startup flag, see this

[jpreese]

@dewe @kr3cj The new copy command, which should be able to replace push in the near future does have flags for specifying architectures and operating systems (https://github.com/plexsystems/sinker/blob/main/internal/commands/copy.go)

does something like:

sinker copy --manifest test/push --override-arch=amd64 --override-os=linux work for you?

[dewe]

Looks promising!

[jpreese]

Would love to hear your feedback and see if it works out for you!

After using it some myself as of late I'm leaning towards providing some sensible default for the architecture and os when using the command, and it may be more succinct to use a flag similar to your original proposal (--platform=linux/amd64)

[dewe]

I did a quick check with copy, pushing into our private ECR registry, but it threw an error:

$ sinker copy -m .images.generated.yaml --override-arch amd64 --override-os linux
INFO[0000] Finding images that need to be copied ...
INFO[0001] Copying image ghcr.io/fluxcd/image-automation-controller:v0.22.1 to [redacted].dkr.ecr.eu-west-1.amazonaws.com/fluxcd/image-automation-controller:v0.22.1
Error: copy: copy image: trying to reuse blob sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139 at destination: checking whether a blob sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139 exists in [redacted].dkr.ecr.eu-west-1.amazonaws.com/fluxcd/image-automation-controller: unauthorized: authentication required

Just to rule out any setup issues, I ran a sinker push with the same image manifest, and it worked fine:

$ sinker push -m .images.generated.yaml
INFO[0000] Finding images that need to be pushed ...
INFO[0000] Pulling ghcr.io/fluxcd/image-automation-controller:v0.22.1
INFO[0004] Pushing [redacted].dkr.ecr.eu-west-1.amazonaws.com/fluxcd/image-automation-controller:v0.22.1
INFO[0010] All images have been pushed!

After using it some myself as of late I'm leaning towards providing some sensible default for the architecture and os when using the command, and it may be more succinct to use a flag similar to your original proposal (--platform=linux/amd64)

Yes, I think it is logical to use the same flag as docker build. Regarding defaults, I believe runtime.GOOS and runtime.GOARCH is the way to go, but this might have issues when running on macos.

[deeTEEcee]

what's the workaround for this right now if im on the wrong platform?

Following up my own comment, here's a couple tools I found that are working for me:

• https://github.com/regclient/regclient
• https://github.com/containers/skopeo (I prefer this one. The other one doesn't log or show a status bar)

[jpreese]

@deeTEEcee does the new command and flags work for you? https://github.com/plexsystems/sinker/issues/62#issuecomment-1145081874

or anything else that needs supported to solve your use case?

[deeTEEcee]

ah i didnt realize that copy command was already released. Is "--override-arch the right word for that argument? It seems we want to just select it instead of "all the architectures."

Overall, I'm just looking for something that allows me to do a simple command like: sinker push -i <images> -t gcr.io/project-name but internally, the docker manifest and all the docker architectures needed would get copied over (whether that's with an --all or --platform option). Right now, skopeo allows me to do that.

[toabi]

I currently have the same issue as @dewe in https://github.com/plexsystems/sinker/issues/62#issuecomment-1146564430 - we have some workflow where we copy images to our AWS ECR. We wrote some nice bash script wrapper around sinker which collect the images and so on. In the past only people on linux/amd64 should run it because otherwise wrong os/arch could get copied. Now we rewrote it to use "copy" with --override-arch/--override-os but when I run it on macOS I get this weird error with

Error: copy: copy image: trying to reuse blob sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 at destination:
checking whether a blob sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 
exists in [ACCOUNT_ID].dkr.ecr.eu-central-1.amazonaws.com/alpine: unauthorized: authentication required

When using push it works, so authentication should be fine (but push pushes linux/arm64 version which is bad)? Is there any workaround/solution for this?

Workaround is using skopeo login to write the proper credentials at the spot where sinker expects them.