[Feature]: Keep image architecture stable when using --arch

[ctron]

Feature request description

Using the --arch argument overwrites the image, so subsequent runs (not using --arch) might get a random architecture:

--arch=ARCH Override the architecture, defaults to hosts, of the image to be pulled. For example, arm. Unless overridden, subsequent lookups of the same image in the local storage will match this architecture, regardless of the host.

This makes it problematic to work with multiple architectures, as you never really know which architecture gets executes.

And during multi-staged builds, this might also just change between stages.

Suggest potential solution

Images of different architectures should not overwrite each other.

Have you considered any alternatives?

Docker using buildx seems to support this.

Additional context

Add any other context or screenshots about the feature request here.

[Luap99]

@vrothberg PTAL

I think this was asked a couple of times before.

[vrothberg]

Yes, this issue pops up frequently. The issue with multi-arch images is that some images get things wrong, so their configs may claim they're of arch foo although they are of arch bar. That means, once an image is pulled, we cannot trust what it claims to be. That's unfortunate but happened that often that we decided to always pull an image when --arch is specified.

One thing that struck my eyes in the issue is "a random architecture". @ctron can you elaborate what you mean by "random"? I am unaware of any randomness. The code and behavior is deterministic.

Images of different architectures should not overwrite each other.

That is an interesting idea. I am not sure it's feasible as it would require drastic changes that would hence be expensive to develop. The image namespace is flat so pulling an existing image but with another architecture will untag the previous one.

[rhatdan]

Would it be a crazy idea to automatically create a manifest if you pull a second image of a different architecture?

[vrothberg]

In my experience it is impossible to satisfy all requirements. Some users expect behavior A, other B or C. A, B and C are conflicting.

Creating a manifest would satisfy this request (clever idea btw) but it would likely break existing deployments. Multi-arch is a cool thing for builds but for running (i.e., podman run --arch) it's full of traps. There's also a bug in Docker which conflates certain ARM variants (AFAIR v7 and v8).

[Romain-Geissler-1A]

Hi,

On a side note, I also look at docker issues regularly, and the same complain reaches docker folks regularly as well. On docker side, there is an environment variable DOCKER_DEFAULT_PLATFORM and it seems the StackOverflow litterature and the likes seems to advise definining this one to typically linux/amd64 to avoid issues and have the most "logical" behavior by default (if you ask me, this is the behavior docker should have WITHOUT the need to define this variable). Note that I have never used this myself, now I advise my coworkers to always use the --pull flag for both build and run, which in a sense avoid this issue as well. I also know the docker folks are quite aware that their multi-arch support in Docker 20 isn't good, I don't know what they changed exactly in docker 23, but IMO we shall first find a way to "fix" the default behavior of docker, and the make podman adapt to this new behavior. If podman and docker works differently, it will be a nightmare to move from one to the other.

One pointer of the currently known issue on docker side with mutli-arch interaction: https://github.com/moby/moby/issues/44582

Cheers, Romain

[ctron]

One thing that struck my eyes in the issue is "a random architecture". @ctron can you elaborate what you mean by "random"? I am unaware of any randomness. The code and behavior is deterministic.

The code itself is. However, "a system" is not. Assume you have multiple processes, scripts, … running on the same system. You can't predict which architecture was pulled last.

[vrothberg]

The code itself is. However, "a system" is not. Assume you have multiple processes, scripts, … running on the same system. You can't predict which architecture was pulled last.

Thanks for elaborating!

Yes, that's not something Podman can protect the user from. Same could happen when a new image was pushed to the registry. The best way to make sure to always pull the same image is by using a digest: podman run image@sha256:...

I understand that's not easy to use but if there's a scenario where multiple tools may be pulling the same image but for a different architecture, that's the way to go (at the moment).

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[ijackson]

Yes, that's not something Podman can protect the user from. Same could happen when a new image was pushed to the registry. The best way to make sure to always pull the same image is by using a digest: podman run image@sha256:...

A new version is quite a different situation, to a different platform.

IMO what platform (architecture) you get should never depend on the contents of the cache.

If this is not doable by default for some compatibility reasons, perhaps we could have a command line flag or environment variable to always decide first what platform is wanted, and then never use any image for a different platform, even if cached?

[notAmine]

This is an issue especially for developers using M1/2/3 macbooks which run apple silicon. I solved this previously when I used docker desktop with DOCKER_DEFAULT_PLATFORM=linux/amd64 is it possible to adopt the same env var which you can define on your host or maybe we can use PODMAN_DEFAULT_PLATFORM env var for this purpose?