Can't build multi arch images on M1 chips (Jib 3.4.1+) (Works on x64 chips)

[rsmartins78]

Environment:

• Jib version: 3.4.2
• Build tool: Gradle 7.6.4
• OS: macOS and Linux

Description of the issue: When running ./gradlew jibDockerBuild on macOS (M1 chip), it returns an error saying the the from image is not a manifest file, but this base image is a manifest list, and the same command when executed on my CI environment running on containerized Debian (Intel x64) just works. Problem started after upgrading jib to 3.4.1 and persists on 3.4.2. It was not happening on 3.1.2

Expected behavior: Image to be built successfully on both OS and Processor Architecture.

Steps to reproduce:

1. On a macOS system with M1+ chips
2. Run the command ./gradle jibDockerBuild

jib-gradle-plugin Configuration:

// Projects that generate a JVM based Docker image
apply(plugin = "com.google.cloud.tools.jib")
jib {
    to {
        image = "local/project"
    }
    from {
        image = "europe-docker.pkg.dev/<redacted>/repo-docker/base-17-jre"
        platforms {
            platform {
                architecture = "amd64"
                os = "linux"
            }
            platform {
                architecture = "arm64"
                os = "linux" //also tried darwin here.
            }
        }
    }
    container {
        creationTime.set("USE_CURRENT_TIMESTAMP")
    }
}

Log output:

• What went wrong: Execution failed for task '::jibDockerBuild'.

  com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: cannot build for multiple platforms since the base image 'europe-docker.pkg.dev/redacted/repo-docker/base-17-jre' is not a manifest list.

Additional Information: This base image is based on gcr.io/distroless/java17-debian11, hosted internally without modifications. But I also tried from the original distroless repo and also tried eclipse-temurin:17 image. None of them worked on my macOS env.

[chanseokoh]

Hmm... just to rule out something: can you try with ./gradlew jibDockerBuild -Djib.useOnlyProjectCache=true? This way, Jib won't use centrally cached images.

[rsmartins78]

Same result unfortunately.

[chanseokoh]

What if you specify the SHA pointing to the manifest list as the from image reference?

You can take note of the correct SHA when you run the following command on x64 (not on Mac). Mind clean and -Djib.useOnlyProjectCache=true. And it's be interesting to see if the same command reports a different SHA on Mac.

$ ./gradlew -Djib.useOnlyProjectCache=true clean jibDockerBuild
...
Containerizing application to Docker daemon as ...
The base image requires auth. Trying again for alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321...
Using credentials from Docker config (/home/chanseok/.docker/config.json) for alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Using base image with digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321

And make sure the SHA points to a manifest list. In my case above,

$ docker manifest inspect alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   ...

[rsmartins78]

I tried informing the image sha256, but it didn't work. But I noticed something. Although the sha256 I'm providing is a manifest (when I check on the registry itself), the downloaded version on my laptop is the linux/arm64 version.

About the command you recommended, the error was the same. My docker version is 4.29.0, and I tried with containerd enabled and disabled, both returns the same error and the same result when running docker inspect <image>.

PS: Randomly during my tests I saw a message about not being able to build multi-arch using docker, but it doesn't show-up all the time.

[rsmartins78]

Quick update. If I run docker manifest inspect the result is the same as the registry.

[chanseokoh]

As a last resort, I'd like to see all the raw HTTP requests and responses between Jib and your registry. Can yo follow these instructions to capture the traffic? Make sure you include both clean and -Djib.useOnlyProjectCache=true. (That is, it should not use the cached image.) You may also want -X (debug logging) instead of --info, but actually I am not sure if debug logging will be useful.

You may redact the registry address, but please do not redact SHA strings.

---

PS: Randomly during my tests I saw a message about not being able to build multi-arch using docker, but it doesn't show-up all the time.

Yeah, regardless of this issue, jibDockerBuild will not work anyway, because the Docker engine doesn't support storing a manifest list.

[rsmartins78]

Sorry for the late reply. I'll try this as soon as possible and get back with the results.