search

fabric8io / docker-maven-plugin

Maven plugin for running and creating Docker images
https://dmp.fabric8.io
Apache License 2.0
1.87k stars 642 forks source link

Cannot build multiarch in CI using buildx and DMP 0.43.3 #1701

Open poikilotherm opened 1 year ago

poikilotherm commented 1 year ago

Description

While the new release 0.43.3 fixed the build problem on a Mac M1, it creates a new problem: multiarch builds using Buildx in a Github Action pipeline are no longer working after upgrading. Using an older version of the plugin, things are well in the CI (using 0.43.0 for quite some time now).

I see the following log entries from the runner. Here's the complete log output.

[INFO] --- docker-maven-plugin:0.43.3:push (default-push) @ dataverse ---
[INFO] Expanding: /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build.tar into /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build
[INFO] DOCKER> docker buildx build --progress=plain --builder maven --platform linux/amd64,linux/arm64 --tag ghcr.io/gdcc/dataverse:9771-fix-docker-build --build-arg BASE_IMAGE=gdcc/base:unstable --file=/home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build/Dockerfile /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build --push
[INFO] DOCKER> ERROR: no builder "maven" found
Error:  DOCKER> Error status (1) when building

Info

rohanKanojia commented 1 year ago

@poikilotherm : Thanks for reporting. Could you please share the difference in executed buildx commands in 0.43.0 and 0.43.3? 0.43.3 adds --config if auth credentials are not provided via docker config.

poikilotherm commented 1 year ago

@rohanKanojia thx for the reply. Here are some outputs as requested.

0.43.0, taken from https://github.com/IQSS/dataverse/actions/runs/5866182102/job/15904575945:

$ mvn  -Dapp.image.tag=unstable -Dbase.image.tag=unstable  -Ddocker.platforms=linux/amd64,linux/arm64 -P ct deploy
...
[INFO] --- docker-maven-plugin:0.43.0:build (default-build) @ dataverse ---
[INFO] DOCKER> docker --config /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/unstable/docker buildx create --driver docker-container --name maven
[INFO] DOCKER> docker --config /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/unstable/docker buildx build --progress=plain --builder maven --platform linux/amd64 --tag gdcc/dataverse:unstable --build-arg BASE_IMAGE=gdcc/base:unstable --file=/home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/unstable/tmp/docker-build/Dockerfile /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/unstable/tmp/docker-build --load
...
[INFO] --- docker-maven-plugin:0.43.0:push (default-push) @ dataverse ---
[INFO] DOCKER> docker --config /home/runner/work/dataverse/dataverse/target/docker/gdcc/configbaker/unstable/docker buildx build --progress=plain --builder maven --platform linux/amd64,linux/arm64 --tag gdcc/configbaker:unstable --build-arg SOLR_VERSION=8.11.1 --file=/home/runner/work/dataverse/dataverse/target/docker/gdcc/configbaker/unstable/tmp/docker-build/Dockerfile /home/runner/work/dataverse/dataverse/target/docker/gdcc/configbaker/unstable/tmp/docker-build --push

0.43.3, taken from https://github.com/IQSS/dataverse/actions/runs/5860235371/job/15888066573:

mvn  -Dapp.image.tag=9771-fix-docker-build -Dbase.image.tag=unstable '-Ddocker.registry=ghcr.io' 
-Ddocker.platforms=linux/amd64,linux/arm64 -P ct deploy
...
[INFO] --- docker-maven-plugin:0.43.3:build (default-build) @ dataverse ---
[INFO] DOCKER> docker --config /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/docker buildx create --driver docker-container --name maven --node maven0
[INFO] DOCKER> docker --config /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/docker buildx build --progress=plain --builder maven --platform linux/amd64 --tag gdcc/dataverse:9771-fix-docker-build --build-arg BASE_IMAGE=gdcc/base:unstable --file=/home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build/Dockerfile /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build --load
...
[INFO] --- docker-maven-plugin:0.43.3:push (default-push) @ dataverse ---
[INFO] DOCKER> docker buildx build --progress=plain --builder maven --platform linux/amd64,linux/arm64 --tag ghcr.io/gdcc/dataverse:9771-fix-docker-build --build-arg BASE_IMAGE=gdcc/base:unstable --file=/home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build/Dockerfile /home/runner/work/dataverse/dataverse/target/docker/gdcc/dataverse/9771-fix-docker-build/tmp/docker-build --push
[INFO] DOCKER> ERROR: no builder "maven" found

I'm spotting the difference that with 0.43.3, a parameter --node maven0 is added which wasn't present before and --config is not added in the second run with multiple platforms. (Though it is added in the build phase...? :thinking: )

rohanKanojia commented 1 year ago

@poikilotherm : What authentication method are you using? I had changed plugin behavior to not add --config when user is using docker login (since default value of config is ~/.docker/config.json) .

Change of --node maven0 was added since 0.43.2 . Do you face this issue just on 0.43.3 or 0.43.2? I think we can change the default value of --node parameter to null.

tomking1982 commented 1 year ago

@rohanKanojia You're right that --node maven0 was added since 0.43.2 which is working fine for our CIs. But we met the similar issue that it's failed after upgrading from 0.43.2 to 0.43.3. This issue can be found only when building in platform linux/arm64

[INFO] DOCKER> error: open /root/.docker/buildx/instances/maven: no such file or directory
Error:  DOCKER> Error status (1) when building

The only difference is that --config /tmp/****/docker was added before buildx** AWS authentication - https://dmp.fabric8.io/#extended-authentication which is our authentication method.

rohanKanojia commented 1 year ago

Looks like this is due to difference in docker CLI versions. Newer docker CLIs require --config option to be added always. Older docker CLIs (20.x) seem to be having problems with passing --config all the time

rohanKanojia commented 1 year ago

@poikilotherm : I've released 0.43.4 which should always send --config for docker CLIs > 23.x . Could you please check if your issue is resolved by upgrading to this version :pray: ?

poikilotherm commented 1 year ago

Works like a charm in CI. Will respond later about status on the M1 machine where things failed before. Thanks a ton for fixing this @rohanKanojia ! Much appreciated!

jramosf commented 1 year ago

This is breaking compatibility with previous version:

$ docker --version
Docker version 24.0.2, build cb74dfc
$ mvn -U clean docker:build -Ddocker.build=true
[INFO] Scanning for projects...
[INFO] 
...
[INFO] DOCKER> Using Docker CLI 24.0.2
[INFO] DOCKER> docker --config /... create --driver docker-container --name maven
[INFO] DOCKER> unknown flag: --driver
[INFO] DOCKER> See 'docker --help'.
[INFO] DOCKER> 
[INFO] DOCKER> Usage:  docker [OPTIONS] COMMAND
[INFO] DOCKER> 
[INFO] DOCKER> A self-sufficient runtime for containers
[INFO] DOCKER> 
[INFO] DOCKER> Common Commands:
[INFO] DOCKER>   run         Create and run a new container from an image
[INFO] DOCKER>   exec        Execute a command in a running container
[INFO] DOCKER>   ps          List containers
[INFO] DOCKER>   build       Build an image from a Dockerfile
[INFO] DOCKER>   pull        Download an image from a registry
[INFO] DOCKER>   push        Upload an image to a registry
[INFO] DOCKER>   images      List images
[INFO] DOCKER>   login       Log in to a registry
[INFO] DOCKER>   logout      Log out from a registry
[INFO] DOCKER>   search      Search Docker Hub for images
[INFO] DOCKER>   version     Show the Docker version information
[INFO] DOCKER>   info        Display system-wide information
[INFO] DOCKER> 
[INFO] DOCKER> Management Commands:
[INFO] DOCKER>   builder     Manage builds
[INFO] DOCKER>   container   Manage containers
[INFO] DOCKER>   context     Manage contexts
[INFO] DOCKER>   image       Manage images
[INFO] DOCKER>   manifest    Manage Docker image manifests and manifest lists
[INFO] DOCKER>   network     Manage networks
[INFO] DOCKER>   plugin      Manage plugins
[INFO] DOCKER>   system      Manage Docker
[INFO] DOCKER>   trust       Manage trust on Docker images
[INFO] DOCKER>   volume      Manage volumes
[INFO] DOCKER> 
[INFO] DOCKER> Swarm Commands:
[INFO] DOCKER>   swarm       Manage Swarm
[INFO] DOCKER> 
[INFO] DOCKER> Commands:
[INFO] DOCKER>   attach      Attach local standard input, output, and error streams to a running container
[INFO] DOCKER>   commit      Create a new image from a container's changes
[INFO] DOCKER>   cp          Copy files/folders between a container and the local filesystem
[INFO] DOCKER>   create      Create a new container
[INFO] DOCKER>   diff        Inspect changes to files or directories on a container's filesystem
[INFO] DOCKER>   events      Get real time events from the server
[INFO] DOCKER>   export      Export a container's filesystem as a tar archive
[INFO] DOCKER>   history     Show the history of an image
[INFO] DOCKER>   import      Import the contents from a tarball to create a filesystem image
[INFO] DOCKER>   inspect     Return low-level information on Docker objects
[INFO] DOCKER>   kill        Kill one or more running containers
[INFO] DOCKER>   load        Load an image from a tar archive or STDIN
[INFO] DOCKER>   logs        Fetch the logs of a container
[INFO] DOCKER>   pause       Pause all processes within one or more containers
[INFO] DOCKER>   port        List port mappings or a specific mapping for the container
[INFO] DOCKER>   rename      Rename a container
[INFO] DOCKER>   restart     Restart one or more containers
[INFO] DOCKER>   rm          Remove one or more containers
[INFO] DOCKER>   rmi         Remove one or more images
[INFO] DOCKER>   save        Save one or more images to a tar archive (streamed to STDOUT by default)
[INFO] DOCKER>   start       Start one or more stopped containers
[INFO] DOCKER>   stats       Display a live stream of container(s) resource usage statistics
[INFO] DOCKER>   stop        Stop one or more running containers
[INFO] DOCKER>   tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
[INFO] DOCKER>   top         Display the running processes of a container
[INFO] DOCKER>   unpause     Unpause all processes within one or more containers
[INFO] DOCKER>   update      Update configuration of one or more containers
[INFO] DOCKER>   wait        Block until one or more containers stop, then print their exit codes
[INFO] DOCKER> 
[INFO] DOCKER> Global Options:
[INFO] DOCKER>       --config string      Location of client config files (default
[INFO] DOCKER>                            "/Users/<REDACTED>/.docker")
[INFO] DOCKER>   -c, --context string     Name of the context to use to connect to the
[INFO] DOCKER>                            daemon (overrides DOCKER_HOST env var and
[INFO] DOCKER>                            default context set with "docker context use")
[INFO] DOCKER>   -D, --debug              Enable debug mode
[INFO] DOCKER>   -H, --host list          Daemon socket to connect to
[INFO] DOCKER>   -l, --log-level string   Set the logging level ("debug", "info",
[INFO] DOCKER>                            "warn", "error", "fatal") (default "info")
[INFO] DOCKER>       --tls                Use TLS; implied by --tlsverify
...
[INFO] DOCKER>   -v, --version            Print version information and quit
[INFO] DOCKER> 
[INFO] DOCKER> Run 'docker COMMAND --help' for more information on a command.
[INFO] DOCKER> 
[INFO] DOCKER> For more help on how to use Docker, head to https://docs.docker.com/go/guides/
[INFO] DOCKER> 
[ERROR] DOCKER> Error status (125) while creating builder maven
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.094 s
[INFO] Finished at: 2023-08-23T12:50:29+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.43.4:build (default-cli) ...
...
Error status (125) while creating builder maven -> [Help 1]
rohanKanojia commented 1 year ago

@jramosf : Could you please share a reproducer project? Which platform are you on?

rohanKanojia commented 1 year ago

@jramosf : polite ping, Could you please share a reproducer project and the type of environment you're on? What difference do you see in executed docker commands when you run it with v0.43.0 and v0.43.4?

saravanaa14 commented 1 year ago

This appears to be a problem with M2 Macs. I have an existing project that builds properly on Intel Mac however it throws the above error that @jramosf posted

[INFO] DOCKER> [dockerhub.com/sample-project/sample-artifact-id:1.0.2-SNAPSHOT-20230905134913] "service": Created docker-build.tar in 34 milliseconds [INFO] DOCKER> Credentials helper reply for "docker-credential-desktop" is docker-credential-desktop (github.com/docker/docker-credential-helpers) v0.7.0 [INFO] DOCKER> docker --config /Users/ABCD/github/sample-project/target/docker/dockerhub.com/sample-project/sample-artifact-id/1.0.2-SNAPSHOT-20230905134913/docker buildx create --driver docker-container --name maven --config /Users/ABCD/github/sample-project/target/docker/buildkit.toml [INFO] DOCKER> unknown flag: --driver [INFO] DOCKER> See 'docker --help'. [INFO] DOCKER> [INFO] DOCKER> Usage: docker [OPTIONS] COMMAND

I noticed something different in the structure of target folder between old & new Mac but I'm unsure if its of any significance

Intel Macbook pro: image

M2 Max MacBook Pro:

image

We are currently using maven plugin version 0.43.0 but I see the same issue in latest version also. Both Macs running Docker version 24.0.5, build ced0996

Edit: We use enterprise SSO for docker login if that matters

rohanKanojia commented 1 year ago

@saravanaa14 : Do you mean this could be a bug in Docker CLI?

saravanaa14 commented 1 year ago

no. All I'm saying is, this problem seems to be related to the plugin only.

docker --config /Users/ABCD/github/sample-project/target/docker/dockerhub.com/sample-project/sample-artifact-id/1.0.2-SNAPSHOT-20230905134913/docker buildx create --driver docker-container --name maven --config /Users/ABCD/github/sample-project/target/docker/buildkit.toml [INFO] DOCKER> unknown flag: --driver [INFO] DOCKER> See 'docker --help'. [INFO] DOCKER> [INFO] DOCKER> Usage: docker [OPTIONS] COMMAND

In this above command generated, if I remove --config at the start of the line, the buildx instance is successfully getting created.

rohanKanojia commented 1 year ago

@saravanaa14 : Could you please share your docker CLI version?

saravanaa14 commented 1 year ago

Docker version 24.0.5, build ced0996 Docker desktop 4.22.1 (118664)

jramosf commented 1 year ago

@jramosf : polite ping, Could you please share a reproducer project and the type of environment you're on? What difference do you see in executed docker commands when you run it with v0.43.0 and v0.43.4?

Apologies, did not see this. It's exactly the error you pasted above with Macs using Apple Silicon.

tfactor2 commented 1 year ago

Looks like we need to pass --config after docker buildx.

This works:

docker buildx create --config /Users/user/project/target/docker/repo/project/version/docker/config.json --driver docker-container --name service-name

I had to create a config.json inside the docker folder, passing the docker folder as it's done by the plugin causes:

ERROR: failed to parse config: read /Users/user/project/target/docker/repo/project/version/docker: is a directory
drcca commented 1 year ago

In case this helps anyone. I ran into this issue with these versions:

Docker version 24.0.5, build ced0996 Docker desktop 4.22.1 (118664)

I then upgraded to the latest and greatest 4.23.0 and 24.0.6 and that still didn't help.

A colleague had a setup that worked but was on an older version. One major difference was that he had this symlink. /usr/local/lib/docker/cli-plugins -> /Applications/Docker.app/Contents/Resources/cli-plugins Once I created that symlink stuff started to work again. Not sure if this is correct or could end up masking other problems but I decided to put it out there... (I'm sure most people have cli-plugins living under ~/.docker/cli-plugins which symlink to the same directory under /Applications)

dadoonet commented 11 months ago

Hey

I'm getting the same issue when building a multiarch on macOS. The same exact code with 0.43.2 is passing. Upgrading to 0.43.4 is failing.

I'm feeling that the error is coming from this change: https://github.com/fabric8io/docker-maven-plugin/commit/a339536df08246483ccc56de59638783e77ab142

To describe a bit more, I'm building 2 images.

With 0.43.2:

First image:

[INFO] DOCKER> docker buildx create --driver docker-container --name maven --node maven0
[INFO] DOCKER> maven

Second image:

[INFO] DOCKER> docker buildx create --driver docker-container --name maven --node maven0
[INFO] DOCKER> maven

With 0.43.4:

First image:

[INFO] DOCKER> docker buildx create --driver docker-container --name maven
[INFO] DOCKER> maven

Second image:

[INFO] DOCKER> docker buildx create --driver docker-container --name maven
[INFO] DOCKER> ERROR: existing instance for "maven" but no append mode, specify --node to make changes for existing instances

Sounds like it was caused by this change: https://github.com/fabric8io/docker-maven-plugin/pull/1703

Some details about the versions I'm using:

Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/dpilato/.sdkman/candidates/maven/current
Java version: 19.0.2, vendor: Oracle Corporation, runtime: /Users/dpilato/.sdkman/candidates/java/19.0.2-oracle
Default locale: fr_FR, platform encoding: UTF-8
OS name: "mac os x", version: "13.6", arch: "x86_64", family: "mac"
Docker version 20.10.23, build 7155243

I tried to add manually the node name but no luck. It's not taken into account:

            <plugin>
                <groupId>io.fabric8</groupId>
                <artifactId>docker-maven-plugin</artifactId>
                <configuration>
                    <authConfig>
                        <push>
                            <username>${docker.push.username}</username>
                            <password>${docker.push.password}</password>
                        </push>
                    </authConfig>
                    <!-- Create multiple images -->
                    <images combine.self="override">
                        <image>
                            <external>
                                <type>properties</type>
                                <prefix>docker.noocr</prefix>
                            </external>
                            <build>
                                <buildx>
                                    <nodeName>maven</nodeName>
                                </buildx>
                            </build>
                        </image>
                        <image>
                            <external>
                                <type>properties</type>
                                <prefix>docker.ocr</prefix>
                            </external>
                            <build>
                                <buildx>
                                    <nodeName>maven</nodeName>
                                </buildx>
                            </build>
                        </image>
                    </images>
                </configuration>
                <executions>
                    <!-- Generate the docker images during the package phase -->
                    <execution>
                        <id>docker-build</id>
                        <phase>package</phase>
                        <goals>
                            <goal>build</goal>
                        </goals>
                    </execution>
                    <!-- Push the docker images during the "deploy" phase -->
                    <execution>
                        <id>docker-push</id>
                        <phase>deploy</phase>
                        <goals>
                            <goal>push</goal>
                        </goals>
                    </execution>
                    <execution>
                        <!-- There is no integration test, so we can skip that execution -->
                        <id>start-elasticsearch</id>
                        <phase>none</phase>
                    </execution>
                    <execution>
                        <!-- There is no integration test, so we can skip that execution -->
                        <id>stop-elasticsearch</id>
                        <phase>none</phase>
                    </execution>
                </executions>
            </plugin>

This is still calling docker buildx create --driver docker-container --name maven with no node.

[INFO] DOCKER> docker buildx create --driver docker-container --name maven
[INFO] DOCKER> ERROR: existing instance for "maven" but no append mode, specify --node to make changes for existing instances

As a workaround, I'm switching back to 0.43.2.

Hope this helps.

rohanKanojia commented 7 months ago

@poikilotherm @dadoonet @drcca @tfactor2 @saravanaa14 : Hello, I've reverted some of my docker --config related changes that I did in previous releases.

Could you please give a try to Docker Maven Plugin 0.44-SNAPSHOT and provide feedback on whether docker-maven-plugin is working as expected for your pipelines?

rohanKanojia commented 7 months ago

I've released 0.44.0 . I've tested it by building and pushing image to ghcr.io using buildx on Windows, MacOS (via GitHub action) and Linux.

I think your issue might be fixed by this new release. I would appreciate it if you could try it out and provide feedback.

dadoonet commented 6 months ago

I've released 0.44.0 . I've tested it by building and pushing image to ghcr.io using buildx on Windows, MacOS (via GitHub action) and Linux.

I think your issue might be fixed by this new release. I would appreciate it if you could try it out and provide feedback.

This is indeed fixing my issue. Thanks a lot!