Building docker image with spring boot 2.4.1 failes with “Missing 'io.buildpacks.stack.id' stack label” when behind a firewall

[kajh]

I'm, trying to build a docker image with ./mvnw -DskipTests spring-boot:build-image using spring boot 2.4.1 and java 11 (openjdk version "11.0.9" 2020-10-20 LTS) on RHEL7.

I do this on a host behind a strict firewall so I have to fetch the build- and runimage from a private repo. I have configured the spring-boot-maven-plugin to use this repo:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <docker>
            <builderRegistry>
                <username>my-username</username>
                <password>xxx</password>
                <url>https://my-mirror.com</url>
                <email>kaj.hejer@usit.uio.no</email>
            </builderRegistry>
        </docker>

        <image>
            <builder>my-mirror.com/library/docker.io-paketobuildpacks-builder:base</builder>
            <runImage>my-mirror.com/library/docker.io-paketobuildpacks-run:base</runImage>
            <name>my-mirror.com/my-group/my-app:latest</name>
            <verboseLogging>true</verboseLogging>
        </image>
    </configuration>
</plugin>

The build fails with

[INFO]  > Pulling builder image 'my-mirror.com/library/docker.io-paketobuildpacks-builder:base' 100%
[INFO]  > Pulled builder image 'my-mirror.com/library/docker.io-paketobuildpacks-builder@sha256:cf90221a33966e42f8b1960123dea4406c65fc6a410142ded573ed850ccc313b'
[INFO]  > Pulling run image 'my-mirror.com/library/docker.io-paketobuildpacks-run:base' 100%
[INFO]  > Pulled run image 'my-mirror.com/library/docker.io-paketobuildpacks-run@sha256:56fb7587103da155db6d4f9434fd7e2f9e45d7540a062847fd84e9132a28101b'
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.090 s
[INFO] Finished at: 2020-12-17T08:36:48+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.4.1:build-image (default-cli) on project my-app: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.4.1:build-image failed: Missing 'io.buildpacks.stack.id' stack label -> [Help 1]
[ERROR] 

sudo journalctl -f don't list anything when running the mvnw command.

When I try to build a docker image the same way but without the configuration block for the spring-boot-maven-plugin on my mac which is not behind a firewall it works just fine.

I have asked a question about this on https://stackoverflow.com/questions/65336624/building-docker-image-with-spring-boot-2-4-1-failes-with-missing-io-buildpacks and found a similar question for using gradle on https://stackoverflow.com/questions/65344886/unable-to-build-image-using-gradle-bootbuildimage-in-offline-environment.

[wilkinsona]

It looks like the image in your private repository has lost its io.buildpacks.stack.id label. You can check for its presence using docker inspect. Can you please double-check that the images in your private repository have the same metadata as those available publicly?

[scottfrederick]

When copying the Paketo builder into a private repository, make sure you are copying docker.io/paketobuildpacks/builder not docker.io/paketobuildpacks/build. I've made that mistake before with similar results.

[kajh]

@wilkinsona Thanks for replying! It seems like the "io.buildpacks.stack.id" is present:

$ docker inspect myrepo.com/library/docker.io-paketobuildpacks-builder:base | grep io.buildpacks.stack.id
                "io.buildpacks.stack.id": "io.buildpacks.stacks.bionic",

I diffed docker inspect for the image in my private repo with the one in dockerhub. There are som differences but I'm unsure if they are relevant. I put a textfile with the diff on https://www.dropbox.com/s/ri46btqyn25ws8j/diff.txt?dl=0.

[kajh]

@scottfrederick Thanks for replying! It is the builder image that is copied into our privare repo as far as I can understand. We have a list of images that are synced from dockerhub.

[wilkinsona]

Thanks, @kajh. It could also be the run image that is missing its stack id. Can you please double-check its metadata as well? If that appears to be correct, the stack trace of the failure (available when running Maven with -X) may help.

[kajh]

@wilkinsona Thanks! I have diffed the run-images and the diff is available at https://www.dropbox.com/s/fwv65ovpho0mfup/diff-run.txt?dl=0. Neither of images contains any labels. This is from my mac outside our firewall:

$ docker pull docker.io/paketobuildpacks/run:base 
base: Pulling from paketobuildpacks/run
Digest: sha256:15d03f2ceb34ddedf8be3654d2e1e8ff2f60aa43e1b4f60d0d6b3624b34162dc
Status: Image is up to date for paketobuildpacks/run:base
docker.io/paketobuildpacks/run:base

$ docker inspect  docker.io/paketobuildpacks/run:base | grep Label
            "Labels": null
            "Labels": null

I ran a ./mvnw -X spring-boot:build-image and the buildlog is available at https://www.dropbox.com/s/iwwglb0vv8a6djq/build-image.txt?dl=0

[wilkinsona]

I think that's the wrong run image. The run image is identified in the io.buildpacks.builder.metadata label of the builder. The metadata for docker.io/paketobuildpacks/builder:base lists its run image as index.docker.io/paketobuildpacks/run:base-cnb. Note that's it's base-cnb rather than base.

$ docker inspect index.docker.io/paketobuildpacks/run:base-cnb | grep -C 1 Labels
            "OnBuild": null,
            "Labels": {
                "io.buildpacks.stack.id": "io.buildpacks.stacks.bionic",
--
--
            "OnBuild": null,
            "Labels": {
                "io.buildpacks.stack.id": "io.buildpacks.stacks.bionic",

[kajh]

@wilkinsona Aha, I see. Thank you! I will try this and let you know how it goes.

[bwgjoseph]

Hi,

I was redirected from SO. I hope its ok to post the information in this issue as well, but if it's better to create as a separate issue, do let me know.

Based on the conversion above, I have checked my own private repo and compared with the one in my online machine, all the information from Labels (io.buildpacks.stack.*) have the exact same information.

Something like this

"Labels": {
                "io.buildpacks.stack.description": "ubuntu:bionic + openssl + CA certs",
                "io.buildpacks.stack.distro.name": "Ubuntu",
                "io.buildpacks.stack.distro.version": "18.04",
                "io.buildpacks.stack.homepage": "https://github.com/paketo-buildpacks/stacks",
                "io.buildpacks.stack.id": "io.buildpacks.stacks.bionic",
                "io.buildpacks.stack.maintainer": "Paketo Buildpacks",
                "io.buildpacks.stack.metadata": "{\"base-image\": \"paketobuildpacks/run@sha256:15d03f2ceb34ddedf8be3654d2e1e8ff2f60aa43e1b4f60d0d6b3624b34162dc\"}",
                "io.buildpacks.stack.mixins": "[\"adduser\",\"apt\",\"base-files\",\"base-passwd\",\"bash\",\"bsdutils\",\"bzip2\",\"ca-certificates\",\"coreutils\",\"dash\",\"debconf\",\"debianutils\",\"diffutils\",\"dpkg\",\"e2fsprogs\",\"fdisk\",\"findutils\",\"gcc-8-base\",\"gpgv\",\"grep\",\"gzip\",\"hostname\",\"init-system-helpers\",\"libacl1\",\"libapt-pkg5.0\",\"libattr1\",\"libaudit-common\",\"libaudit1\",\"libblkid1\",\"libbz2-1.0\",\"libc-bin\",\"libc6\",\"libcap-ng0\",\"libcom-err2\",\"libdb5.3\",\"libdebconfclient0\",\"libext2fs2\",\"libfdisk1\",\"libffi6\",\"libgcc1\",\"libgcrypt20\",\"libgmp10\",\"libgnutls30\",\"libgpg-error0\",\"libhogweed4\",\"libidn2-0\",\"liblz4-1\",\"liblzma5\",\"libmount1\",\"libncurses5\",\"libncursesw5\",\"libnettle6\",\"libp11-kit0\",\"libpam-modules\",\"libpam-modules-bin\",\"libpam-runtime\",\"libpam0g\",\"libpcre3\",\"libprocps6\",\"libseccomp2\",\"libselinux1\",\"libsemanage-common\",\"libsemanage1\",\"libsepol1\",\"libsmartcols1\",\"libss2\",\"libssl1.1\",\"libstdc++6\",\"libsystemd0\",\"libtasn1-6\",\"libtinfo5\",\"libudev1\",\"libunistring2\",\"libuuid1\",\"libyaml-0-2\",\"libzstd1\",\"locales\",\"login\",\"lsb-base\",\"mawk\",\"mount\",\"ncurses-base\",\"ncurses-bin\",\"openssl\",\"passwd\",\"perl-base\",\"procps\",\"sed\",\"sensible-utils\",\"sysvinit-utils\",\"tar\",\"tzdata\",\"ubuntu-keyring\",\"util-linux\",\"zlib1g\"]",
                "io.buildpacks.stack.released": "2020-12-20"
            }

And the output from gradle is as such (copied from SO)

Gradle Test Executor 6 finished executing tests.

> Task :test
Finished generating test XML results (0.008 secs) into: C:\Users\joseph\Desktop\demo (9)\demo\build\test-results\test
Generating HTML test report...
Finished generating test html results (0.004 secs) into: C:\Users\joseph\Desktop\demo (9)\demo\build\reports\tests\test
:test (Thread[Execution worker for ':' Thread 7,5,main]) completed. Took 2.708 secs.
:check (Thread[Execution worker for ':' Thread 7,5,main]) started.

> Task :check
Skipping task ':check' as it has no actions.
:check (Thread[Execution worker for ':' Thread 7,5,main]) completed. Took 0.0 secs.
:build (Thread[Execution worker for ':' Thread 7,5,main]) started.

> Task :build
Skipping task ':build' as it has no actions.
:build (Thread[Execution worker for ':' Thread 7,5,main]) completed. Took 0.0 secs.
:bootBuildImage (Thread[Execution worker for ':' Thread 7,5,main]) started.

> Task :bootBuildImage
Caching disabled for task ':bootBuildImage' because:
  Build cache is disabled
Task ':bootBuildImage' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Building image 'docker.io/library/demo:0.0.1-SNAPSHOT'

 > Pulling builder image 'cr.io/paketobuildpacks/builder:base' ..................................................
 > Pulled builder image 'cr.io/paketobuildpacks/builder@sha256:dfbd2831033f37161f5027dfa46d5a658b29a3302c33aaf0219160a05fe5c12e'

> Task :bootBuildImage FAILED
:bootBuildImage (Thread[Execution worker for ':' Thread 7,5,main]) completed. Took 2.298 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootBuildImage'.
> No 'io.buildpacks.builder.metadata' label found in image config labels 'io.buildpacks.stack.description,io.buildpacks.stack.distro.name,io.buildpacks.stack.distro.version,io.buildpacks.stack.homepage,io.buildpacks.stack.id,io.buildpacks.stack.maintainer,io.buildpacks.stack.metadata,io.buildpacks.stack.mixins,io.buildpacks.stack.released'

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 7s
8 actionable tasks: 8 executed

Let me know if there is any information I can provide. Thanks!

[kajh]

@wilkinsona Just to be sure... is base the correct tag for the builder image?

[wilkinsona]

@bwgjoseph Your image's labels are also incorrect. As the error message has indicated, the builder image should have a io.buildpacks.builder.metadata label as the public paketobuildpacks/builder:base image does.

[wilkinsona]

@kajh Yes, base is the correct tag for the builder image.

[kajh]

@wilkinsona Thanks :)

[bwgjoseph]

@wilkinsona thanks! I will check out my private repo label again to verify

Any idea why would it be missing though? I exported the image tar from my online machine, and copy to my offline machine and load then push to my private repo

[wilkinsona]

@bwgjoseph I can't think of any reason why some labels would remain while others are now missing. I wondered if it could be a size limit (io.buildpacks.builder.metadata's value is quite large) but there are other labels with smaller values are also missing. For example, io.buildpacks.buildpack.order appears to be missing and its value is shorter than that of io.buildpacks.stack.mixins which is present.

[bwgjoseph]

@wilkinsona on that note, what can I do to restore the missing labels? I could try to do another export, but not sure if the same would happen again

[wilkinsona]

Sorry, I'm not sure as I'm not an expert in exporting and importing Docker images. A Docker-focused question on Stack Overflow would be a better approach at this point as your problem isn't specifically related to Spring Boot.

[bwgjoseph]

Agreed, thank you for the pointers. Will try to figure it out

[bwgjoseph]

Hi, just wish to provide an update

After trying to figure out what was the issue (missing labels and all), I realize that I had tag wrong image name to the image. After I re-loaded both images to the correct name and tag, I was able to run the process. But it seem like the process had to download some additional stuff from the internet, it breaks again.

> Task :bootBuildImage
Building image 'docker.io/library/service:0.0.1-SNAPSHOT'

 > Pulling builder image 'm.cr.io/paketobuildpacks/builder:base' ..................................................
 > Pulled builder image 'm.cr.io/paketobuildpacks/builder@sha256:7de2396a3b3c135c7638e695a2cd83f0594f9fb8db039be207f2a6e17bd8e00a'
 > Pulling run image 'm.cr.io/paketobuildpacks/run:base-cnb' ..................................................
 > Pulled run image 'm.cr.io/paketobuildpacks/run@sha256:dfbd2831033f37161f5027dfa46d5a658b29a3302c33aaf0219160a05fe5c12e'
 > Executing lifecycle version v0.9.3
 > Using build cache volume 'pack-cache-c5e5d281afed.build'

 > Running creator
    [creator]     ===> DETECTING
    [creator]     5 of 18 buildpacks participating
    [creator]     paketo-buildpacks/ca-certificates   1.0.1
    [creator]     paketo-buildpacks/bellsoft-liberica 6.0.0
    [creator]     paketo-buildpacks/executable-jar    3.1.3
    [creator]     paketo-buildpacks/dist-zip          2.2.2
    [creator]     paketo-buildpacks/spring-boot       3.5.0
    [creator]     ===> ANALYZING
    [creator]     Previous image with name "docker.io/library/service:0.0.1-SNAPSHOT" not found
    [creator]     ===> RESTORING
    [creator]     ===> BUILDING
    [creator]
    [creator]     Paketo CA Certificates Buildpack 1.0.1
    [creator]       https://github.com/paketo-buildpacks/ca-certificates
    [creator]       Launch Helper: Contributing to layer
    [creator]         Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
    [creator]         Writing profile.d/helper
    [creator]
    [creator]     Paketo BellSoft Liberica Buildpack 6.0.0
    [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
    [creator]       Build Configuration:
    [creator]         $BP_JVM_VERSION              11.*            the Java version
    [creator]       Launch Configuration:
    [creator]         $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    [creator]         $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    [creator]         $JAVA_TOOL_OPTIONS                           the JVM launch flags
    [creator]       BellSoft Liberica JRE 11.0.9: Contributing to layer
    [creator]         Downloading from https://github.com/bell-sw/Liberica/releases/download/11.0.9.1+1/bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz
    [creator]     unable to invoke layer creator
    [creator]     unable to get dependency jre
    [creator]     unable to download https://github.com/bell-sw/Liberica/releases/download/11.0.9.1+1/bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz
    [creator]     unable to request https://github.com/bell-sw/Liberica/releases/download/11.0.9.1+1/bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz
    [creator]     Get "https://github.com/bell-sw/Liberica/releases/download/11.0.9.1+1/bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz": dial tcp: lookup github.com on 192.168.65.1:53: no such host
    [creator]     ERROR: failed to build: exit status 1

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootBuildImage'.
> Builder lifecycle 'creator' failed with status code 145

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 25s
18 actionable tasks: 17 executed, 1 up-to-date

This has nothing to do with spring-boot, and there is some way to override that. For future reference, see bellsoft-liberica, although it seem that the current way to override is still quite troublesome. Will attempt to do so.

But thanks once again for your help.

[kajh]

@wilkinsona Using tag base-cnb worked fine, so you can close this issue for my part. Now I got a [creator] ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: permission denied but that is not related to this issue. Thank you for helping me! :)

[wilkinsona]

No problem. Thanks for letting us know.

[wsams]

fwiw I'm getting ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: permission denied as well when running builds inside a Kubernetes pod. I have access to the docker unix socket. I can run docker commands just fine. (Java 11, Spring Boot 2.5.1, k8s 1.19, maven 3.6, docker 20.10.3)

[scottfrederick]

@wsams Your error does not appear to be related to the original issue that was reported. This is a question that would be better suited to Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.