Support caching build layers to a bind mount

[scottfrederick]

The spring-boot:build-image Maven goal and bootBuildImage Gradle task configure the CNB builder to cache layers created by buildpacks to make subsequent image builds run faster. Currently the cache is stored as named volume in the Docker daemon. This doesn't work well when running builds on BitBucket, as discussed in https://github.com/buildpacks/pack/issues/1077#issuecomment-805883369.

To better support running builds in BitBucket pipelines the Boot plugins could be enhanced to support storing the build and launch caches in bind mounts.

For Maven, the build configuration would be:

<project>
  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
          <image>
            <buildCache>
              <bind>
                <source>./build-cache-dir</source>
              </bind>
            </buildCache>
            <launchCache>
              <bind>
                <source>./launch-cache-dir</source>
              </bind>
            </launchCache>
          </image>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

For Gradle, the build configuration would be:

bootBuildImage {
  buildCache {
    bind {
      source = "./build-cache-dir"
    }
  }
  launchCache {
    bind {
      source = "./launch-cache-dir"
    }
  }
}

[lazystone]

This is really needed for us, poor souls, forced to use BitBucket pipelines...

It seems like it's not possible to run buildpacks in BitBucket CI/CD, and Atlassian being Atlassian, swept this problem under the rug 5 years ago, so, maybe Spring Boot being Spring Boot can solve it instead?

[tompson]

Yes, please! We had to switch back to using JIB because BitBucket pipelines do not support this!

[scottfrederick]

Thanks for the feedback and the confidence in our ability to help developers. This does help us prioritize enhancements. I'll tentatively tag this for 3.2.

[scottfrederick]

@lazystone @tompson This change should be available in a 3.2.0-SNAPSHOT build. If you have the ability to test this in your BitBucket pipelines, any feedback would be appreciated.

[tompson]

@scottfrederick I would be glad to test this - what do I have to configure?

just using version 3.2.0-SNAPSHOT and running gradle bootBuildImage still ends in

Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' ..................................................

 > Pulled builder image 'paketobuildpacks/builder-jammy-base@sha256:750c1b2e8e6e82decdbec96fcd061e31d7f73b492319cfd16ae2aca174e71ee5'

 > Pulling run image '[docker.io/paketobuildpacks/run-jammy-base:latest](http://docker.io/paketobuildpacks/run-jammy-base:latest)' ..................................................

 > Pulled run image 'paketobuildpacks/run-jammy-base@sha256:4cf369b562808105d3297296efea68449a2ae17d8bb15508f573cc78aa3b3772'

 > Executing lifecycle version v0.17.0

 > Using build cache bind mount './build-cache-dir'

 > Running creator

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:

Execution failed for task ':bootBuildImage'.

> Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"

[scottfrederick]

@tompson There is an example of configuring the caches to use bind mounts in the snapshot documentation here: https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/gradle-plugin/reference/htmlsingle/#build-image.examples.caches. It looks like there is a BITBUCKET_WORKSPACE environment variable available to BitBucket pipelines that you might be able to use in the source values as a root directory.

[tompson]

I configured

tasks.named("bootBuildImage") {
    buildCache {
        bind {
            source = System.getenv('BITBUCKET_CLONE_DIR') + "/cache-build"
        }
    }
    launchCache {
        bind {
            source = System.getenv('BITBUCKET_CLONE_DIR') + "/cache-launch"
        }
    }
}

an still get

> Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' ..................................................
> Pulled builder image 'paketobuildpacks/builder-jammy-base@sha256:750c1b2e8e6e82decdbec96fcd061e31d7f73b492319cfd16ae2aca174e71ee5'
> Pulling run image '[docker.io/paketobuildpacks/run-jammy-base:latest](http://docker.io/paketobuildpacks/run-jammy-base:latest)' ..................................................
> Pulled run image 'paketobuildpacks/run-jammy-base@sha256:4cf369b562808105d3297296efea68449a2ae17d8bb15508f573cc78aa3b3772'
> Executing lifecycle version v0.17.0
> Using build cache bind mount '/opt/atlassian/pipelines/agent/build/cache-build'

> Running creator

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:

Execution failed for task ':bootBuildImage'.
> Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"

the docker log says

time="2023-08-23T14:38:15.758346838Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2023-08-23T14:38:15.758632799Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
time="2023-08-23T14:38:15.758648160Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
time="2023-08-23T14:38:16.758905080Z" level=warning msg="Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message" host="tcp://0.0.0.0:2375"
time="2023-08-23T14:38:16.758946704Z" level=warning msg="Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network" host="tcp://0.0.0.0:2375"
time="2023-08-23T14:38:16.758963821Z" level=warning msg="You can override this by explicitly specifying '--tls=false' or '--tlsverify=false'" host="tcp://0.0.0.0:2375"
time="2023-08-23T14:38:16.758970670Z" level=warning msg="Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release" host="tcp://0.0.0.0:2375"
time="2023-08-23T14:38:31Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
time="2023-08-23T14:38:31.873315013Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-08-23T14:38:31.873870093Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-08-23T14:38:31.936312944Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"
time="2023-08-23T14:38:31.936376385Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2023-08-23T14:38:32.363653787Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
time="2023-08-23T14:38:32.363675551Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2023-08-23T14:38:32.363680705Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2023-08-23T14:38:58Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?[fromImage=docker.io](http://fromimage=docker.io/)%2Fpaketobuildpacks%2Fbuilder-jammy-base%3Alatest"
time="2023-08-23T14:39:16Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/builder-jammy-base:latest/json"
time="2023-08-23T14:39:16Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?[fromImage=docker.io](http://fromimage=docker.io/)%2Fpaketobuildpacks%2Frun-jammy-base%3Alatest"
time="2023-08-23T14:39:17Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/run-jammy-base:latest/json"
time="2023-08-23T14:39:17Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.24/images/load
time="2023-08-23T14:39:17Z" level=info msg="Container create request." ArgsEscaped=false AttachStderr=false AttachStdin=false AttachStdout=false ExposedPorts="map[]" Healthcheck="<nil>" Labels="map[author:spring-boot]" MacAddress= NetworkDisabled=false OnBuild="[]" OpenStdin=false StdinOnce=false StopSignal= StopTimeout="<nil>" Tty=false plugin=pipelines
time="2023-08-23T14:39:17Z" level=info msg="Container create request." AutoRemove=false BlkioDeviceReadBps="[]" BlkioDeviceReadIOps="[]" BlkioDeviceWriteBps="[]" BlkioDeviceWriteIOps="[]" BlkioWeight=0 BlkioWeightDevice="[]" CPUCount=0 CPUPercent=0 CPUPeriod=0 CPUQuota=0 CPURealtimePeriod=0 CPURealtimeRuntime=0 CPUShares=0 CapAdd="[]" CapDrop="[]" Cgroup= CgroupParent= ConsoleSize="[0 0]" ContainerIDFile= CpusetCpus= CpusetMems= DNS="[]" DNSOptions="[]" DNSSearch="[]" DeviceCgroupRules="[]" Devices="[]" ExtraHosts="[]" GroupAdd="[]" IOMaximumBandwidth=0 IOMaximumIOps=0 Init="<nil>" IpcMode= Isolations= KernelMemory=0 Links="[]" LogConfig="{ map[]}" MaskedPaths="[]" Memory=0 MemoryReservation=0 MemorySwap=0 MemorySwappiness="<nil>" Mounts="[]" NanoCPUs=0 NetworkMode=default OomKillDisable="<nil>" OomScoreAdj=0 PidMode= PidsLimit="<nil>" PortBindings="map[]" Privileged=false PublishAllPorts=false ReadOnlyPaths="[]" RestartPolicy="{ 0}" Runtime= SecurityOpt="[label=disable]" ShmSize=0 StorageOpt="map[]" Sysctls="map[]" Ulimits="[]" UsernsMode= VolumeDriver= VolumesFrom="[]" plugin=pipelines
time="2023-08-23T14:39:17Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.24/containers/create
time="2023-08-23T14:39:17.706802923Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"
time="2023-08-23T14:39:17Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-layers-nclwtvgtxp?force=1"
time="2023-08-23T14:39:17Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-app-wknaqmhhnj?force=1"
time="2023-08-23T14:39:17Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/images/pack.local/builder/fvcqizalhe:latest?force=1"

[scottfrederick]

@tompson Thanks for testing the changes. I'm looking into what's going on now. We do mount a few other volumes, but I didn't think those were a problem. We'll do what we can to get this working on BitBucket.

[tompson]

thanks, will be happy to test again!

[scottfrederick]

@tompson @lazystone #37478 and #37479 have been implemented to further address the challenges of building images on BitBucket pipelines. With the following configuration, I can get the builder to start running without throwing any errors about container configuration:

tasks.named("bootBuildImage") {
    securityOptions = []
    buildWorkspace {
        bind {
            source = "${System.env.BITBUCKET_CLONE_DIR}/build/work"
        }
    }
    buildCache {
        bind {
            source = "${System.env.BITBUCKET_CLONE_DIR}/build/cache-build"
        }
    }
    launchCache {
        bind {
            source = "${System.env.BITBUCKET_CLONE_DIR}/build/cache-launch"
        }
    }
}

With this configuration, the CNB builder tries to run the phases of the lifecycle but fails the first time that a process in the builder tries to access the Docker daemon. By default, the Boot plugins and the CNB lifecycle running in the builder container will try to access the Docker daemon with a socket at /var/run/docker.sock. In the pipelines I used for testing, BitBucket sets the environment variable DOCKER_HOST to a value like tcp://localhost:2375, which the Boot plugins will use instead of the default.

The following configuration can be applied to tell the builder to use the same value from DOCKER_HOST:

tasks.named("bootBuildImage") {
    docker {
        bindHostToBuilder = true
    }
    ...
}

Unfortunately, I get a permissions error when the builder tries to use /var/run/docker.sock and an error like this when the provided tcp://localhost:2375 Docker host is used:

> Running creator
    [creator]     ===> ANALYZING
    [creator]     ERROR: failed to initialize analyzer: getting previous image: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?

This is out of Spring Boot's control, as the code that has the problem is now inside the CNB builder. There might be a way to allow the processes inside the builder container to use the Docker daemon, but I haven't found it yet. Hopefully someone with more knowledge of BitBucket pipelines, or a support channel to Atlassian, can get more information on this.

[lazystone]

I tried following config locally, but got an exception at the end.

        <configuration>
          <docker>
            <bindHostToBuilder>true</bindHostToBuilder>
          </docker>
          <image>
            <env>
              <BP_JVM_VERSION>21</BP_JVM_VERSION>
            </env>
            <!-- https://github.com/spring-projects/spring-boot/issues/28387 -->
            <buildWorkspace>
              <bind>
                <source>/tmp/cache-${project.artifactId}.work</source>
              </bind>
            </buildWorkspace>
            <buildCache>
              <bind>
                <source>/tmp/cache-${project.artifactId}.build</source>
              </bind>
            </buildCache>
            <launchCache>
              <bind>
                <source>/tmp/cache-${project.artifactId}.launch</source>
              </bind>
            </launchCache>
          </image>
        </configuration>

[INFO]     [creator]     Timer: Saving docker.io/library/webhook-sender:0.0.1-SNAPSHOT... ran for 393.724383ms and ended at 2023-10-04T12:55:49Z
[INFO]     [creator]     Timer: Exporter ran for 3.541767394s and ended at 2023-10-04T12:55:49Z
[INFO]     [creator]     Timer: Cache started at 2023-10-04T12:55:49Z
[INFO]     [creator]     Adding cache layer 'paketo-buildpacks/syft:syft'
[INFO]     [creator]     Adding cache layer 'buildpacksio/lifecycle:cache.sbom'
[INFO]     [creator]     Timer: Cache ran for 190.425586ms and ended at 2023-10-04T12:55:49Z
[INFO] 
[INFO] Successfully built image 'docker.io/library/webhook-sender:0.0.1-SNAPSHOT'
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17.222 s
[INFO] Finished at: 2023-10-04T14:55:49+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image (default-cli) on project webhook-sender: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:3.2.0-SNAPSHOT:build-image failed: Error cleaning bind mount directory '/tmp/cache-webhook-sender.work-layers': /tmp/cache-webhook-sender.work-layers/report.toml -> [Help 1]

[lazystone]

Ok, to whoever comes here - I've got it running in bitbucket.

Secret ingredient one. In bitbucket pipeline:

export DOCKER_HOST=tcp://172.17.0.1:2375

Secret ingredient two(note docker configuration). If you use maven, then:

        <configuration>
          <docker>
            <host>tcp://172.17.0.1:2375</host>
            <bindHostToBuilder>true</bindHostToBuilder>
          </docker>
          <image>
            <env>
              <BP_JVM_VERSION>${maven.compiler.release}</BP_JVM_VERSION>
            </env>
            <securityOptions></securityOptions>
            <buildWorkspace>
              <bind>
                <source>/opt/atlassian/bitbucketci/agent/build/cache-${project.artifactId}.work</source>
              </bind>
            </buildWorkspace>
            <buildCache>
              <bind>
                <source>/opt/atlassian/bitbucketci/agent/build/cache-${project.artifactId}.build</source>
              </bind>
            </buildCache>
            <launchCache>
              <bind>
                <source>/opt/atlassian/bitbucketci/agent/build/cache-${project.artifactId}.launch</source>
              </bind>
            </launchCache>
          </image>
        </configuration>

There is a huge room for improvement, but at least it works now.

[oliverheil]

@scottfrederick Hi Scott, Is there any chance that this new param 'buildWorkspace' (which you did in https://github.com/spring-projects/spring-boot/commit/4433fcd1f2c6d9c9a912eb89b2b44ce2d3f7b512) will be available in the latest Spring boot 2.7-x Version?

[oliverheil]

@scottfrederick This gradle config works for us with the latest Spring boot Snapshot (3.2.0-SNAPSHOT) in Bitbucket Cloud pipeline (export DOCKER_HOST=tcp://172.17.0.1:2375). Thanks!

tasks.named('bootBuildImage') {
    docker {
        host = "tcp://172.17.0.1:2375"
        bindHostToBuilder = true
        buildWorkspace {
            bind {
                source = "/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.work"
            }
        }
        buildCache {
            bind {
                source = "/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.build"
            }
        }
        launchCache {
            bind {
                source = "/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.launch"
            }
        }
    }
}

[lazystone]

@oliverheil SB 2.7 is EOL on 18 Nov 2023

[scottfrederick]

@oliverheil These changes for the additional configuration to use bind mounts instead of volumes were enhancements, and we don't backport enhancements to older versions of Spring Boot. They will only be available in versions 3.2.0 and later.

[oliverheil]

@scottfrederick ok, thanks for the informations.

[dbaltor]

@lazystone Thank you very much for finding and sharing the solution. :tada: :pray:

I've just tested and we don't need to set the docker host within the spring-boot-maven-plugin configuration. You can remove <host>tcp://172.17.0.1:2375</host> . The export DOCKER_HOST=tcp://172.17.0.1:2375 config within the pipeline step is enough.

[physiologykurt]

@scottfrederick The snippets featured here work with Spring Boot 3.2.5 and lower and fail with Spring Boot 3.2.6 and 3.3.0. Any ideas why?

[scottfrederick]

@physiologykurt Can you provide more information on how it fails, and the build configuration that you're using?

There was one issue that was fixed in Boot 3.2.6 and 3.3.0. That issue only affected the Gradle plugin (not the Maven plugin), and it had to do with the securityOptions setting. On Linux, the default for securityOptions should be ["label=disable"], but that bug was causing the default to be [] instead. The fix restored the proper default, but it now might be necessary to explicitly set securityOptions = [] as shown in my snippet above. I see that securityOptions = [] was omitted from some of the other Gradle snippets in the thread.

[kurtostfeld]

@scottfrederick when I try to add securityOptions = [] to my build.gradle.kts, I get the error Val cannot be reassigned on org.springframework.boot.gradle.tasks.bundling.BootBuildImage getSecurityOptions

I used the spring initializr website (https://start.spring.io/) to generate a Spring Boot 3.2.6 skeleton project with:

• Project: Gradle - Kotlin
• Language: Java
• Spring Boot: 3.2.6
• Packaging: jar
• Java: 21
• Dependencies: None

I add the following to the build.gradle.kts:

tasks.named<BootBuildImage>("bootBuildImage") {
    // The block below is for compatibility with Spring Boot 3.2.x and bitbucket-pipelines
    // https://github.com/spring-projects/spring-boot/issues/28387
    docker {
        host = "tcp://172.17.0.1:2375"
        bindHostToBuilder = true
        buildWorkspace {
            bind {
                source = "/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.work"
            }
        }
        buildCache {
            bind {
                source = "/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.build"
            }
        }
        launchCache {
            bind {
                source = "/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.launch"
            }
        }
    }
}

and have the following bitbucket-pipelines.yml:

pipelines:
  default:
    - step:
        name: bootBuildImage
        image: gradle:8.8-jdk21
        services:
          - docker
        size: 2x
        script:
          - export DOCKER_HOST=tcp://172.17.0.1:2375
          - gradle bootBuildImage --warning-mode all --imageName demo:${BITBUCKET_BUILD_NUMBER}

This works with

id("org.springframework.boot") version "3.2.5"

This fails with:

id("org.springframework.boot") version "3.2.6"

[scottfrederick]

@kurtostfeld The examples above are for Gradle with Groovy. When using Gradle with Kotlin, the configuration should look like this:

tasks.named<BootBuildImage>("bootBuildImage") {
    docker {
        host.set("tcp://172.17.0.1:2375")
        bindHostToBuilder.set(true)
        securityOptions.set(listOf())
        buildWorkspace {
            bind {
                source.set("/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.work")
            }
        }
        buildCache {
            bind {
                source.set("/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.build")
            }
        }
        launchCache {
            bind {
                source.set("/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.launch")
            }
        }
    }
}

Please test again with securityOptions set this way with Spring Boot 3.2.6 and see if that solves your problem.

[kurtostfeld]

      securityOptions.set(listOf())

This fixed it! bootBuildImage is now working with Spring Boot 3.2.6 and 3.3.0 as well as the previous versions. Thank you! :)

[TMRGZ]

I think I found a problem that could be related to this, in this case is building the image with CDS using Bitbucket pipelines, I've reported it in the buildpacks repository.

https://github.com/paketo-buildpacks/spring-boot/issues/497

[scottfrederick]

@TMRGZ Your problem does seem to be related in that it is caused by restricted permissions in Docker containers running on BitBucket, but not directly related to any of the changes we have made to the Spring Boot plugins to allow users to configure the builder container in a way that works with BitBucket. If you can find out from Atlassian documentation or support if there are any container configuration options that would allow this to work, we could consider adding more options to the Spring Boot plugins. Otherwise, I'm afraid there's nothing Spring Boot can do here.

[TMRGZ]

I've found another issue that could be related (https://github.com/buildpacks/community/discussions/229), probably just changing the folder /workspace permissions works, There is any documentation about what files/folers are affected during the CDS contribution step?

I'll report back if I can make this work for anyone that is interested in all these Bitbucket workarounds

[scottfrederick]

There is any documentation about what files/folders are affected during the CDS contribution step?

Any documentation about this would belong in the Paketo buildpack documentation, not in the Spring Boot documentation, since those decisions are out of Spring Boot's control.

[tompson]

it seems there is a regression because this worked in Spring Boot 3.2.10 but does not work anymore with Spring Boot 3.3.4

[lazystone]

Can't confirm - still works for us in BitBucket CI/CD.

Spring Boot 3.3.4, Java 21.

[tompson]

@lazystone I created a public repository https://bitbucket.org/troiisoftware/pipelines-regression-demo/

the main branch with Spring Boot 3.2.4 works, the branch spring-3-3 with Spring Boot 3.3.4 fails

Could you please have a look at it?

[lazystone]

Ah, we use maven. But I've noticed that you might be missing "securityOptions.set(listOf())" comparing to https://github.com/spring-projects/spring-boot/issues/28387#issuecomment-2148453495

Could you try to set it? In maven config we also have it setting it to empty.

[tompson]

thanks @lazystone - adding securityOptions helps, interestingly that was not required for 3.2

[scottfrederick]

the main branch with Spring Boot 3.2.4 works, the branch spring-3-3 with Spring Boot 3.3.4 fails

@tompson This comment above explains what you are seeing.