Improve exploded structure experience for efficient deployments

[snicoll]

The reference guide has a section on unpacking the executable jar to get extra boost. It also has an additional hint for bypassing the bootlader with the impact on losing classpath ordering.

Our work on AppCDS (see https://github.com/spring-projects/spring-framework/issues/31497) has shown that a predicable classpath has an impact on how effective the cache is going to be. Investigating a bit more, it looks doable to provide the above as a first class concept.

Here is a proposal that is hacking layertools with an additional extract2(sic) command: https://github.com/snicoll/spring-boot/tree/gh-38276

Ignoring the fact that this does use layertools for convenience, you can execute the following on any repackaged archive:

java -Djarmode=layertools -jar target/my-app-1.0.0-SNAPSHOT.jar extract2 --destination target/app

This creates a directory with the following structure:

target/app/
├── application
│   └── my-app-1.0.0-SNAPSHOT.jar
├── dependencies
│   ├── ...
│   ├── spring-context-6.1.0-RC2.jar
│   ├── spring-context-support-6.1.0-RC2.jar
│   ├── ...
└── run-app.jar

You can then run the app as follows (assuming you're in the same directory as the previous command):

java -jar target/app/run-app.jar

The run-app.jar has the following characteristics:

• It starts the application class directly (no bootlader used)
• The manifest defines a classpath with the same order as classpath.idx, with application/my-app-1.0.0-SNAPSHOT.jar being in front

my-app-1.0.0-SNAPSHOT.jar has some manifest entries of the original jar so that package.getImplementationVersion() continues to work.

The work on this prototype has led to a number of questions:

• Layertools provide a Command infrastructure and several utilities related to extracting/copying that are not specific to layers. If we want to go the same route with a different jar mode, a significant number of classes should be copied/reimplemented. Perhaps a single jar with a public API and sub-packages for layertools and this new mode could be an option?
• This work could potentially apply to layers themselves. Rather than having dependencies/BOOT-INF/lib and a layer for the bootloader, we could use the same exact structure. Or perhaps the structure above could become a layer configuration (where all libs go to dependencies).
• It's hard for run-app.jar to have sensible File attributes. The command tries to respect the file attributes of files it extracts from the repackaged archive, but run-jar.jar is created on the spot.

We have confirmed that with the prototype, AppCDS is effective (close to 95% classes loaded from the cache).

[sergmain]

About performance - In my project a data exchange between agents conducted via zip files. In one case I had 1k+ zips inside other zip and preparing data for agent by unziping all archives had taken minutes. After switching to com.google.jimfs the time of preparing reduced to hundreds milliseconds. So spending additional memory you can achieve almost zero time for unpacking without making any changes in structure.of archive. You can see a general concept here https://github.com/sergmain/metaheuristic/blob/master/apps/commons/src/test/java/ai/metaheuristic/commons/utils/ZipUtilsTest.java

pom dependency

<dependency>
    <groupId>com.google.jimfs</groupId>
    <artifactId>jimfs</artifactId>
    <version>1.2</version>
</dependency>

repo is here - https://github.com/google/jimfs

[snicoll]

@bclozel and I had a bit of brainstorming on this one and he raised a use case that could become problematic. The solution above ignores the Spring Boot launcher on purpose to make the startup as straightforward as possible. However, it currently does not allow to augment the classpath, which is something that can be problematic. I don't know if this is something we need to take into consideration, but the POC could be adapted accordingly.

[snicoll]

The PoC has been updated to provide the ability to include additional libraries. This is typically used by buildpacks, for instance:

$ java -Djarmode=layertools -jar my-app.jar extract2 --destination target/app --additional-jars jar1.jar,jar2.jar

This copies jar1.jar and jar2.jar available in the current directory to target/app/ext. The two jars are added at the end of the classpath in the manifest of run-app.jar.

One decision could help move the POC into something that can be reviewed. If the above is working for buildpacks, perhaps this could become the default mode of operation for layer tools as well. In other words, merging extract and extract2.

The structure above reuse the concept of "layers" to nicely separate concerns, except it does not expand the structure of the far jar (i.e. dependencies has libraries directly nested rather than BOOT-INF/lib). One could argue it is completely unnecessary and a simple lib directory with all jars (including the generated jar for the application) could be enough.

Merging the two means we need to bring back the launcher, perhaps as an opt-in?

Keeping things separate would be less disturbing. However, the argument that buildpacks would work with the new mechanism, and has to for optimal CDS performance, means two mode of operations that might not be desirable. There are also a lot of infrastructure in jarmode that would either need to be duplicated or abstracted.

I am happy to improve the proposal based on what the team decides.

[wilkinsona]

I spent some time looking at this last week and we discussed it a bit yesterday. The high-level conclusion was that we need to combine support for layers and CDS in a single command. This is necessary as the former needs to know about the latter so that the runner jar can go in the most frequently changing layer.

At the moment, I'm imagining a single command with two options – one for CDS and one for layers – that can each be used individually and that can also be used together. There's quite a lot to figure out in terms of how we get from where we are now to where we want to be in a non-breaking manner. We hope that we might be able to look at this in March but there are other, higher priority items that require our attention in the 3.3.0-M2 timeframe.

[snicoll]

I spent some time looking at this last week and we discussed it a bit yesterday. The high-level conclusion was that we need to combine support for layers and CDS in a single command.

Yes. I guess I should have dumped my thoughts somewhere as I reached the same conclusion. Considering the buildpacks use case, I see it as a continuation of what we've started when we introduce layertools back then. Buildpacks still need all the features that we've built so far, but with a mode of operation that allows to further tune the application before final packaging.

Given the classpath cannot be extended further, a mode of operation could be that run-app.jar is created as part of repackaging the archive, so that its timestamp matches, rather than creating it on the fly when we extract the archive.

Then, we need to review layers.xsd. So far, extract has taken the route of including all artifacts that are present in the archive, but this is no longer the case. Instead, there are libraries, application classes and resources, and a "loader". The loader can either be the Spring Boot loader or this new run-app.jar mode of operation. It's interesting to see that layers.xsd list the resources to include in a layer, not where they end up: the current mode extracts the resource as is in a directory that matches the layer name. This gives us the flexibility to know in advance how many layers there are, how many directories we need, and how to structure run-app.jar upfront.

With buildpacks being updated upfront to support the new infrastructure, we could imagine a mode of operation where the build plugins introduce a flag that defaults to this run-app.jar mode rather than the classic mode we had so far. This would obviously force those who are crafting layered docker images to manually set a property when upgrading.

There are a ton of things that I overlooked, but I am happy to dig further based on the team's guidance when it gets to this. Thanks!

[wilkinsona]

Given the classpath cannot be extended further, a mode of operation could be that run-app.jar is created as part of repackaging the archive, so that its timestamp matches, rather than creating it on the fly when we extract the archive.

We'd like to maintain the current model where the archive produced by the build plugins is the single source of truth. It can then be run as-is using java -jar or it can be further processed to produce a deployment-specific arrangement as it is today. We don't want the build plugins to produce the runner jar (and presumably the directory of dependencies along side it) directly as our feeling is that it will create too much confusion and introduce too much overlap with things like Gradle's application plugin and Maven's assembly plugin.

[mhalbritter]

I've integrated Stephane's work into a new extract command.

When building your application, the resulting uber jar contains the new jarmode tools, which provides list-layers (supersedes layertools list) and extract (supersedes layertools extract).

This provides four ways of extracting:

• java -Djarmode=tools -jar yourjar.jar extract creates a runner.jar and libraries in the lib folder. The runner jar can be executed with java -jar and is CDS friendly.
• java -Djarmode=tools -jar yourjar.jar extract --layers is the same as above, but with layer support. The runner.jar is in the application layer.
• java -Djarmode=tools -jar yourjar.jar extract --launcher extracts the JAR as is, and can then be launched with java org.springframework.boot.loader.launch.JarLauncher.
• java -Djarmode=tools -jar yourjar.jar extract --launcher --layers is the same as above, but with layer support.

java -Djarmode=tools -jar yourjar.jar help gives you the available commands.

java -Djarmode=tools -jar yourjar.jar help extract gives you the help for extract command:

> java -Djarmode=tools -jar yourjar.jar help extract
Extract the contents from the jar

Usage:
  java -Djarmode=tools -jar yourjar.jar extract [options]

Options:
  --launcher                Whether to extract the Spring Boot launcher
  --layers string list      Layers to extract
  --destination string      Directory to extract files to. Defaults to the current working directory
  --libraries string        Name of the libraries directory. Only applicable when not using --launcher. Defaults to lib/
  --runner-filename string  Name of the runner JAR file. Only applicable when not using --launcher. Defaults to runner.jar

[Sineaggi]

@mhalbritter is only the first extraction option (extract) CDS-friendly?

[anthonydahanne]

Hello @mhalbritter ! Thanks for the update! I tested it (with start.spring.io Java 21, 3.3.0-SNAPSHOT, actuator + spring web), and I have a few questions for you!

java -Djarmode=tools -jar yourjar.jar extract creates a runner.jar and libraries in the lib folder. The runner jar can be executed with java -jar and is CDS friendly.

Using that option, simple extract, we obtain a folder with this content:

tree .
.
├── lib
...
│   ├── spring-beans-6.1.4.jar
│   ├── spring-boot-3.3.0-SNAPSHOT.jar
│   ├── spring-boot-actuator-3.3.0-SNAPSHOT.jar
│   ├── spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar
│   ├── spring-boot-autoconfigure-3.3.0-SNAPSHOT.jar
...
└── runner.jar

While I have been able to CDS'ize my app

java -XX:ArchiveClassesAtExit=application.jsa -Dspring.context.exit=onRefresh -jar runner.jar
java -XX:SharedArchiveFile=application.jsa -jar runner.jar

I was wondering if there was any reason you did not pick @sdeleuze unpacking layout, which is:

tree .
.
├── application
│   └── demo-0.0.1-SNAPSHOT.jar
├── dependencies
...
│   ├── spring-beans-6.1.4.jar
│   ├── spring-boot-3.3.0-SNAPSHOT.jar
│   ├── spring-boot-actuator-3.3.0-SNAPSHOT.jar
│   ├── spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar
│   ├── spring-boot-autoconfigure-3.3.0-SNAPSHOT.jar
...
└── run-app.jar

The difference being Sebastien created 2 jars: one with a manifest only (run-app.jar) and one with the user classes only (demo-0.0.1-SNAPSHOT.jar) compared to you generating 1 single jar with user classes + manifest (runner.jar)

At first, I thought it was kind of the same, but when I tried adding a jar to the classpath (such as spring-cloud-bindings), I found out that with @sdeleuze layout, I could add it to the classpath with:

java -cp "/Users/anthonyd2/Downloads/spring-cloud-bindings-2.0.2.jar:application/demo-0.0.1-SNAPSHOT.jar:dependencies/*" \ 
 -Dorg.springframework.cloud.bindings.boot.enable=true   com.example.demo.DemoApplication

skipping the run-app.jar and its manifest to rewrite the classpath.

But with your layout, I can't skip runnner.jar and its manifest, so... the only to change the classpath would be to rewrite the MANIFEST, which would be... not ideal. I could be wrong though, if so , please let me know how to enrich the classpath with your layout.

---

java -Djarmode=tools -jar yourjar.jar extract --layers is the same as above, but with layer support. The runner.jar is in the application layer.

Using this, I could obtain such a layout:

tree .
.
├── application
│   └── runner.jar
├── dependencies
│   └── lib
...
│       ├── spring-beans-6.1.4.jar
...
├── snapshot-dependencies
│   └── lib
│       ├── spring-boot-3.3.0-SNAPSHOT.jar
│       ├── spring-boot-actuator-3.3.0-SNAPSHOT.jar
│       ├── spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar
│       └── spring-boot-autoconfigure-3.3.0-SNAPSHOT.jar
└── spring-boot-loader

First, not sure why there's an empty folder named spring-boot-loader ; but it does not matter. Then, the manifest in runner.jar references the simple extract layout , for example: Class-Path: lib/spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar... making the jar non usable:

java -jar application/runner.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/boot/SpringApplication
    at com.example.demo.DemoApplication.main(DemoApplication.java:10)

I don't think this is usable as such, please let me know if I missed something

---

java -Djarmode=tools -jar yourjar.jar extract --launcher extracts the JAR as is, and can then be launched with java org.springframework.boot.loader.launch.JarLauncher.

the layout I got was:

tree .
.
├── BOOT-INF
│   ├── classes
│   │   ├── application.properties
│   │   └── com
│   │       └── example
│   │           └── demo
│   │               └── DemoApplication.class
│   ├── classpath.idx
│   ├── layers.idx
│   └── lib
...
│       ├── spring-beans-6.1.4.jar
│       ├── spring-boot-3.3.0-SNAPSHOT.jar
│       ├── spring-boot-actuator-3.3.0-SNAPSHOT.jar
│       ├── spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar
│       ├── spring-boot-autoconfigure-3.3.0-SNAPSHOT.jar
│       ├── spring-boot-jarmode-tools-3.3.0-SNAPSHOT.jar
...
├── META-INF
│   ├── MANIFEST.MF
│   └── services
│       └── java.nio.file.spi.FileSystemProvider
└── org
    └── springframework
        └── boot
            └── loader
                ├── jar
                │   ├── ManifestInfo.class
...

No surprises here, it just worked. I could even drop my spring-boot-bindings jar in BOOT-INF/lib and it got picked up. WIth CDS, though,

java -XX:ArchiveClassesAtExit=application.jsa \ 
 -Dspring.context.exit=onRefresh org.springframework.boot.loader.launch.JarLauncher

and then

java -XX:SharedArchiveFile=application.jsa \ 
 org.springframework.boot.loader.launch.JarLauncher

it worked but... apparently very few classes got cached (I don't know the command you used to create a CDS caching report) from the training run output; and anyway, with the caching dsa file, I just went from 1.2 secs to 1.0 secs; I think the JarLauncher somehow hid the classes to CDS...

Seems like the JarLauncher way is not usable efficiently with CDS....

---

java -Djarmode=tools -jar yourjar.jar extract --launcher --layers is the same as above, but with layer support.

I obtained this layout:

tree .
.
├── application
│   ├── BOOT-INF
│   │   ├── classes
│   │   │   ├── application.properties
│   │   │   └── com
│   │   │       └── example
│   │   │           └── demo
│   │   │               └── DemoApplication.class
│   │   ├── classpath.idx
│   │   └── layers.idx
│   └── META-INF
│       ├── MANIFEST.MF
│       └── services
│           └── java.nio.file.spi.FileSystemProvider
├── dependencies
│   └── BOOT-INF
│       └── lib
...
│           ├── spring-beans-6.1.4.jar
...
├── snapshot-dependencies
│   └── BOOT-INF
│       └── lib
│           ├── spring-boot-3.3.0-SNAPSHOT.jar
│           ├── spring-boot-actuator-3.3.0-SNAPSHOT.jar
│           ├── spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar
│           ├── spring-boot-autoconfigure-3.3.0-SNAPSHOT.jar
│           └── spring-boot-jarmode-tools-3.3.0-SNAPSHOT.jar
└── spring-boot-loader
    └── org
        └── springframework
            └── boot
                └── loader
                    ├── jar
                    │   ├── ManifestInfo.class
...

This one, same as layers no launcher, I could not use it:

 cd spring-boot-loader
~/Downloads/demo/extract-launcher-layers/spring-boot-loader ❯ java org.springframework.boot.loader.launch.JarLauncher
Exception in thread "main" java.lang.IllegalStateException: No 'Start-Class' manifest entry specified in org.springframework.boot.loader.launch.JarLauncher@1540e19d

Not sure how to proceed with this one.

---

Those are the results of my experimentation. Now, as a Paketo Java Buildpack maintainer, I think that extract --layers is the most promising;

• provided the Manifest Class-Path properly matches what's on the file system
• provided the runner.jar does not include the application classes (ala @sdeleuze layout)
• probably does not need the intermediary lib folder in dependencies and snapshot-dependencies
• would allow me to drop additional libs (such as spring-cloud-bindings) in dependencies and add them to the classpath using -cp (if point 2. is implemented)

Sorry for the long post post merge; I wish I had provided the feedback before you merged, but that was so much easier for me to test using published SNAPSHOTs !

Thank you

[snicoll]

it worked but... apparently very few classes got cached

I don't understand why you seem surprised. The whole point of this issue is to provide a CDS friendly unpack structure as the extract method we had before this issue wasn't and we can't get rid of it. This is the "expected" behavior.

[mhalbritter]

@Sineaggi

is only the first extraction option (extract) CDS-friendly?

extract and extract --layers is CDS friendly. As soon as you include --launcher, CDS doesn't work anymore, as the JarLauncher is not CDS friendly.

[mhalbritter]

@anthonydahanne

Thanks for giving it a try!

java -cp "/Users/anthonyd2/Downloads/spring-cloud-bindings-2.0.2.jar:application/demo-0.0.1-SNAPSHOT.jar:dependencies/*" \ 
 -Dorg.springframework.cloud.bindings.boot.enable=true   com.example.demo.DemoApplication

This isn't CDS friendly, is it? Because you're adding the whole dependencies/ folder with a wildcard to the classpath.

If you want to add additional JARs to the classpath, I think you need to edit the manifest. But I also think @sdeleuze wanted to investigate how that's possible without editing the JAR, maybe with some clever -cp options.

[mhalbritter]

@anthonydahanne

Then, the manifest in runner.jar references the simple extract layout , for example: Class-Path: lib/spring-boot-actuator-autoconfigure-3.3.0-SNAPSHOT.jar... making the jar non usable:

The extract --layers is essentially the same as layertools extract, and is not intended to be run out of the box. Instead it's used like described in the documentation.

[mhalbritter]

@anthonydahanne

it worked but... apparently very few classes got cached

Yeah, that's expected. As soon as you include --launcher, it's using the JarLauncher, which is not CDS friendly. Only extract and extract --layers is CDS friendly.

This one, same as layers no launcher, I could not use it:

See https://github.com/spring-projects/spring-boot/issues/38276#issuecomment-1985185342.

[mhalbritter]

@anthonydahanne

Thanks for giving it a try!

java -cp "/Users/anthonyd2/Downloads/spring-cloud-bindings-2.0.2.jar:application/demo-0.0.1-SNAPSHOT.jar:dependencies/*" \ 
 -Dorg.springframework.cloud.bindings.boot.enable=true   com.example.demo.DemoApplication

This isn't CDS friendly, is it? Because you're adding the whole dependencies/ folder with a wildcard to the classpath.

If you want to add additional JARs to the classpath, I think you need to edit the manifest. But I also think @sdeleuze wanted to investigate how that's possible without editing the JAR, maybe with some clever -cp options.

Soo, i played around with it. If you want to add additional libraries, this works with:

java -cp runner.jar:ext/lib.jar com.example.cdstest.CdsTestApplication

where com.example.cdstest.CdsTestApplication is the Main-Class from the manifest, and ext/lib.jar is the library you want to add to the classpath.

Java then takes the Class-Path from the manifest of the runner.jar and adds ext/lib.jar.

This is CDS friendly, too:

--------------------------------------------------------------------------
Class Loading Report:
      6326 classes and JDK proxies loaded
      6074 (96.02%) from cache
       252 ( 3.98%) from classpath

Categories:
   Lambdas  738 (11.67%): 97.83% from cache
   Proxies   62 ( 0.98%): 6.45% from cache
   Classes 5527 (87.37%): 96.78% from cache

Top 10 locations from classpath:
       143 __JVM_LookupDefineClass__
        58 __dynamic_proxy__
        24 __ClassDefiner__
         4 java.util.Comparator
         4 org.springframework.boot.autoconfigure.task.TaskExecutorConfigurations
         2 lib/spring-boot-3.3.0-SNAPSHOT.jar
         2 java.util.stream.StreamSpliterators
         2 lib/spring-boot-autoconfigure-3.3.0-SNAPSHOT.jar
         2 jrt:/java.base
         2 org.springframework.boot.autoconfigure.task.TaskSchedulingConfigurations

Top 10 packages:
      2812 org.springframework (99.36% from cache)
       627 java.util (99.04% from cache)
       580 java.lang (75.34% from cache)
       511 com.fasterxml (100.00% from cache)
       302 org.apache (99.67% from cache)
       253 jdk.internal (89.72% from cache)
       176 ch.qos (100.00% from cache)
        98 java.time (100.00% from cache)
        92 sun.security (100.00% from cache)
        91 sun.util (100.00% from cache)
--------------------------------------------------------------------------

[anthonydahanne]

Hello all! Thanks for your answers, they clear things up! All in all: JarLauncher: not CDS friendly ; --layers : designed for consumption from Dockerfiles (or at least, not usable without copying each folder in specific places)

So there remains the simple extract option, and yes, with this option and:

java -cp runner.jar:/Users/anthony/Downloads/spring-cloud-bindings-2.0.2.jar com.example.demo.DemoApplication

It works fine (CDS'ifying worked too); not sure why I could not make it work during my testing 🤷

[mhalbritter]

After a bit of discussion we want to change the names of the produced artifacts a bit.

The plan is:

If you don't specify --destination, it will default to a directory named after the input JAR instead of the current working directory. Additionally, the runner.jar will be named like the input JAR.

So given this:

java -Djarmode=tools -jar app-0.0.1-SNAPSHOT.jar extract

you'll get ./app-0.0.1-SNAPSHOT/app-0.0.1-SNAPSHOT.jar (which can be executed with java -jar) and the ./app-0.0.1-SNAPSHOT/libs/ directory.

If you specify a directory with --destination, it will create the "runner" with the name like the input jar.

So given this:

java -Djarmode=tools -jar app-0.0.1-SNAPSHOT.jar extract --destination output

you'll get ./output/app-0.0.1-SNAPSHOT.jar (which can be executed with java -jar) and the ./output/libs/ directory.

[anthonydahanne]

and the ./app-0.0.1-SNAPSHOT.jar/libs/ directory

You meant:

and the ./app-0.0.1-SNAPSHOT/libs/ directory

Right? Otherwise, changes dully noted. Thanks!

[mhalbritter]

Yeah, you're right, that's a typo. I've edited the original post.

[wyhasany]

Hello, I'm strugging with preparing multi stage dockerfile. It works fine when I don't use layers:

FROM --platform=linux/arm64/v8 bellsoft/liberica-openjre-alpine-musl:21.0.3-10-cds AS builder
WORKDIR application
COPY build/libs/*.jar app.jar
RUN java -Djarmode=tools -jar app.jar extract --layers --destination out --application-filename app.jar
RUN cp -R ./out/dependencies/* ./
#When no loader and snapshot dependencies
#RUN cp -R ./out/spring-boot-loader/* ./
#RUN cp -R ./out/snapshot-dependencies/* ./
RUN cp -R ./out/application/* ./
RUN rm -rf ./out
RUN java -Xshare:dump #-XX:+UseZGC
RUN java \
    -Dspring.context.exit=onRefresh \
    -XX:ArchiveClassesAtExit=/application/app-cds.jsa \
    -Xshare:on \
    -jar app.jar

FROM --platform=linux/arm64/v8 bellsoft/liberica-openjre-alpine-musl:21.0.3-10-cds
WORKDIR application
COPY --from=builder /usr/lib/jvm/jre/lib/server/classes.jsa /usr/lib/jvm/jre/lib/server/classes.jsa
COPY --from=builder application .
ENTRYPOINT ["java", "-XX:SharedArchiveFile=app-cds.jsa", "-Xshare:on", "-jar", "app.jar"]

However if I would like to pass layers separately:

FROM --platform=linux/arm64/v8 bellsoft/liberica-openjre-alpine-musl:21.0.1-cds AS builder
WORKDIR application
COPY build/libs/*.jar app.jar
RUN java -Djarmode=tools -jar app.jar extract --layers --destination out --application-filename app.jar
RUN cp -R ./out/dependencies/* ./
#When no loader and snapshot dependencies
#RUN cp -R ./out/spring-boot-loader/* ./
#RUN cp -R ./out/snapshot-dependencies/* ./
RUN cp -R ./out/application/* ./
RUN rm -rf ./out
RUN java -Xshare:dump #-XX:+UseZGC
RUN java \
    -Dspring.context.exit=onRefresh \
    -XX:ArchiveClassesAtExit=/application/app-cds.jsa \
    -Xshare:on \
    -jar app.jar

FROM --platform=linux/arm64/v8 bellsoft/liberica-openjre-alpine-musl:21.0.1-cds
WORKDIR application
COPY --from=builder /usr/lib/jvm/jre/lib/server/classes.jsa /usr/lib/jvm/jre/lib/server/classes.jsa
COPY --from=builder application/lib ./lib                            
COPY --from=builder application/app.jar ./app.jar                    
COPY --from=builder application/app-cds.jsa /application/app-cds.jsa 
ENTRYPOINT ["java", "-XX:SharedArchiveFile=app-cds.jsa", "-Xshare:on", "-jar", "app.jar"]

I can see following error:

[0.049s][warning][cds] A jar file is not the one used while building the shared archive file: lib/spring-webmvc-6.1.6.jar
2024-05-13T12:51:53.072884000Z [0.049s][warning][cds] A jar file is not the one used while building the shared archive file: lib/spring-webmvc-6.1.6.jar
2024-05-13T12:51:53.072885625Z [0.049s][warning][cds] lib/spring-webmvc-6.1.6.jar timestamp has changed.
2024-05-13T12:51:53.073443041Z [0.050s][warning][cds,dynamic] Unable to use shared archive. The top archive failed to load: app-cds.jsa
2024-05-13T12:51:53.073568750Z [0.050s][error  ][cds        ] An error has occurred while processing the shared archive file.
2024-05-13T12:51:53.073572458Z [0.050s][error  ][cds        ] Unable to map shared spaces
2024-05-13T12:51:53.073573500Z Error occurred during initialization of VM
2024-05-13T12:51:53.073574375Z Unable to use shared archive.

it seems that coping changes timestamps, can I avoid it somehow? The very same happens with Liberica JVM 21.0.3-10-cds

[wilkinsona]

@wyhasany please see https://github.com/moby/moby/issues/17175. I don't think there's anything we can do about it in Spring Boot, unfortunately.

[wyhasany]

@wilkinsona I think it is worth mentioning in docs, because using layered images could be confusing for users. I'm going to check behavior when the files are prepared outside of multistage docker. The other option is jib.

[anthonydahanne]

The other option is jib.

Have you considered Paketo Buildpacks, the default Spring Boot container solution that supports CDS?

[wyhasany]

Unfortunately buildpacks images are bigger from the ones manually created.

[anthonydahanne]

well, I invite you to join the Paketo community: if you're interested in "small" images you'll learn about tiny builders, how to rebase on small run images, and if you're still unsatisfied, you'll learn about custom stack you could build with Chainguard, etc.

See you there 👋

[wyhasany]

@wilkinsona Hmm... it seems to be not specific to the Docker:

$ java -Djarmode=tools -jar spring-cold-startup-0.0.1-SNAPSHOT.jar extract --destination out --application-filename app.jar

$ cd out

$ tree -D
[May 13 14:16]  .
├── [May 13 14:16]  application
│   └── [May 13 14:16]  app.jar
├── [May 13 14:16]  dependencies
│   └── [May 13 14:16]  lib
│       ├── [Apr 23 15:16]  byte-buddy-1.14.13.jar
│       ├── [Mar 19 13:53]  jackson-annotations-2.17.0.jar
│       ├── [Mar 19 13:53]  jackson-core-2.17.0.jar
│       ├── [Mar 19 13:53]  jackson-databind-2.17.0.jar
│       ├── [Apr 26 18:25]  jackson-datatype-jdk8-2.17.0.jar
│       ├── [Apr 26 18:25]  jackson-datatype-jsr310-2.17.0.jar
│       ├── [Apr 26 18:25]  jackson-module-parameter-names-2.17.0.jar
│       ├── [May 13 11:34]  jakarta.annotation-api-2.1.1.jar
│       ├── [Apr 23 15:16]  jul-to-slf4j-2.0.13.jar
│       ├── [Apr 26 18:25]  log4j-api-2.23.1.jar
│       ├── [Apr 26 18:25]  log4j-to-slf4j-2.23.1.jar
│       ├── [Apr 23 10:25]  logback-classic-1.5.6.jar
│       ├── [Apr 23 10:25]  logback-core-1.5.6.jar
│       ├── [Apr 26 18:25]  micrometer-commons-1.13.0-RC1.jar
│       ├── [Apr 26 18:25]  micrometer-observation-1.13.0-RC1.jar
│       ├── [Apr 23 10:25]  slf4j-api-2.0.13.jar
│       ├── [May 13 11:34]  snakeyaml-2.2.jar
│       ├── [Apr 23 15:16]  spring-aop-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-beans-6.1.6.jar
│       ├── [Apr 26 18:25]  spring-boot-3.3.0-RC1.jar
│       ├── [Apr 26 18:25]  spring-boot-autoconfigure-3.3.0-RC1.jar
│       ├── [Apr 23 15:16]  spring-context-6.1.6.jar
│       ├── [Apr 16 12:10]  spring-core-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-expression-6.1.6.jar
│       ├── [Apr 16 12:10]  spring-jcl-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-web-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-webmvc-6.1.6.jar
│       ├── [Apr 23 15:16]  tomcat-embed-core-10.1.20.jar
│       ├── [Apr 23 15:16]  tomcat-embed-el-10.1.20.jar
│       └── [Apr 23 15:16]  tomcat-embed-websocket-10.1.20.jar
├── [May 13 14:16]  snapshot-dependencies
└── [May 13 14:16]  spring-boot-loader

$ cp -R application/ .

$ cp -R dependencies/ .

$ ls -la
.rw-r--r-- 3.4k rowickim 15 May 16:09 app.jar
drwxr-xr-x    - rowickim 13 May 14:16 application
drwxr-xr-x    - rowickim 13 May 14:16 dependencies         
drwxr-xr-x    - rowickim 15 May 16:10 lib            #date was changed after the copy, the very same goes for files underneath

$ ls -la lib
.rw-r--r-- 4.2M rowickim 15 May 16:10 byte-buddy-1.14.13.jar
.rw-r--r--  78k rowickim 15 May 16:10 jackson-annotations-2.17.0.jar
.rw-r--r-- 582k rowickim 15 May 16:10 jackson-core-2.17.0.jar
.rw-r--r-- 1.6M rowickim 15 May 16:10 jackson-databind-2.17.0.jar
.rw-r--r--  36k rowickim 15 May 16:10 jackson-datatype-jdk8-2.17.0.jar
.rw-r--r-- 132k rowickim 15 May 16:10 jackson-datatype-jsr310-2.17.0.jar
.rw-r--r--  10k rowickim 15 May 16:10 jackson-module-parameter-names-2.17.0.jar
.rw-r--r--  26k rowickim 15 May 16:10 jakarta.annotation-api-2.1.1.jar
.rw-r--r-- 6.3k rowickim 15 May 16:10 jul-to-slf4j-2.0.13.jar
.rw-r--r-- 343k rowickim 15 May 16:10 log4j-api-2.23.1.jar
#...

$ java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app-cds.jsa -Xshare:on -jar app.jar

$ rm -rf app.jar lib

$ cp -R application/ .

$ cp -R dependencies/ .

$ java -XX:SharedArchiveFile=app-cds.jsa -Xshare:on -jar app.jar
[0.123s][warning][cds] A jar file is not the one used while building the shared archive file: app.jar
[0.123s][warning][cds] A jar file is not the one used while building the shared archive file: app.jar
[0.123s][warning][cds] app.jar timestamp has changed.
[0.123s][warning][cds,dynamic] Unable to use shared archive. The top archive failed to load: app-cds.jsa
[0.124s][error  ][cds        ] An error has occurred while processing the shared archive file.
[0.124s][error  ][cds        ] Unable to map shared spaces
Error occurred during initialization of VM
Unable to use shared archive.

To fix that we can just preserve metadata of files during copy: https://man7.org/linux/man-pages/man1/cp.1.html

$ rm -rf app.jar lib

$ cp -pR application/ .

$ cp -pR dependencies/ .

[May 13 14:16]  .
├── [May 15 16:14]  app-cds.jsa
├── [May 13 14:16]  app.jar
├── [May 13 14:16]  application
│   └── [May 13 14:16]  app.jar
├── [May 13 14:16]  dependencies
│   └── [May 13 14:16]  lib
│       ├── [Apr 23 15:16]  byte-buddy-1.14.13.jar
│       ├── [Mar 19 13:53]  jackson-annotations-2.17.0.jar
│       ├── [Mar 19 13:53]  jackson-core-2.17.0.jar
│       ├── [Mar 19 13:53]  jackson-databind-2.17.0.jar
│       ├── [Apr 26 18:25]  jackson-datatype-jdk8-2.17.0.jar
│       ├── [Apr 26 18:25]  jackson-datatype-jsr310-2.17.0.jar
│       ├── [Apr 26 18:25]  jackson-module-parameter-names-2.17.0.jar
│       ├── [May 13 11:34]  jakarta.annotation-api-2.1.1.jar
│       ├── [Apr 23 15:16]  jul-to-slf4j-2.0.13.jar
│       ├── [Apr 26 18:25]  log4j-api-2.23.1.jar
│       ├── [Apr 26 18:25]  log4j-to-slf4j-2.23.1.jar
│       ├── [Apr 23 10:25]  logback-classic-1.5.6.jar
│       ├── [Apr 23 10:25]  logback-core-1.5.6.jar
│       ├── [Apr 26 18:25]  micrometer-commons-1.13.0-RC1.jar
│       ├── [Apr 26 18:25]  micrometer-observation-1.13.0-RC1.jar
│       ├── [Apr 23 10:25]  slf4j-api-2.0.13.jar
│       ├── [May 13 11:34]  snakeyaml-2.2.jar
│       ├── [Apr 23 15:16]  spring-aop-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-beans-6.1.6.jar
│       ├── [Apr 26 18:25]  spring-boot-3.3.0-RC1.jar
│       ├── [Apr 26 18:25]  spring-boot-autoconfigure-3.3.0-RC1.jar
│       ├── [Apr 23 15:16]  spring-context-6.1.6.jar
│       ├── [Apr 16 12:10]  spring-core-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-expression-6.1.6.jar
│       ├── [Apr 16 12:10]  spring-jcl-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-web-6.1.6.jar
│       ├── [Apr 23 15:16]  spring-webmvc-6.1.6.jar
│       ├── [Apr 23 15:16]  tomcat-embed-core-10.1.20.jar
│       ├── [Apr 23 15:16]  tomcat-embed-el-10.1.20.jar
│       └── [Apr 23 15:16]  tomcat-embed-websocket-10.1.20.jar
├── [May 13 14:16]  lib
│   ├── [Apr 23 15:16]  byte-buddy-1.14.13.jar
│   ├── [Mar 19 13:53]  jackson-annotations-2.17.0.jar
│   ├── [Mar 19 13:53]  jackson-core-2.17.0.jar
│   ├── [Mar 19 13:53]  jackson-databind-2.17.0.jar
│   ├── [Apr 26 18:25]  jackson-datatype-jdk8-2.17.0.jar
│   ├── [Apr 26 18:25]  jackson-datatype-jsr310-2.17.0.jar
│   ├── [Apr 26 18:25]  jackson-module-parameter-names-2.17.0.jar
│   ├── [May 13 11:34]  jakarta.annotation-api-2.1.1.jar
│   ├── [Apr 23 15:16]  jul-to-slf4j-2.0.13.jar
│   ├── [Apr 26 18:25]  log4j-api-2.23.1.jar
│   ├── [Apr 26 18:25]  log4j-to-slf4j-2.23.1.jar
│   ├── [Apr 23 10:25]  logback-classic-1.5.6.jar
│   ├── [Apr 23 10:25]  logback-core-1.5.6.jar
│   ├── [Apr 26 18:25]  micrometer-commons-1.13.0-RC1.jar
│   ├── [Apr 26 18:25]  micrometer-observation-1.13.0-RC1.jar
│   ├── [Apr 23 10:25]  slf4j-api-2.0.13.jar
│   ├── [May 13 11:34]  snakeyaml-2.2.jar
│   ├── [Apr 23 15:16]  spring-aop-6.1.6.jar
│   ├── [Apr 23 15:16]  spring-beans-6.1.6.jar
│   ├── [Apr 26 18:25]  spring-boot-3.3.0-RC1.jar
│   ├── [Apr 26 18:25]  spring-boot-autoconfigure-3.3.0-RC1.jar
│   ├── [Apr 23 15:16]  spring-context-6.1.6.jar
│   ├── [Apr 16 12:10]  spring-core-6.1.6.jar
│   ├── [Apr 23 15:16]  spring-expression-6.1.6.jar
│   ├── [Apr 16 12:10]  spring-jcl-6.1.6.jar
│   ├── [Apr 23 15:16]  spring-web-6.1.6.jar
│   ├── [Apr 23 15:16]  spring-webmvc-6.1.6.jar
│   ├── [Apr 23 15:16]  tomcat-embed-core-10.1.20.jar
│   ├── [Apr 23 15:16]  tomcat-embed-el-10.1.20.jar
│   └── [Apr 23 15:16]  tomcat-embed-websocket-10.1.20.jar
├── [May 13 14:16]  snapshot-dependencies
└── [May 13 14:16]  spring-boot-loader

$ rm app-cds.jsa

$ java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app-cds.jsa -Xshare:on -jar app.jar 

$ rm -rf app.jar lib

$ cp -pR application/ .

$ cp -pR dependencies/ .

$ java -XX:SharedArchiveFile=app-cds.jsa -Xshare:on -jar app.jar 
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::            (v3.3.0-RC1)

2024-05-15T16:27:32.998+02:00  INFO 5878 --- [           main] t.p.cold.SpringColdStartupApplication    : Starting SpringColdStartupApplication v0.0.1-SNAPSHOT using Java 21.0.2 with PID 5878

Let's try with Dockerfile:

FROM --platform=linux/arm64/v8 bellsoft/liberica-openjre-alpine-musl:21.0.3-10-cds AS builder
WORKDIR application
COPY build/libs/*.jar app.jar
RUN java -Djarmode=tools -jar app.jar extract --layers --destination out --application-filename app.jar
RUN cp -pR ./out/dependencies/* ./
#RUN cp -pR ./out/spring-boot-loader/* ./
#RUN cp -pR ./out/snapshot-dependencies/* ./
RUN cp -pR ./out/application/* ./
RUN java -Xshare:dump #-XX:+UseZGC
RUN java \
    -Dspring.context.exit=onRefresh \
    -XX:ArchiveClassesAtExit=/application/app-cds.jsa \
    -Xshare:on \
    -jar app.jar

FROM --platform=linux/arm64/v8 bellsoft/liberica-openjre-alpine-musl:21.0.3-10-cds
WORKDIR application

COPY --from=builder /usr/lib/jvm/jre/lib/server/classes.jsa /usr/lib/jvm/jre/lib/server/classes.jsa
COPY --from=builder application/out/dependencies/ ./
COPY --from=builder application/out/spring-boot-loader/ ./
COPY --from=builder application/out/snapshot-dependencies/ ./
COPY --from=builder application/out/application/ ./

COPY --from=builder application/app-cds.jsa /application/app-cds.jsa
ENTRYPOINT ["java", "-XX:SharedArchiveFile=/usr/lib/jvm/jre/lib/server/classes.jsa:app-cds.jsa", "-Xshare:on", "-jar", "app.jar"]

it works as expected using correct jars. Based on that this time the issue is not with the docker but missing -p preserve flag on coping files for generation of class data sharing archive. I think it is worth of mentioning in documentation to avoid complaining users.

[wilkinsona]

Very interesting. Thank you, @wyhasany. Let me discuss it with the team to see if we think it would be best documented in Boot's docs, Framework's docs, or perhaps even in both.