search

paketo-buildpacks / libjvm

A library and helper applications that form the basis for building the different Paketo-style JVM-providing buildpacks
Apache License 2.0
19 stars 20 forks source link

BP_JVM_VERSION should have priority over derivation from MANIFEST.MF #350

Closed OLibutzki closed 5 months ago

OLibutzki commented 7 months ago

Although a BP_JVM_VERSION is passed explicitly, the Java version is derived from Build-Jdk-Spec in the MANIFEST.MF.

See https://github.com/orgs/paketo-buildpacks/discussions/237

Describe the Enhancement

Deriving the Java version from the MANIFEST.MF makes sense, but in case there is an explicit BP_JVM_VERSION given that should have precedence.

Possible Solution

The possible solution is mentioned in the enhancement description.

Motivation

The motiviation is explained in the linked discussion.

dmikusa commented 7 months ago

The code here should prefer BP_JVM_VERSION.

Can you provide some more details about your environment where the issue is happening so we can recreate?

  1. Are you building with Spring Boot build tools? or pack? If the former, can you try with pack and see if you get the same result? Please note with pack, you need to use the --path option to have similar behavior to Spring Boot Build tools.
  2. What is the build command you're running?
  3. Can you do a fresh build and provide that build output? The output from the discussion has cached the JVM layer, so we can't see what libjvm did and what env variables it saw.
  4. How/where did you set BP_JVM_VERSION?

Thanks

OLibutzki commented 7 months ago

Are you building with Spring Boot build tools?

Yes. To be honest I am not familiar with using pack, but I created a sample project with a matrix build workflow to demonstrate the behaviour, see below.

What is the build command you're running?

mvn spring-boot:build-image

Can you do a fresh build and provide that build output? The output from the discussion has cached the JVM layer, so we can't see what libjvm did and what env variables it saw.

The sample project schould help here.

How/where did you set BP_JVM_VERSION

It is set by the Spring Boot Maven Plugin automatically.

The sample project is located here: https://github.com/OLibutzki/paketo-buildpacks-libjvm-350 Here is the matrix build which builds the project with Java 17 and Java 21: https://github.com/OLibutzki/paketo-buildpacks-libjvm-350/actions/runs/7258209079

In both builds BP_JVM_VERSION is set to 17 by Spring Boot as it's a Java 17 application as you can see in the pom.xml.

In the Java 17 build one can find Using Java version 17 extracted from MANIFEST.MF and in the Java 21 one can find Using Java version 21 extracted from MANIFEST.MF.

My expectation would have been that Java 17 is chosen, regardless of the Java version I use for building the project.

I hope that helps you.

dmikusa commented 7 months ago

It does help, thanks much!

modulo11 commented 5 months ago

Java 17 is currently used as the default Java version, I suspect that's what we can see in the build logs. Changing java.version version in pom.xml to 21 leads to this output:

[INFO]     [creator]     Paketo Buildpack for BellSoft Liberica 10.5.2
[INFO]     [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
[INFO]     [creator]       Build Configuration:
[INFO]     [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
[INFO]     [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
[INFO]     [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
[INFO]     [creator]         $BP_JVM_VERSION              17                                                           the Java version
[INFO]     [creator]       Launch Configuration:
[INFO]     [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
[INFO]     [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
[INFO]     [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
[INFO]     [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
[INFO]     [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
[INFO]     [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
[INFO]     [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
[INFO]     [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
[INFO]     [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
[INFO]     [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
[INFO]     [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
[INFO]     [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
[INFO]     [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
[INFO]     [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
[INFO]     [creator]         Using Java version 21 extracted from MANIFEST.MF
[INFO]     [creator]       BellSoft Liberica JRE 21.0.2: Reusing cached layer
[INFO]     [creator]       Launch Helper: Reusing cached layer
[INFO]     [creator]       Java Security Properties: Reusing cached layer

So, BP_JVM_VERSION is not read from pom.xml and not explicitly passed to the build. I don't know if that behavior is promised by the Maven plugin, but it looks like that's more an issue on their side.

OLibutzki commented 5 months ago

So, BP_JVM_VERSION is not read from pom.xml and not explicitly passed to the build. I don't know if that behavior is promised by the Maven plugin, but it looks like that's more an issue on their side.

@modulo11 I can confirm that BP_JVM_VERSION is 17 although java.version is set to 21 in pom.xml.

In my opinion that's a second issue. That issue needs to be addressed to the Spring Boot team.

Anyway, this issue is valid, too as Buildpacks uses the Java version used for the build instead of the version which is used to compile the application.

modulo11 commented 5 months ago

The spring-boot-maven-plugin acts as a platform (@dmikusa please correct me if I'm wrong) which makes certain choices.

For example if I specify export BP_JVM_VERSION=17 for a Java 21 project, mvn spring-boot:build-image just ignores it and takes JRE 21 read from the manifest.

Using this explicit configuration for the plugin:

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <image>
                    <env>
                        <BP_JVM_VERSION>17</BP_JVM_VERSION>
                    </env>
                </image>
            </configuration>
        </plugin>
    </plugins>
</build>

in a Java 21 project shows, that the correct choices were made:

[INFO]     [creator]     Paketo Buildpack for BellSoft Liberica 10.5.2
[INFO]     [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
[INFO]     [creator]       Build Configuration:
[INFO]     [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
[INFO]     [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
[INFO]     [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
[INFO]     [creator]         $BP_JVM_VERSION              17                                                           the Java version
[INFO]     [creator]       Launch Configuration:
[INFO]     [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
[INFO]     [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
[INFO]     [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
[INFO]     [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
[INFO]     [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
[INFO]     [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
[INFO]     [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
[INFO]     [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
[INFO]     [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
[INFO]     [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
[INFO]     [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
[INFO]     [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
[INFO]     [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
[INFO]     [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
[INFO]     [creator]         Using Java version 17 from BP_JVM_VERSION

The current priorities are (lowest to highest):

  1. Take the default for BP_JVM_VERSION (this is your scenario at the moment)
  2. Take the version from manifest
  3. Take explicitly configured BP_JVM_VERSION
scottfrederick commented 5 months ago

Versions of Spring Boot from 2.3 through 2.7 set the BP_JVM_VERSION when invoking buildpacks in order to align the buildpack Java version with the version configured in the build. In Spring Boot 3.0 and later, BP_JVM_VERSION is no longer set automatically. Instead, we ensure that Build-Jdk-Spec is set in the jar manifest (Maven does this by default, the Spring Boot Gradle plugin was changed to match Maven's behavior).

So, BP_JVM_VERSION is not read from pom.xml and not explicitly passed to the build. I don't know if that behavior is promised by the Maven plugin, but it looks like that's more an issue on their side.

For example if I specify export BP_JVM_VERSION=17 for a Java 21 project, mvn spring-boot:build-image just ignores it

Nothing in the Spring Boot plugins looks for BP_JVM_VERSION as an operating system environment variable. If you want to use this environment variable to configure the buildpacks when they run, you need to set it in the Spring Boot Maven plugin's build configuration as shown in the documentation.

The spring-boot-maven-plugin acts as a platform (@dmikusa please correct me if I'm wrong) which makes certain choices.

Yes, the Spring Boot plugins do act as CNB platforms. The design goal is that the plugins don't make choices as much as they provide information and hints in the artifacts provided to buildpacks so that buildpacks can make the choices. Initially we provided hints by setting environment variables in the buildpack invocation, but we have since moved more toward using manifest entries or files contained in the archive as with this JVM version example. The Boot and Paketo teams agreed that putting hints in the jar file is a better approach, as it is more portable across buildpack providers, makes for a more consistent experience with pack and other CNB platforms, and made it easier for users to override the default behavior using the environment variable.

OLibutzki commented 5 months ago

I got your points, but I have a concrete question/scenario.

I have a Spring Boot 3 application which is built against Java 17 (java.version is set to 17 in the pom.xml). I build ths application using Java 21 which is fine as newer Java versions are able to build applications which are developed against older Java versions.

Building with Java 21 results in a Build-Jdk-Spec: 21 entry which is fine as the application is built by Java 21. Nevertheless, it's still a Java 17 application and in my opinion the reasonable default should be to run the application with the Java version it has been developed against. Don't you agree on this?

Anyway, if it's not the default to use the Java version the application is compiled against as runtime jdk: How can I explicitly configure to use Java 17 as runtime jdk although I ran the build process with Java 21?

dmikusa commented 5 months ago

Anyway, if it's not the default to use the Java version the application is compiled against as runtime jdk: Who can I explicitly configure to use Java 17 as runtime jdk although I ran the build process with Java 21?

If you set BP_JVM_VERSION it will override this. You have to set it so that it's passed through to the build environment though, not in your local shell.

From @modulo11's example above:

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <image>
                    <env>
                        <BP_JVM_VERSION>17</BP_JVM_VERSION>
                    </env>
                </image>
            </configuration>
        </plugin>
    </plugins>
</build>

That should override to Java 17.

scottfrederick commented 5 months ago

Building with Java 21 results in a Build-Jdk-Spec: 21 entry

Build-Jdk-Spec is set by the Maven Archiver plugin. Spring Boot does not do anything to override Maven's default behavior.

Who can I explicitly configure to use Java 17 as runtime jdk although I ran the build process with Java 21?

The typical way of doing this would be to set BP_JVM_VERSION, but this must be done in the Spring Boot plugin configuration as in the example linked above, not in an operating system environment variable provided to Maven. I just tested this myself and it works as documented for me, with this output from the buildpack: 

[INFO]     [creator]     Paketo Buildpack for BellSoft Liberica 10.5.2
[INFO]     [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
[INFO]     [creator]       Build Configuration:
[INFO]     [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
[INFO]     [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
[INFO]     [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
[INFO]     [creator]         $BP_JVM_VERSION              17                                                           the Java version
[INFO]     [creator]       Launch Configuration:
[INFO]     [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
[INFO]     [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
[INFO]     [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
[INFO]     [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
[INFO]     [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
[INFO]     [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
[INFO]     [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
[INFO]     [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
[INFO]     [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
[INFO]     [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
[INFO]     [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
[INFO]     [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
[INFO]     [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
[INFO]     [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
[INFO]     [creator]         Using Java version 17 from BP_JVM_VERSION
[INFO]     [creator]       BellSoft Liberica JRE 17.0.10: Contributing to layer
[INFO]     [creator]         Downloading from https://github.com/bell-sw/Liberica/releases/download/17.0.10+13/bellsoft-jre17.0.10+13-linux-amd64.tar.gz

If this isn't working for you, can you provide a minimal sample that shows what you're trying to do?

scottfrederick commented 5 months ago

Note that you can also set the value of BP_JVM_VERSION using a Maven property if that helps keep your build cleaner: 

    <properties>
        <java.version>17</java.version>
    </properties>

...

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <env>
                            <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
                        </env>
                    </image>
                    ...
                </configuration>
            </plugin>
OLibutzki commented 5 months ago

@scottfrederick thanks! I can confirm that it works, if I set BP_JVM_VERSION explicitly.

Now I understand what confused me. That's the log without setting BP_JVM_VERSION explicitly:

[INFO]     [creator]     Paketo Buildpack for BellSoft Liberica 10.5.2
[INFO]     [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
[...]
[INFO]     [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
[INFO]     [creator]         $BP_JVM_VERSION              17                                                           the Java version
[...]
[INFO]     [creator]         Using Java version 21 extracted from MANIFEST.MF

That the log with setting BP_JVM_VERSION to 17 explicitly.

[INFO]     [creator]     Paketo Buildpack for BellSoft Liberica 10.5.2
[INFO]     [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
[...]
[INFO]     [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
[INFO]     [creator]         $BP_JVM_VERSION              17                                                           the Java version
[...]
[INFO]     [creator]         Using Java version 17 from BP_JVM_VERSION

In both cases $BP_JVM_VERSION is logged as 17, but only if I set it explicitly the variable is used to determine the runtime jdk. That's confusing, isn't it?

dmikusa commented 5 months ago

I can see what you're saying. When it's not set, it shows the default version which is 17. We try to clarify how the version was selected below the configuration settings.

You'll see if it's picked due to the env variable:

[INFO] [creator] Using Java version 17 from BP_JVM_VERSION

You'll see if it's picked due to the manifest setting:

[INFO] [creator] Using Java version 21 extracted from MANIFEST.MF

I wouldn't be opposed to trying to differentiate between default and user-specified settings in the config table. I'm not sure off hand how to do that. The table is kind of big and unwieldy already. Adding columns makes that worse. Maybe an asterisk next to default values or something. 🤔

I'll open a new issue for that and close out this one. Let me know if you have thoughts on how we could display that over on the new issue. Thanks

OLibutzki commented 5 months ago

I'll open a new issue for that and close out this one. Let me know if you have thoughts on how we could display that over on the new issue. Thanks

I'm fine with that idea...

@scottfrederick I still think that it would make sense if this would be the default behaviour of the spring-boot-maven-plugin. Don't you think so?

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <env>
                            <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
                        </env>
                    </image>
                    ...
                </configuration>
            </plugin>
scottfrederick commented 5 months ago

I still think that it would make sense if this would be the default behaviour of the spring-boot-maven-plugin. Don't you think so?

That's the path we started down with CNB integration in the Spring Boot plugins, but we've moved away from that for a few reasons (all of which also apply to other configuration like BP_NATIVE_IMAGE that Spring Boot users might care about).

First, Spring Boot acts as a CNB platform interacting with the CNB lifecycle. Boot defaults to using Paketo buildpacks, and works closely with the Paketo team, but tries to keep the code agnostic of buildpack providers. Moving away from Paketo-specific env vars toward manifest entries and files gives other buildpack providers the same opportunity to make decisions as the Paketo team has.

Second, and IMO more importantly, the current arrangement makes the experience with mvn spring-boot:build-image consistent with pack and other platforms. Purely relying on manifest entries and files means that all platforms get the exact same input. If Boot were setting BP_JVM_VERSION as suggested then you'd get different results from Spring Boot than from pack or any other platform.

OLibutzki commented 5 months ago

@scottfrederick thanks a lot for giving this detailed explanation. Highly appreciated!