search

oracle / graal

GraalVM compiles Java applications into native executables that start instantly, scale fast, and use fewer compute resources 🚀
https://www.graalvm.org
Other
19.99k stars 1.6k forks source link

[GR-49403] Native Image: GraalVM 21+35.1 on macos-x64 hangs during inlining stage #7523

Open mmoayyed opened 9 months ago

mmoayyed commented 9 months ago

Describe the issue

GraalVM 21+35.1 on macos-x64 hangs indefinitely when building native images. This issue happens both during local development and in GitHub Actions.

Sample build run where the native image build process times out: https://github.com/mmoayyed/cas/actions/runs/6366202699/job/17283751171

The timeout happens after 90 minutes.

Steps to reproduce the issue

Then: 

./gradlew :webapp:cas-server-webapp-native:build :webapp:cas-server-webapp-native:nativeCompile \
    -x check -x test -x javadoc --configure-on-demand --no-configuration-cache \
    -DskipNestedConfigMetadataGen=true --no-daemon

Describe GraalVM and your environment:

GRAALVM: /Users/runner/hostedtoolcache/graalvm-jdk-21_macos-x64_bin/21.0.0/x64/graalvm-jdk-21+35.1/Contents/Home JDK: 21 Corretto OS: MacOS 12 (Github Actions) x64 Gradle: 8.4 RC3

More details

Output from the native image build process. The hang up appears to be during the stage-inlining phase.

2023-09-30T23:13:33.9790950Z > Task :webapp:cas-server-webapp-native:nativeCompile
2023-09-30T23:13:33.9792240Z [native-image-plugin] GraalVM Toolchain detection is disabled
2023-09-30T23:13:33.9793470Z [native-image-plugin] GraalVM location read from environment variable: GRAALVM_HOME
2023-09-30T23:13:33.9796080Z [native-image-plugin] Native Image executable path: /Users/runner/hostedtoolcache/graalvm-jdk-21_macos-x64_bin/21.0.0/x64/graalvm-jdk-21+35.1/Contents/Home/lib/svm/bin/native-image
2023-09-30T23:13:33.9797870Z Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9799710Z Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/tomcat-reflection.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9801570Z Warning: The option '-H:+BuildReport' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9803180Z Warning: The option '-H:+InlineBeforeAnalysis' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9804990Z Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/tomcat-resource.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9808620Z Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/tomcat-reflection.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9810890Z Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/tomcat-resource.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9812630Z Warning: The option '-H:+TraceSecurityServices' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9814460Z Warning: The option '-H:+BuildOutputColorful' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
2023-09-30T23:13:33.9816450Z Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
2023-09-30T23:13:34.1503780Z Warning: Option 'BuildOutputColorful' is deprecated and might be removed in a future release: Please use '--color' instead.. Please refer to the GraalVM release notes.
2023-09-30T23:13:56.1570140Z ========================================================================================================================
2023-09-30T23:13:56.1584170Z ]8;;https://www.graalvm.org/native-image/\GraalVM Native Image]8;;\: Generating 'cas' (]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-imagekind\executable]8;;\)...
2023-09-30T23:13:56.1585630Z ========================================================================================================================
2023-09-30T23:13:58.1575020Z Warning: Using -g is not supported on macOS
2023-09-30T23:13:58.1577720Z Registering BouncyCastle security provider
2023-09-30T23:14:09.9496480Z Warning: Cannot register dynamic proxy for interface list: jakarta.jms.Session. Reason: Class jakarta.jms.Session not found..
2023-09-30T23:14:09.9572150Z Warning: Cannot register dynamic proxy for interface list: jakarta.jms.MessageConsumer. Reason: Class jakarta.jms.MessageConsumer not found..
2023-09-30T23:14:09.9582010Z Warning: Cannot register dynamic proxy for interface list: jakarta.jms.MessageProducer. Reason: Class jakarta.jms.MessageProducer not found..
2023-09-30T23:14:10.3125090Z The URL protocol jar is not tested and might not work as expected.
2023-09-30T23:14:10.3131210Z Supported URL protocols enabled by default: file,resource. Supported URL protocols available on demand: http,https.
2023-09-30T23:14:10.3148030Z The URL protocol file is enabled by default. The option --enable-url-protocols=file is not needed.
2023-09-30T23:14:10.3179820Z The URL protocol resource is enabled by default. The option --enable-url-protocols=resource is not needed.
2023-09-30T23:14:13.1275160Z [1/8] ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-initializing\Initializing]8;;\...                                                                                   (38.4s @ 1.19GB)
2023-09-30T23:14:13.1277890Z  ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-java-info\Java version]8;;\: 21+35, ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-java-info\vendor version]8;;\: Oracle GraalVM 21+35.1
2023-09-30T23:14:13.1280100Z  ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-graal-compiler\Graal compiler]8;;\: optimization level: 2, target machine: native, ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#recommendation-pgo\PGO]8;;\: instrument
2023-09-30T23:14:13.1282090Z  ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-ccompiler\C compiler]8;;\: cc (apple, x86_64, 14.0.0)
2023-09-30T23:14:13.1284560Z  ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-gc\Garbage collector]8;;\: Serial GC (]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-gc-max-heap-size\max heap size]8;;\: 80% of RAM)
2023-09-30T23:14:13.1286280Z  6 ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-user-specific-features\user-specific feature(s)]8;;\:
2023-09-30T23:14:13.1287600Z  - com.oracle.svm.polyglot.groovy.GroovyIndyInterfaceFeature
2023-09-30T23:14:13.1288970Z  - com.oracle.svm.thirdparty.gson.GsonFeature
2023-09-30T23:14:13.1290160Z  - org.apereo.cas.nativex.features.BouncyCastleProviderFeature
2023-09-30T23:14:13.1291640Z  - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
2023-09-30T23:14:13.1292900Z  - org.eclipse.angus.mail.nativeimage.AngusMailFeature
2023-09-30T23:14:13.1294260Z  - org.springframework.aot.nativex.feature.PreComputeFieldFeature
2023-09-30T23:14:13.1295420Z ------------------------------------------------------------------------------------------------------------------------
2023-09-30T23:14:13.1296790Z  7 ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-experimental-options\experimental option(s)]8;;\ unlocked:
2023-09-30T23:14:13.1298160Z  - '-H:+InstallExitHandlers' (alternative API option(s): --install-exit-handlers; origin(s): command line)
2023-09-30T23:14:13.1299260Z  - '-H:+BuildReport' (origin(s): command line)
2023-09-30T23:14:13.1302840Z  - '-H:ResourceConfigurationResources' (origin(s): 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/native-image.properties' in 'file:///Users/runner/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-core/10.1.13/6909967f2ed6c323108c2cc7f20586d6f7eb6455/tomcat-embed-core-10.1.13.jar', 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/native-image.properties' in 'file:///Users/runner/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-el/10.1.13/22c8845f7528334905c582e48a3eeefab616b0a5/tomcat-embed-el-10.1.13.jar')
2023-09-30T23:14:13.1305010Z  - '-H:Optimize' (alternative API option(s): -O2; origin(s): command line)
2023-09-30T23:14:13.1306120Z  - '-H:+TraceSecurityServices' (origin(s): command line)
2023-09-30T23:14:13.1309080Z  - '-H:ReflectionConfigurationResources' (origin(s): 'META-INF/native-image/io.micrometer/micrometer-core/native-image.properties' in 'file:///Users/runner/.gradle/caches/modules-2/files-2.1/io.micrometer/micrometer-core/1.12.0-M3/c09ec03b6569074c91eef7820f983a0c3af1a1a2/micrometer-core-1.12.0-M3.jar', 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/native-image.properties' in 'file:///Users/runner/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-core/10.1.13/6909967f2ed6c323108c2cc7f20586d6f7eb6455/tomcat-embed-core-10.1.13.jar', 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/native-image.properties' in 'file:///Users/runner/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-el/10.1.13/22c8845f7528334905c582e48a3eeefab616b0a5/tomcat-embed-el-10.1.13.jar')
2023-09-30T23:14:13.1312180Z  - '-H:+BuildOutputColorful' (origin(s): command line)
2023-09-30T23:14:13.1313430Z ------------------------------------------------------------------------------------------------------------------------
2023-09-30T23:14:13.1315120Z ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-build-resources\Build resources]8;;\:
2023-09-30T23:14:13.1316520Z  - 7.11GB of memory (50.8% of 14.00GB system memory, set via '-Xmx8G')
2023-09-30T23:14:13.1317610Z  - 3 thread(s) (100.0% of 3 available processor(s), determined at start)
2023-09-30T23:14:13.3748370Z # Printing compilation-target information to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/target_info_20230930_231413.txt
2023-09-30T23:14:41.3539370Z # Printing security services automatic registration to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/security_services_20230930_231441.txt
2023-09-30T23:16:34.7882950Z SLF4J: No SLF4J providers were found.
2023-09-30T23:16:34.7984210Z SLF4J: Defaulting to no-operation (NOP) logger implementation
2023-09-30T23:16:34.8001610Z SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
2023-09-30T23:19:18.6604360Z # Printing native-library information to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/native_library_info_20230930_231918.txt
2023-09-30T23:19:19.4069210Z [2/8] ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-analysis\Performing analysis]8;;\...  [*******]                                                                (305.7s @ 4.58GB)
2023-09-30T23:19:19.4084440Z    53,626 ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability\reachable types]8;;\   (91.8% of   58,446 total)
2023-09-30T23:19:19.5104060Z    80,611 ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability\reachable fields]8;;\  (65.7% of  122,630 total)
2023-09-30T23:19:19.7183940Z   276,145 ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability\reachable methods]8;;\ (68.0% of  405,980 total)
2023-09-30T23:19:19.7288430Z    18,481 types, 2,660 fields, and 36,714 methods ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reflection-registrations\registered for reflection]8;;\
2023-09-30T23:19:19.7387380Z        66 types,    66 fields, and    56 methods ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-jni-access-registrations\registered for JNI access]8;;\
2023-09-30T23:19:19.7489690Z         5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
2023-09-30T23:20:16.0385410Z [3/8] ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-universe\Building universe]8;;\...                                                                              (56.2s @ 5.88GB)
2023-09-30T23:21:14.5117790Z [4/8] ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-parsing\Parsing methods]8;;\...      [*******]                                                                 (58.2s @ 4.95GB)
2023-09-30T23:21:20.8185550Z [5/8] ]8;;https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-inlining\Inlining methods]8;;\...     [**]                                                                       (4.2s @ 4.64GB)
2023-10-01T00:41:16.5210310Z ##[error]The action has timed out.

GraalVM Gradle plugin configuration:

graalvmNative {
    metadataRepository {
        enabled = true
        version = "0.3.4"
    }

    /*
        The toolchain detection cannot distinguish between
        GraalVM JDKs and standard JDKs without Native Image support.
     */
    toolchainDetection = false

    binaries {
        main {
            imageName = "cas"
            quickBuild = true
            debug = true
            verbose = false
            richOutput = true
            sharedLibrary = false
            mainClass = "org.apereo.cas.nativex.CasNativeWebApplication"

            configurationFileDirectories.from(file('src/graal'))

            buildArgs.add('-H:+ReportExceptionStackTraces')
            buildArgs.add('-H:+InlineBeforeAnalysis')
            buildArgs.add('-H:+TraceSecurityServices')
            buildArgs.add('-H:+BuildReport')
            buildArgs.add('-H:+AddAllCharsets')
            buildArgs.add('-H:+UnlockExperimentalVMOptions')

            buildArgs.add('-J-Xmx8G')
            buildArgs.add('-J-Xms4G')
            buildArgs.add('--native-image-info')

            buildArgs.add('-march=native')
            buildArgs.add('--pgo-instrument')
            def profileFile = rootProject.file("default.iprof")
            if (profileFile.exists()) {
                ansi.blue("Using data collected for profile-guided optimizations of AOT-compiled code")
                buildArgs.add("--pgo=${profileFile.path}")
            }
            buildArgs.add('--trace-class-initialization=true')

            buildArgs.add('--initialize-at-build-time=java.beans')
            buildArgs.add('--initialize-at-build-time=com.sun.beans')
            buildArgs.add('--initialize-at-build-time=groovyjarjarantlr4.v4.runtime')
            buildArgs.add('--initialize-at-build-time=org.codehaus.groovy')
            buildArgs.add('--initialize-at-build-time=org.apache.groovy')
            buildArgs.add('--initialize-at-build-time=groovy.lang')

            buildArgs.add('--initialize-at-run-time=com.sun.xml.ws')
            buildArgs.add('--initialize-at-run-time=groovy.grape.GrapeIvy')
            buildArgs.add("--initialize-at-run-time=org.bouncycastle.jcajce.provider.drbg.DRBG\$Default")
            buildArgs.add("--initialize-at-run-time=org.bouncycastle.jcajce.provider.drbg.DRBG\$NonceAndIV")

            buildArgs.add('--features=org.apereo.cas.nativex.features.BouncyCastleProviderFeature')

            buildArgs.add('--enable-url-protocols=jar,file,resource,http,https')
        }
    }
}
fniephaus commented 9 months ago

macOS runners come with 14GB of memory, so the first thing I'd try is to increase the max heap limit to something like 12GB. Maybe the application you're trying to compile is too big to compile with your current settings. Given you're initializing a lot at build time, it could also be a configuration problem.

mmoayyed commented 9 months ago

Thank you! That helped to move past that stage, though the build likely requires even more memory:

]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-build-resources\Build](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-build-resources%1B\Build) resources]8;;\:
 - 10.67GB of memory (76.2% of 14.00GB system memory, set via '-Xmx12G')
 - 3 thread(s) (100.0% of 3 available processor(s), determined at start)
# Printing compilation-target information to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/target_info_20231003_035334.txt
# Printing security services automatic registration to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/security_services_20231003_035341.txt
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
# Printing native-library information to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/native_library_info_20231003_035919.txt
[2/8] ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-analysis\Performing](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-analysis%1B\Performing) analysis]8;;\...  [*******]                                                                (345.4s @ 5.61GB)
   53,627 ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability\reachable](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability%1B\reachable) types]8;;\   (91.8% of   58,449 total)
   80,612 ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability\reachable](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability%1B\reachable) fields]8;;\  (65.7% of  122,605 total)
  276,150 ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability\reachable](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reachability%1B\reachable) methods]8;;\ (68.0% of  405,977 total)
   18,482 types, 2,660 fields, and 36,714 methods ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reflection-registrations\registered](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-reflection-registrations%1B\registered) for reflection]8;;\
       66 types,    66 fields, and    56 methods ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-jni-access-registrations\registered](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#glossary-jni-access-registrations%1B\registered) for JNI access]8;;\
        5 native libraries: -framework CoreServices, -framework Foundation, dl, pthread, z
[3/8] ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-universe\Building](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-universe%1B\Building) universe]8;;\...                                                                              (57.4s @ 6.96GB)
[4/8] ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-parsing\Parsing](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-parsing%1B\Parsing) methods]8;;\...      [*********]                                                               (86.4s @ 6.37GB)
[5/8] ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-inlining\Inlining](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-inlining%1B\Inlining) methods]8;;\...     [**]                                                                       (4.9s @ 4.79GB)
[6/8] ]8;;[https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-compiling\Compiling](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md#stage-compiling%1B\Compiling) methods]8;;\...    [*****************************************************]                 (2847.1s @ 9.35GB)
GC warning: 1797.3s spent in 191 GCs during the last stage, taking up 63.10% of the time.
            Please ensure more than 9.97GB of memory is available for Native Image
            to reduce GC overhead and improve image build time.

Does this really mean that '-Xmx12G' is not enough, and perhaps I can switch up to 13G? Is there anything else one might do to reduce the amount of memory required here?

A lot of what is initialized at build time has to do with Groovy, and sadly, I don't think these can be avoided:

            buildArgs.add('--initialize-at-build-time=groovyjarjarantlr4.v4.runtime')
            buildArgs.add('--initialize-at-build-time=org.codehaus.groovy')
            buildArgs.add('--initialize-at-build-time=org.apache.groovy')
            buildArgs.add('--initialize-at-build-time=groovy.lang'
fniephaus commented 9 months ago

Hi @mmoayyed, it seems --pgo-instrument introduces a significant memory overhead when building your application. We are currently looking into what is going on and how to fix this, but at the moment, I'm afraid you can't build your app with a max heap of 13GB. I was able to build it on a machine with 32GB, but that's a lot more than what GitHub's free runners provide.

While we are looking into this, maybe disable --pgo-instrument for now and see whether it unblocks the build.

mmoayyed commented 9 months ago

Thank you @fniephaus. Removing PGO instrumentation allows the build to proceed in about 14mins with 12GB:

Finished generating 'cas' in 14m 33s.
[native-image-plugin] Native Image written to: /Users/runner/work/cas/cas/webapp/cas-server-webapp-native/build/native/nativeCompile

I do however see this when the image runs:

2023-10-13 23:38:21 ERROR o.s.b.SpringApplication - Application run failed
java.lang.NoClassDefFoundError: org.graalvm.home.Version
    at org.apereo.cas.nativex.CasNativeBannerContributor.contribute(CasNativeBannerContributor.java:21)

The CasNativeBannerContributor does the following:

import org.graalvm.home.Version;
...
var version = Version.getCurrent().toString();

Is this related to the above issue somehow, or may I submit a new report?

My understanding is that GraalVM SDK must not be packaged with the app so I don't get why this would show up (and it did not before in prior releases of GraalVM for JDK 17)

mmoayyed commented 3 months ago

I have switched to Java 22 and looks like I am still seeing the same sort of error:

java version "22" 2024-03-19
Java(TM) SE Runtime Environment Oracle GraalVM 22+36.1 (build 22+36-jvmci-b02)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 22+36.1 (build 22+36-jvmci-b02, mixed mode, sharing)

This is with Gradle 8.7-rc-3:

> Task :webapp:cas-server-webapp-native:nativeCompile
[native-image-plugin] GraalVM Toolchain detection is disabled
[native-image-plugin] GraalVM location read from environment variable: JAVA_HOME
[native-image-plugin] Native Image executable path: /Library/Java/JavaVirtualMachines/graalvm/Contents/Home/lib/svm/bin/native-image
Warning: The option '-H:ServiceLoaderFeatureExcludeServices=org.hibernate.bytecode.spi.BytecodeProvider' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/tomcat-reflection.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+BuildReport' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+InlineBeforeAnalysis' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/tomcat-resource.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/tomcat-reflection.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ResourceConfigurationResources=META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/tomcat-resource.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+TraceSecurityServices' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
Warning: Option 'InlineBeforeAnalysis' is deprecated and might be removed in a future release: It no longer has any effect, and no replacement is available. Please refer to the GraalVM release notes.
========================================================================================================================
GraalVM Native Image: Generating 'cas' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
Warning: Using -g is not supported on macOS
Registering BouncyCastle security provider
The URL protocol jar is not tested and might not work as expected.
Supported URL protocols enabled by default: file,resource. Supported URL protocols available on demand: http,https.
The URL protocol file is enabled by default. The option --enable-url-protocols=file is not needed.
The URL protocol resource is enabled by default. The option --enable-url-protocols=resource is not needed.
[1/8] Initializing...                                                                                   (12.0s @ 0.74GB)
 Java version: 22+36, vendor version: Oracle GraalVM 22+36.1
 Graal compiler: optimization level: b, target machine: native, PGO: off
 C compiler: cc (apple, arm64, 15.0.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 7 user-specific feature(s):
 - com.oracle.svm.polyglot.groovy.GroovyIndyInterfaceFeature
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - org.apereo.cas.nativex.features.DefaultNativeImageFeature
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
 - org.eclipse.angus.mail.nativeimage.AngusMailFeature
 - org.graalvm.home.HomeFinderFeature: Finds GraalVM paths and its version number
 - org.springframework.aot.nativex.feature.PreComputeFieldFeature
------------------------------------------------------------------------------------------------------------------------
 6 experimental option(s) unlocked:
 - '-H:ServiceLoaderFeatureExcludeServices' (origin(s): 'META-INF/native-image/org.springframework/spring-orm/native-image.properties' in 'file:///Users/misagh/.gradle/caches/modules-2/files-2.1/org.springframework/spring-orm/6.1.5/d2dc2b996680fcc8ae5aea294f0ce6bda5577c7c/spring-orm-6.1.5.jar')
 - '-H:+BuildReport' (origin(s): command line)
 - '-H:+InlineBeforeAnalysis' (origin(s): command line)
 - '-H:ResourceConfigurationResources': Use a resource-config.json in your META-INF/native-image/<groupID>/<artifactID> directory instead. (origin(s): 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/native-image.properties' in 'file:///Users/misagh/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-core/10.1.19/3dbbca8acbd4dd6a137c3d6f934a2931512b42ce/tomcat-embed-core-10.1.19.jar', 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/native-image.properties' in 'file:///Users/misagh/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-el/10.1.19/c61a582c391aca130884a5421deedfe1a96c7415/tomcat-embed-el-10.1.19.jar')
 - '-H:+TraceSecurityServices' (origin(s): command line)
 - '-H:ReflectionConfigurationResources': Use a reflect-config.json in your META-INF/native-image/<groupID>/<artifactID> directory instead. (origin(s): 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-core/native-image.properties' in 'file:///Users/misagh/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-core/10.1.19/3dbbca8acbd4dd6a137c3d6f934a2931512b42ce/tomcat-embed-core-10.1.19.jar', 'META-INF/native-image/org.apache.tomcat.embed/tomcat-embed-el/native-image.properties' in 'file:///Users/misagh/.gradle/caches/modules-2/files-2.1/org.apache.tomcat.embed/tomcat-embed-el/10.1.19/c61a582c391aca130884a5421deedfe1a96c7415/tomcat-embed-el-10.1.19.jar')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 10.67GB of memory (16.7% of 64.00GB system memory, set via '-Xmx12G')
 - 10 thread(s) (100.0% of 10 available processor(s), determined at start)
# Printing compilation-target information to: /Users/misagh/Workspace/GitWorkspace/cas-server/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/target_info_20240320_171746.txt
# Printing security services automatic registration to: /Users/misagh/Workspace/GitWorkspace/cas-server/webapp/cas-server-webapp-native/build/native/nativeCompile/reports/security_services_20240320_171752.txt

=== Image generator watchdog detected no activity. This can be a sign of a deadlock during image building. Dumping all stack traces. Current time: Wed Mar 20 17:27:59 GST 2024
"main" Id=1 in BLOCKED on lock=com.oracle.graal.pointsto.flow.MethodTypeFlow@126edb08
      owned by ForkJoinPool.commonPool-worker-3 Id=29
    at app/org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:158)
    at app/org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:152)
    at app/org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:110)
    at app/org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultStaticInvokeTypeFlow.lambda$update$0(DefaultStaticInvokeTypeFlow.java:75)
    at app/org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultStaticInvokeTypeFlow$$Lambda/0x000003fe0366ec90.accept(Unknown Source)

I can share the full stacktrace if it helps.

fniephaus commented 3 months ago

What happens if you double the memory limit from -Xmx12G to -Xmx24G? If the build works, then 12GB is unfortunately not enough for the application and config you are building with.

mmoayyed commented 3 months ago

Unfortunately I have run into a snag because Gradle 8.7 is not quite able to build with java 22 so I'll have to revisit.

Is it alarming that the same exact application setup builds ok under GraalVM 21? Under 10 minutes?

fniephaus commented 3 months ago

Unfortunately I have run into a snag because Gradle 8.7 is not quite able to build with java 22 so I'll have to revisit.

That's what GRAALVM_HOME is for: set JAVA_HOME to a JDK 21 and GRAALVM_HOME to the JDK 22 release and you should be good. Gradle will run on JDK 21 but then use the native-image launcher from JDK 22.

Is it alarming that the same exact application setup builds ok under GraalVM 21? Under 10 minutes?

Yes, it is and it'd be nice to track down the problem. Do you have --strict-image-heap enabled in the GraalVM 21 build? If not, can you turn it on and see whether that slows down your build times in the same way?

mmoayyed commented 3 months ago

Thank you. Looks like I needed to add --initialize-at-build-time=groovyjarjarasm.asm.Handle. With JDK 21 and GraalVM 22 and Gradle 8.7, I am able to build in less than 2 minutes now.

I recall you mentioned a while ago that using pgo is somewhat problematic. Is that still the case here with GraalVM 22?

fniephaus commented 3 months ago

that using pgo is somewhat problematic. Is that still the case here with GraalVM 22?

What I meant was that --pgo-instrument requires additional memory, and in some cases that can be significant. We are working on improving this but I don't think this has been resolved entirely in 22.