Improve native image build error message

[johnaohara]

Building a native image with graalvm ce rc11 & rc12, I occasionally see the following error;

Error: Image building with exit status 137

This occurs if the system runs out of ram (including free swap pages) and the OOM Killer terminates the native image build process.

The error message seen by users is not very informative. Could we capture the exit code of the native-image process, and if equal to 137, display a more helpful message to users?

[johnaohara]

[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] /opt/graalvm-ce-1.0.0-rc11//bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar broker-service-0.10.0-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:+PrintAnalysisCallTree -H:EnableURLProtocols=http -H:NativeLinkerOption=-no-pie -H:-SpawnIsolates -H:-JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace
[broker-service-0.10.0-runner:15792]    classlist:   7,589.30 ms
17:04:19,971 INFO  [com.sun.xml.bin.v2.run.ref.opt.AccessorInjector] The optimized code generation is disabled
[broker-service-0.10.0-runner:15792]        (cap):   1,006.80 ms
[broker-service-0.10.0-runner:15792]        setup:  28,626.98 ms
17:04:57,314 INFO  [org.hib.Version] HHH000412: Hibernate Core {5.4.1.Final}
17:04:57,410 INFO  [org.hib.ann.com.Version] HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
17:04:57,565 INFO  [org.hib.dia.Dialect] HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL9Dialect
17:04:57,771 INFO  [org.jbo.res.res.i18n] RESTEASY002225: Deploying javax.ws.rs.core.Application: class io.quarkus.benchmark.insurance.agents.AgentsApplication
17:05:14,042 INFO  [com.arj.ats.arjuna] ARJUNA012170: TransactionStatusManager started on port 35787 and host 127.0.0.1 with service com.arjuna.ats.arjuna.recovery.ActionStatusService
17:05:15,285 INFO  [org.xnio] XNIO version 3.6.5.Final
17:05:15,704 INFO  [org.xni.nio] XNIO NIO Implementation Version 3.6.5.Final
17:05:18,459 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Alpha4
[broker-service-0.10.0-runner:15792]   (typeflow):  70,162.22 ms
[broker-service-0.10.0-runner:15792]    (objects):  29,723.67 ms
[broker-service-0.10.0-runner:15792]   (features):     808.31 ms
[broker-service-0.10.0-runner:15792]     analysis: 141,753.91 ms
Printing call tree to /home/johara/Work/projects/istio-benchmark/broker-service/target/reports/call_tree_broker-service-0.10.0-runner_20190301_170757.txt
Printing list of used classes to /home/johara/Work/projects/istio-benchmark/broker-service/target/reports/used_classes_broker-service-0.10.0-runner_20190301_170805.txt
Printing list of used packages to /home/johara/Work/projects/istio-benchmark/broker-service/target/reports/used_packages_broker-service-0.10.0-runner_20190301_170805.txt
[broker-service-0.10.0-runner:15792]     universe:   2,513.22 ms
[broker-service-0.10.0-runner:15792]      (parse):   9,026.74 ms
[broker-service-0.10.0-runner:15792]     (inline):   8,000.62 ms
[broker-service-0.10.0-runner:15792]    (compile):  79,834.58 ms
[broker-service-0.10.0-runner:15792]      compile: 136,350.99 ms
[broker-service-0.10.0-runner:15792]        image:   6,252.18 ms
Error: Image building with exit status 137

[johnaohara]

This appears to be related to memory available to build native image. The native-image tool used to report an OutOfMemory exception, but in recent graal release it appears that a build can fail with exit status 137 if there is insufficient memory to complete a native image build

[johnaohara]

re-opening to track enhancement for capturing exit code of native-image builder process

[Cotillion277]

+1 for improvement. Capturing these error codes and providing an actual verbose and useful error message would remove a lot of frustration, saving businesses and employees time and money.

[gasgithub]

You really need to improve memory usage on this build process, I had to set 10GB !!! in Windows Docker desktop to successfully built very basic app, as it failed with 8GB.

[SuppieRK]

Same issue here but with both Micronaut and Quarkus. Maybe we need to create an issue towards GraalVM itself?

[koziolk]

Do J-Xmx10G and -J-Xms5G help in the build process?

[nurkiewicz]

I get the same error with Spring Native 0.9.0, just created using start.spring.io:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.4.3</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demo</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>11</java.version>
        <spring-native.version>0.9.0</spring-native.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.experimental</groupId>
            <artifactId>spring-native</artifactId>
            <version>${spring-native.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <builder>paketobuildpacks/builder:tiny</builder>
                        <env>
                            <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
                        </env>
                    </image>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.experimental</groupId>
                <artifactId>spring-aot-maven-plugin</artifactId>
                <version>${spring-native.version}</version>
                <executions>
                    <execution>
                        <id>test-generate</id>
                        <goals>
                            <goal>test-generate</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>generate</id>
                        <goals>
                            <goal>generate</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <repositories>
        <repository>
            <id>spring-releases</id>
            <name>Spring Releases</name>
            <url>https://repo.spring.io/release</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <name>Spring Releases</name>
            <url>https://repo.spring.io/release</url>
        </pluginRepository>
    </pluginRepositories>

</project>

java --version:

openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment GraalVM CE 21.0.0.2 (build 11.0.10+8-jvmci-21.0-b06)
OpenJDK 64-Bit Server VM GraalVM CE 21.0.0.2 (build 11.0.10+8-jvmci-21.0-b06, mixed mode, sharing)

Output of mvn spring-boot:build-image:

...
[INFO]     [creator]     WARNING: Could not register reflection metadata for org.springframework.boot.autoconfigure.validation.ValidationAutoConfiguration. Reason: java.lang.NoClassDefFoundError: javax/validation/ValidatorFactory.
[INFO]     [creator]     Error: Image build request failed with exit status 137
[INFO]     [creator]     unable to invoke layer creator
[INFO]     [creator]     unable to contribute native-image layer
[INFO]     [creator]     error running build
[INFO]     [creator]     exit status 137
[INFO]     [creator]     ERROR: failed to build: exit status 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:40 min
[INFO] Finished at: 2021-03-12T08:42:14+01:00
[INFO] ------------------------------------------------------------------------

Where to put -J-Xmx10G -J-Xms5G?

[abvijaykumar]

Hi Guys - I am also haivng the same issue with simple Micronaut application

[abvijaykumar]

I fixed the issue by increasing the memory allocated to my docker engine. I was using Docker Desktop, so its a configuration, I did in the application

[rafareyes7]

Setting -Dquarkus.native.native-image-xmx=8g and Docker RAM memory = 8g (Docker App -> Preferences) worked for me.

[mshannongit]

Run docker stats to see how much memory the container associated with the docker build activity is consuming. If the mem usage of the container starts reaching the specified limit, then the 137 status will result. In which case, as others have mentioned above, you will need to tweak the docker configuration on the machine performing the build to allocate additional memory to containers. This could be in Docker deskop, Rancher desktop, or whatever docker/container solution one is leveraging on their build machine.

[H-Lo]

LOL I got this error a minute ago. I thought it was a joke. Reminds me of Hitchhiker's guide to the galaxy and the answer to the ultimate question.

[davidabreu7]

+1 to this issue. I just got an error saying "exit code 20" I already lost time trying to figure out what is error code 20? Where can I find a list of error codes?

[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: io.quarkus.deployment.pkg.steps.NativeImageBuildStep$ImageGenerationFailureException: Image generation failed. Exit code: 20

[H-Lo]

@davidabreu7 Actually, you got an error code 32 because it was hex! LOL

[ibak-pje]

@davidabreu7 Did you find more info on that exit code?

[frim-blum]

@davidabreu7 any progress with exit code 20?