search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 720 forks source link

Build script problem #13328

Open zl-wang opened 3 years ago

zl-wang commented 3 years ago

I frequently ran into build failures as below:

Default configuration looks something like: Build performance summary:

  • Cores to use: 16
  • Memory limit: 63290 MB

Build failed as:

Creating images/lib/ct.sym Compiling tools to generate DDR pointer classes and structure stubs JVMJ9GC070E Failed to startup the Garbage Collector JVMJ9GC070E Failed to startup the Garbage Collector JVMJ9GC070E Failed to startup the Garbage Collector JVMJ9GC070E Failed to startup the Garbage Collector Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

When i reconfigured it with --with-num-cores=8 as below:

Build performance summary:

  • Cores to use: 7
  • Memory limit: 63290 MB

It succeeded.

Setting environment variable OPENJ9_JAVA_OPTIONS="-Xmx2g" didn't help though.

I think somewhere the java heap needs to set reasonably, instead of default 1/4th of physical memory.

keithc-ca commented 3 years ago

Which version is this about (8, 11, all)?

zl-wang commented 3 years ago

java8 (although I haven't done the rest ... so, not sure about them)

keithc-ca commented 3 years ago

Setting environment variable OPENJ9_JAVA_OPTIONS="-Xmx2g" didn't help though.

I think the build explicitly puts -Xmx on command lines so that setting in the environment variable would be superseded (or ignored if your bootjdk is not based on openj9).

I don't have access to a system with as many cores or as much memory. Can you restart a build with debug logging:

make LOG=debug images

and share the command lines of jobs that are failing? We may need to add $(JAVA_FLAGS_SMALL) (or $(JAVA_FLAGS_BIG)) to some commands.

zl-wang commented 3 years ago

the command lines are captured in a certain log file? I didn't find it.

zl-wang commented 3 years ago

so, i copied my screen output here:

`Copying images/j2sdk-image/demo/nbproject/project.xml
/bin/echo  Generating images/j2sdk-image/jre/lib/meta-index
Copying images/j2sdk-image/demo/README
/bin/echo  Generating images/j2sdk-image/jre/lib/ext/meta-index
Generating images/j2sdk-image/jre/lib/meta-index
cd /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/jre/lib && /usr/lib/jvm/adoptojdk-java-80/bin/java -XX:+UseSerialGC -Xms32M -Xmx512M -XX:-PrintVMOptions -XX:-UnlockDiagnosticVMOptions -XX:-LogVMOutput  -cp /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/jdk/btclasses build.tools.buildmetaindex.BuildMetaIndex -o meta-index *.jar
Generating images/j2sdk-image/jre/lib/ext/meta-index
/bin/echo  Copying images/j2sdk-image/man/man1/java.1
cd /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/jre/lib/ext && /usr/lib/jvm/adoptojdk-java-80/bin/java -XX:+UseSerialGC -Xms32M -Xmx512M -XX:-PrintVMOptions -XX:-UnlockDiagnosticVMOptions -XX:-LogVMOutput  -cp /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/jdk/btclasses build.tools.buildmetaindex.BuildMetaIndex -o meta-index *.jar
/bin/echo  Copying images/j2sdk-image/man/man1/jjs.1
Copying images/j2sdk-image/man/man1/java.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
Copying images/j2sdk-image/man/man1/jjs.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/keytool.1
/bin/echo  Copying images/j2sdk-image/man/man1/orbd.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/java.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/java.1'
Copying images/j2sdk-image/man/man1/keytool.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
Copying images/j2sdk-image/man/man1/orbd.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jjs.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jjs.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/pack200.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/keytool.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/keytool.1'
/bin/echo  Copying images/j2sdk-image/man/man1/policytool.1
Copying images/j2sdk-image/man/man1/pack200.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/orbd.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/orbd.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
Copying images/j2sdk-image/man/man1/policytool.1
/bin/echo  Copying images/j2sdk-image/man/man1/rmid.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
Copying images/j2sdk-image/man/man1/rmid.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/pack200.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/pack200.1'
/bin/echo  Copying images/j2sdk-image/man/man1/rmiregistry.1
/bin/echo  Copying images/j2sdk-image/man/man1/servertool.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/policytool.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/policytool.1'
Copying images/j2sdk-image/man/man1/rmiregistry.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/rmid.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/rmid.1'
Copying images/j2sdk-image/man/man1/servertool.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/tnameserv.1
/bin/echo  Copying images/j2sdk-image/man/man1/unpack200.1
Copying images/j2sdk-image/man/man1/tnameserv.1
Copying images/j2sdk-image/man/man1/unpack200.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/rmiregistry.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/rmiregistry.1'
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/servertool.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/servertool.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/appletviewer.1
Copying images/j2sdk-image/man/man1/appletviewer.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/tnameserv.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/tnameserv.1'
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/unpack200.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/unpack200.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/extcheck.1
Copying images/j2sdk-image/man/man1/extcheck.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/appletviewer.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/appletviewer.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/idlj.1
Copying images/j2sdk-image/man/man1/idlj.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/extcheck.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/extcheck.1'
/bin/echo  Copying images/j2sdk-image/man/man1/jar.1
Copying images/j2sdk-image/man/man1/jar.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/idlj.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/idlj.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/jarsigner.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jar.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jar.1'
Copying images/j2sdk-image/man/man1/jarsigner.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/javac.1
Copying images/j2sdk-image/man/man1/javac.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jarsigner.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jarsigner.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/javadoc.1
Copying images/j2sdk-image/man/man1/javadoc.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/javac.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/javac.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/javah.1
Copying images/j2sdk-image/man/man1/javah.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/javadoc.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/javadoc.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/javap.1
Copying images/j2sdk-image/man/man1/javap.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/javah.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/javah.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/jconsole.1
Copying images/j2sdk-image/man/man1/jconsole.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/javap.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/javap.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/jdb.1
Copying images/j2sdk-image/man/man1/jdb.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jconsole.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jconsole.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/jdeps.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jdb.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jdb.1'
Copying images/j2sdk-image/man/man1/jdeps.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/jmap.1
Copying images/j2sdk-image/man/man1/jmap.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jdeps.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jdeps.1'
/bin/echo  Copying images/j2sdk-image/man/man1/jrunscript.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jmap.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jmap.1'
Copying images/j2sdk-image/man/man1/jrunscript.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/jsadebugd.1
Copying images/j2sdk-image/man/man1/jsadebugd.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jrunscript.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jrunscript.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/native2ascii.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/jsadebugd.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/jsadebugd.1'
Copying images/j2sdk-image/man/man1/native2ascii.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/rmic.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/native2ascii.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/native2ascii.1'
Copying images/j2sdk-image/man/man1/rmic.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/schemagen.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/rmic.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/rmic.1'
Copying images/j2sdk-image/man/man1/schemagen.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/serialver.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/schemagen.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/schemagen.1'
Copying images/j2sdk-image/man/man1/serialver.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/wsgen.1
Copying images/j2sdk-image/man/man1/wsgen.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/serialver.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/serialver.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/wsimport.1
Copying images/j2sdk-image/man/man1/wsimport.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/wsgen.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/wsgen.1'
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
/bin/echo  Copying images/j2sdk-image/man/man1/xjc.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/java.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/wsimport.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/wsimport.1'
Copying images/j2sdk-image/man/man1/xjc.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/java.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/jjs.1
/bin/cp -fP '/root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/xjc.1' '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/man1/xjc.1'
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/jjs.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/java.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/java.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/keytool.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/orbd.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/jjs.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/jjs.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/keytool.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/pack200.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/orbd.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/pack200.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/policytool.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/policytool.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/keytool.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/keytool.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/orbd.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/orbd.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/rmid.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/pack200.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/pack200.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/policytool.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/policytool.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/rmid.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/rmiregistry.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/servertool.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/rmid.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/rmid.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/rmiregistry.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/servertool.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/tnameserv.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/unpack200.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/tnameserv.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/unpack200.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/rmiregistry.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/rmiregistry.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/servertool.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/servertool.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/appletviewer.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/appletviewer.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/tnameserv.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/tnameserv.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/unpack200.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/unpack200.1
/bin/echo  Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/extcheck.1
Converting images/j2sdk-image/man/ja_JP.UTF-8/man1/extcheck.1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/appletviewer.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/appletviewer.1
/bin/mkdir -p /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1
/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/extcheck.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/extcheck.1
JVMJ9GC069E Failed to initialize Finalize thread
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Images.gmk:436: recipe for target '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/policytool.1' failed
make[2]: *** [/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/policytool.1] Error 1
make[2]: *** Waiting for unfinished jobs....
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Images.gmk:436: recipe for target '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/orbd.1' failed
make[2]: *** [/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/orbd.1] Error 1
Images.gmk:436: recipe for target '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/keytool.1' failed
make[2]: *** [/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/keytool.1] Error 1
Generating DDR pointer class source files
Generating DDR structure stub source files
Generating DDR pointer and structure class files
superset directory name: /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/vm/
superset file name: superset.dat
superset directory name : /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/vm/
superset file name : superset.dat
Writing generated classes to /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/jdk/ddr/gensrc/com/ibm/j9ddr/vm29/pointer/generated
Writing generated classes to /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/jdk/ddr/gensrc/com/ibm/j9ddr/vm29/structure
Unhandled structure type: _IO_FILEPointer->_lock _IO_lock_t*
Structure stub generation complete
Pointer class generation complete
make[3]: Leaving directory '/root/openj9-openjdk-jdk8/jdk/make'
make[2]: Leaving directory '/root/openj9-openjdk-jdk8/jdk/make'
make[1]: *** [images] Error 2
BuildJdk.gmk:101: recipe for target 'images' failed
make[1]: Leaving directory '/root/openj9-openjdk-jdk8/jdk/make'
/root/openj9-openjdk-jdk8//make/Main.gmk:136: recipe for target 'images-only' failed
make: *** [images-only] Error 2`
zl-wang commented 3 years ago

however, my quick skimming indicated java has mx512m setting. now, i am not sure what the problem is anymore. we have plenty of memory.

JVMJ9GC069E Failed to initialize Finalize thread might mean something.

keithc-ca commented 3 years ago

All the uses of the bootjdk in those commands contain -Xmx512M, so 16 such processes should fit comfortably in the nearly 64GB of available RAM.

Have you checked ulimit settings (e.g. ulimit -u)? Perhaps a limit prevents creation of a sufficient number of threads?

zl-wang commented 3 years ago

plentiful to me:

root@1d7b74e63c3e:~/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 252877
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4194304
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
zl-wang commented 3 years ago

it sounds like a limit problem, since it succeeded if make is launched to continue. it failed from scratch. but i set everything i can change to unlimited. that didn't help.

keithc-ca commented 3 years ago

If I pretend my virtual machine with 8GB of RAM has 64GB, my bootjdk

OpenJDK 64-Bit Server VM (24.232-b6) for linux-amd64 JRE (Zulu 7.31.0.5-CA-linux64) (1.7.0_232-b6), built on Jul 10 2019 14:03:06 by "zulu_re" with gcc 4.4.7 20120313 (Red Hat 4.4.7-3)

fails:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 7158628352 bytes for committing reserved memory.

If in your build, 16 processes trying to allocate 7GB each will find insufficient memory as well.

zl-wang commented 3 years ago

but, didn't all java have -J-Xmx512m in place? still not sure. that was my initial suspicion though.

keithc-ca commented 3 years ago

The -J option is not for java, but other, more specific, launchers. The failure may be due to a process that was started earlier (and is consuming a significant amount of memory).

zl-wang commented 3 years ago

what we can do to pin it down?

keithc-ca commented 3 years ago

You could try my suggestion:

make LOG=debug images 2>&1 | tee debug.log

and share debug.log.

zl-wang commented 3 years ago

you might spot something out of this. "make all" cannot be continued (i.e. previously "make images" 2nd time succeeded, "make all" 2nd time still failed) ... seemed consistent failure.

zl-wang commented 3 years ago 
[100%] Built target j9ddr
## Starting langtools
## Finished langtools (build time 00:00:01)

## Starting corba
## Finished corba (build time 00:00:00)

## Starting jaxp
## Finished jaxp (build time 00:00:01)

## Starting jaxws
## Finished jaxws (build time 00:00:03)

## Starting jdk
## Finished jdk (build time 00:00:10)

## Starting demos
## Finished demos (build time 00:00:02)

## Starting nashorn
## Finished nashorn (build time 00:00:00)

## Starting images
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
JVMJ9GC070E Failed to startup the Garbage Collector
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
JVMJ9GC069E Failed to initialize Finalize thread
JVMJ9GC070E Failed to startup the Garbage Collector
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
JVMJ9GC070E Failed to startup the Garbage Collector
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
zl-wang commented 3 years ago

debug.log

keithc-ca commented 3 years ago

The only failures I see are:

Images.gmk:436: recipe for target '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/rmid.1' failed
Images.gmk:436: recipe for target '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/rmiregistry.1' failed
Images.gmk:436: recipe for target '/root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/appletviewer.1' failed

Each command to produce those targets follows this pattern:

/bin/cat /root/openj9-openjdk-jdk8/jdk/src/linux/doc/man/ja/rmid.1 | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -encoding eucJP | /bin/sed 's/@@VERSION@@//g' | /usr/lib/jvm/adoptojdk-java-80/bin/native2ascii -J-XX:+UseSerialGC -J-Xms32M -J-Xmx512M -reverse -encoding  UTF-8 > /root/openj9-openjdk-jdk8/build/linux-ppc64le-normal-server-release/images/j2sdk-image/man/ja_JP.UTF-8/man1/rmid.1

The options -J-Xmx512M should limit the heaps reasonably (although at first I didn't notice two instances of native2ascii in that pipeline).

Is the system being used concurrently for anything else that might limit the amount of available memory?

zl-wang commented 3 years ago

no. i am the only user.

[zlwang@pjavarhel02 ~]$ top
top - 17:02:29 up 3 days,  7:25,  1 user,  load average: 1.01, 1.03, 1.52
Tasks: 1151 total,   1 running, 1150 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  63290.0 total,  23453.7 free,   4075.4 used,  35760.9 buff/cache
MiB Swap:  19695.9 total,  19695.9 free,      0.0 used.  58400.9 avail Mem
keithc-ca commented 3 years ago

The configuration process assumes 1GB per job will be sufficient. The output you quoted says only 23453.7MB are free (not 64GB as configure might think). Something is using ~40GB: I think that's the reason your builds are failing.

zl-wang commented 3 years ago

you saw that 35GB is the file-system buffer/cache. That amount is elastic ... certainly usable by running processes. on the other hand, if free memory is what matters, the build script should get that number, instead of total physical memory.

secondly, i don't think that is the failing reason. if it were, you would expect --with-memory-size=20000 option would have worked (but it didn't).

keithc-ca commented 3 years ago

I did see the 35GB buff/cache figure, and almost didn't offer that comment.

Unless we can identify the consumer(s) of the resources (memory, threads/processes, something else?) it's not clear what can be done other than to recommend effectively what you already tried: --with-jobs=8 (a more direct option compared to --with-num-cores=8).

As I said, configuration assumes that 1GB per job is sufficient, so --with-memory-size=20000 is less constraining than the 16 cores available.

zl-wang commented 3 years ago

if we can specifically limit the number of concurrent java processes started (i.e. --with-jobs option applying to java strictly), we can pin down the reason that java is the culprit. easily do-able?

keithc-ca commented 3 years ago

--with-jobs does not apply just to Java jobs; it's just used for make's job control (i.e. the -j option). Also, it's not clear that a Java job is the culprit. Please explain your idea that makes it 'easily do-able'.

zl-wang commented 3 years ago

i knew that. i meant, if --with-jobs can be changed to apply to java only, then the investigation further will be easily do-able to pin it down. --with-memory-size=20000 --with-num-cores=8 is successful, as well as only --with-num-cores=8. if only with --with-memory-size=20000, it failed.

keithc-ca commented 3 years ago

To me, it doesn't make sense for --with-jobs to apply only to Java; process resources must be managed for native compiles and all other activities that need to occur in a build.

As I said, --with-memory-size=20000 doesn't constrain the configuration, only --with-jobs (influenced by --with-num-cores) reduce the number of concurrent tasks during the build.

zl-wang commented 3 years ago

that is just an experimental proposal as a way to investigate why it failed. not really do it permanently.

keithc-ca commented 3 years ago

My point is that it's not even a realistic approach, even temporarily.