AIX JDK13 nightly build failure: log files not found

[M-Davies]

Platform: jdk13u-aix-ppc64-hotspot jdk13u-aix-ppc64-openj9

Please describe the issue that you are having with our binaries and if possible how to recreate it. Since 04-01-2020, the jdk13u-aix-ppc64-hotspot pipeline at adopt has been suffering from various No such file or directory errors when looking for various log files. What files exactly seem to vary for each run.

The common error message:

/opt/freeware/bin/bash: cannot make pipe for process substitution: No such file or directory
/opt/freeware/bin/bash: cannot make pipe for process substitution: No such file or directory
/opt/freeware/bin/bash: >(/opt/freeware/bin/tee -a /home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/libjvm/objs/gcPolicyCounters.o.log): ambiguous redirect
cp_64: cannot stat '/home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/libjvm/objs/gcPolicyCounters.o.log': No such file or directory                             
gmake[3]: *** [lib/CompileJvm.gmk:177: /home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-aix-ppc64-hotspot/workspace/build/src/build/aix-ppc64-server-release/hotspot/variant-server/libjvm/objs/gcPolicyCounters.o] Error 1

gmake[1]: *** [/home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-aix-ppc64-hotspot/workspace/build/src/make/Init.gmk:312: main] Error 2
gmake: *** [/home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-aix-ppc64-hotspot/workspace/build/src/make/Init.gmk:186: product-images] Error 2

The log files that cause the error (in date descending order): heapRegion.o.log gcPolicyCounters.o.log metaspaceCounters.o.log g1BarrierSetRuntime.o.log loopopts.o.log logDecorators.o.log c1_Compiler.o.log abstractInterpreter_ppc.o.log libretransform002.o.log _the.jdk.jdeps_batch.log _the.java.xml.crypto_batch.log BUILD PASSED 15/01/2020 _the.java.sql.rowset_batch.log 16/01/2020

https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk13u/job/jdk13u-aix-ppc64-hotspot/148/

consoleOutput.txt

[sxa]

I'm going to try bumping the files ulimit up a bit. Have also cleared out /tmp/sh-np* files from the machine

[sej-jackson]

The default ulimit for nofiles (number of open files) is 2000, while jenkins has its own stanza setting it to unlimited. I'm not entirely sure how to explain what exactly is going awry, but increasing the default ulimit value to 8192 appears to have got around the problem.

@sxa555 is running some test builds (copies of jdk13u-aix-ppc64-openj9) and the first one has completed successfully, so currently hoping that's not an anomaly...

[sxa]

I believe this is now resolved, although separate errors have shown up today which do not appear to be a result of the same issue. Closing unless proven otherwise ;-)

[gdams]

This appears to still be failing and is now blocking the jdk13 AIX release (https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk13u/job/jdk13u-aix-ppc64-hotspot/156/).

[M-Davies]

@karianna @gdams Since 13 is no longer in support, this can probably be closed. I haven't spotted the log file failures since Jan 21.

[sxa]

Now that we've released JDK13/HotSpot/Jan2020 release successfully this week this is no longer a concern (and the problem must have disappeared)