Closed felker closed 4 months ago
@changgoo are these builds timing out or some issue with the filesystem/ storage quota? Both this and #591 Jenkins jobs are repeatedly failing, at seemingly random tests, right after the configure step, somewhere in the compile/link step, with non descriptive errors:
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/globals.o': 'No such file or directory'
1 error generated.
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/main.o': 'No such file or directory'
1 error generated.
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals.o': 'No such file or directory'
1 error generated.
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/globals.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/main.o] Error 1
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals.o] Error 1
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_var.o': 'No such file or directory'
1 error generated.
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine.o': 'No such file or directory'
1 error generated.
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_var.o] Error 1
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine.o] Error 1
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine_postmg.o': 'No such file or directory'
1 error generated.
error: unable to open output file '/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/parameter_input.o': 'No such file or directory'
1 error generated.
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine_postmg.o] Error 1
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/parameter_input.o] Error 1
Something bad happened
For the latest build of #591, it made it nearly to the end, dying at the Intel oneAPI-compiled iteration of symmetry
:
https://github.com/PrincetonUniversity/athena/blob/f701e6785d710ef6a90fb23254e4932b8de27ff6/tst/ci/jenkins/run_jenkins_stellar.sh#L148
For the latest build of this PR, it died a few tests earlier at the MPI+Intel Classic C++ compiler + OMP (hybrid) linear wave test: https://github.com/PrincetonUniversity/athena/blob/f701e6785d710ef6a90fb23254e4932b8de27ff6/tst/ci/jenkins/run_jenkins_stellar.sh#L135-L136
Slightly different error:
Compilation command: mpicxx -O3 -std=c++11 -ipo -xhost -inline-forceinline -qopenmp-simd -qopt-prefetch=4 -qoverride-limits -diag-disable=10441 -qopenmp
error: can't open file /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/globals.o for write
compilation aborted for src/globals.cpp (code 1)
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/globals.o] Error 1
make: *** Waiting for unfinished jobs....
error: can't open file /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/parameter_input.o for write
compilation aborted for src/parameter_input.cpp (code 1)
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/parameter_input.o] Error 1
error: can't open file /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine_postmg.o for write
compilation aborted for src/bvals/bvals_refine_postmg.cpp (code 1)
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine_postmg.o] Error 1
error: can't open file /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine.o for write
compilation aborted for src/bvals/bvals_refine.cpp (code 1)
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine.o] Error 1
error: can't open file /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals.o for write
compilation aborted for src/bvals/bvals.cpp (code 1)
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals.o] Error 1
error: can't open file /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/main.o for write
compilation aborted for src/main.cpp (code 1)
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/main.o] Error 1
Something bad happened
I think this error is unique to Intel compilers compilation aborted for src/main.cpp (code 1)
. Might be time to bump the oneAPI compiler version, and drop the testing of the classic Intel compiler on Stellar...
retest this please
this time it died a lot earlier, at the GCC gr/mhd_shocks_hlld
test variant:
Assembler messages:
Fatal error: can't create /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/globals.o: No such file or directory
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/globals.o] Error 1
make: *** Waiting for unfinished jobs....
Assembler messages:
Fatal error: can't create /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine_postmg.o: No such file or directory
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine_postmg.o] Error 1
Assembler messages:
Fatal error: can't create /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_var.o: No such file or directory
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_var.o] Error 1
Assembler messages:
Fatal error: can't create /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine.o: No such file or directory
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/bvals_refine.o] Error 1
Assembler messages:
Fatal error: can't create /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/main.o: No such file or directory
make: *** [Makefile:120: /scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/main.o] Error 1
Something bad happened
Traceback (most recent call last):
File "/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/scripts/utils/athena.py", line 80, in make
subprocess.check_call(make_command, stdout=out_log)
File "/usr/licensed/anaconda3/2020.11/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j8', 'EXE_DIR:=/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/bin/', 'OBJ_DIR:=/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/']' returned non-zero exit status 2.
Exception occurred
Traceback (most recent call last):
File "/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/scripts/utils/athena.py", line 80, in make
subprocess.check_call(make_command, stdout=out_log)
File "/usr/licensed/anaconda3/2020.11/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j8', 'EXE_DIR:=/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/bin/', 'OBJ_DIR:=/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/']' returned non-zero exit status 2.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./run_tests.py", line 134, in main
module.prepare(**kwargs)
File "/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/scripts/tests/gr/mhd_shocks_hlld.py", line 22, in prepare
athena.make()
File "/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/scripts/utils/athena.py", line 84, in make
raise AthenaError('Return code {0} from command \'{1}\''
scripts.utils.athena.AthenaError: Return code 2 from command 'make -j8 EXE_DIR:=/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/bin/ OBJ_DIR:=/scratch/gpfs/changgoo/jenkins/workspace/tigris/athena-PR/tst/regression/obj/'
---> Error in scripts/tests/gr/mhd_shocks_hlld.py
gr.mhd_shocks_hlld test: prepare(), run(), analyze() finished
so definitely not limited to Intel compilers...
Certainly, Princeton research computing experiences an overall crunch of storage. I still have some space in my quota though. I also saw such failure a few times recently. Repeating retest solved the problem, but I think it is an issue of the server.
retest this please
Leaving the contents of
tst/ci/travis/
in place for now, since the scripts are more-or-less general enough to be useful if you were building and testing Athena++ and its dependencies on a bare Docker container, e.g. (apart from the caching stuff which was built to speed up Travis builds back in the day)Future: