search

ukri-excalibur / excalibur-tests

Performance benchmarks and regression tests for the ExCALIBUR project
https://ukri-excalibur.github.io/excalibur-tests/
Apache License 2.0
19 stars 16 forks source link

NVCC Compiler Won't Work #238

Closed kaanolgu closed 6 months ago

kaanolgu commented 10 months ago

I executed spack install nvhpc and it installed the nvhpc compilers. I then added the directories into .spack/compilers.yaml file :

- compiler:
    spec: nvhpc@=23.9
    paths:
      cc: /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-broadwell/gcc-13.1.0/nvhpc-23.9-glmhdcpn2c4zouhzuatdrdj7x7igniik/Linux_x86_64/2023/compilers/bin/nvcc
      cxx: /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-broadwell/gcc-13.1.0/nvhpc-23.9-glmhdcpn2c4zouhzuatdrdj7x7igniik/Linux_x86_64/2023/compilers/bin/nvc++
      f77: /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-broadwell/gcc-13.1.0/nvhpc-23.9-glmhdcpn2c4zouhzuatdrdj7x7igniik/Linux_x86_64/2023/compilers/bin/nvfortran
      fc: /lustre/home/br-kolgu/spack/opt/spack/cray-rhel8-broadwell/gcc-13.1.0/nvhpc-23.9-glmhdcpn2c4zouhzuatdrdj7x7igniik/Linux_x86_64/2023/compilers/bin/nvfortran
    flags: {}
    operating_system: rhel8
    target: any
    modules: []
    environment: {}
    extra_rpaths: []

When I try to run reframe -c benchmarks/apps/babelstream -r --tag thrust --system=isambard-macs:volta --setvar=num_cpus_per_task=40 -S build_locally=false -Sspack_spec='babelstream%nvhpc@23.9 +thrust implementation=cuda cuda_arch=70 backend=cuda'

It gives me the following error message : 

==> Warning: duplicate found for gcc@=12.1.0 on rhel8/any. Edit your compilers.yaml configuration to remove it.
==> Error: ProcessError: Command exited with status 77:
    '/var/tmp/pbs.81951.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-src/configure' '--prefix=/lustre/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/nvhpc-23.9/gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui' '--without-guile' '--disable-nls' '--disable-dependency-tracking'

2 errors found in build log:
     6     checking for gawk... gawk
     7     checking whether make sets $(MAKE)... yes
     8     checking whether make supports nested variables... yes
     9     checking whether make supports the include directive... yes (GNU sty
           le)
     10    checking for gcc... /lustre/home/br-kolgu/spack/lib/spack/env/nvhpc/
           nvc
     11    checking whether the C compiler works... no
  >> 12    configure: error: in `/var/tmp/pbs.81951.gw4head/br-kolgu/spack-stag
           e/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-src
           /spack-build':
  >> 13    configure: error: C compiler cannot create executables
     14    See `config.log' for more details

See build log for details:
  /var/tmp/pbs.81951.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-build-out.txt

==> Warning: Skipping build of babelstream-5.0-fkrqvhfz5jf3di3n26hwl5djcxaky4nm since gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui failed
==> Error: babelstream-5.0-fkrqvhfz5jf3di3n26hwl5djcxaky4nm: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

But this compiler works on when I try spack install ... command so I believe there must be a step I am missing inside ReFrame to configure the compiler to picked up by ReFrame properly.

kaanolgu commented 10 months ago 
[2023-11-27T11:04:55] debug: reframe: Initializing system partition 'volta'
[2023-11-27T11:04:55] debug: reframe: Initializing system 'isambard-macs'
[2023-11-27T11:04:55] debug: reframe: Initializing modules system 'nomod'
[2023-11-27T11:04:55] debug: reframe: detecting topology info for isambard-macs:volta
[2023-11-27T11:04:55] debug: reframe: > topology found in configuration file; skipping...
[2023-11-27T11:04:55] debug: reframe: > device auto-detection is not supported
[2023-11-27T11:04:55] debug: reframe: [ReFrame Environment]
  RFM_AUTODETECT_FQDN=<not set>
  RFM_AUTODETECT_METHOD=<not set>
  RFM_AUTODETECT_METHODS=<not set>
  RFM_AUTODETECT_XTHOSTNAME=<not set>
  RFM_CHECK_SEARCH_PATH=<not set>
  RFM_CHECK_SEARCH_RECURSIVE=<not set>
  RFM_CLEAN_STAGEDIR=<not set>
  RFM_COLORIZE=<not set>
  RFM_COMPRESS_REPORT=<not set>
  RFM_CONFIG_FILES=/home/br-kolgu/excalibur-tests/benchmarks/reframe_config.py
  RFM_CONFIG_PATH=<not set>
  RFM_DUMP_PIPELINE_PROGRESS=<not set>
  RFM_GIT_TIMEOUT=<not set>
  RFM_HTTPJSON_URL=<not set>
  RFM_IGNORE_REQNODENOTAVAIL=<not set>
  RFM_INSTALL_PREFIX=/lustre/home/br-kolgu/macs_venv_2/lib/python3.10/site-packages
  RFM_KEEP_STAGE_FILES=<not set>
  RFM_MODULE_MAPPINGS=<not set>
  RFM_MODULE_MAP_FILE=<not set>
  RFM_NON_DEFAULT_CRAYPE=<not set>
  RFM_OUTPUT_DIR=<not set>
  RFM_PERFLOG_DIR=<not set>
  RFM_PERF_INFO_LEVEL=<not set>
  RFM_PIPELINE_TIMEOUT=<not set>
  RFM_PREFIX=<not set>
  RFM_PURGE_ENVIRONMENT=<not set>
  RFM_REMOTE_DETECT=<not set>
  RFM_REMOTE_WORKDIR=<not set>
  RFM_REPORT_FILE=<not set>
  RFM_REPORT_JUNIT=<not set>
  RFM_RESOLVE_MODULE_CONFLICTS=<not set>
  RFM_SAVE_LOG_FILES=<not set>
  RFM_STAGE_DIR=<not set>
  RFM_SYSLOG_ADDRESS=<not set>
  RFM_SYSTEM=<not set>
  RFM_TIMESTAMP_DIRS=<not set>
  RFM_TRAP_JOB_ERRORS=<not set>
  RFM_UNLOAD_MODULES=<not set>
  RFM_USER_MODULES=<not set>
  RFM_USE_LOGIN_SHELL=true
  RFM_VERBOSE=<not set>
[2023-11-27T11:04:55] info: reframe: [ReFrame Setup]
[2023-11-27T11:04:55] info: reframe:   version:           4.4.1
[2023-11-27T11:04:55] info: reframe:   command:           '/lustre/home/br-kolgu/macs_venv_2/bin/reframe -c benchmarks/apps/babelstream -r --tag thrust --system=isambard-macs:volta --setvar=num_cpus_per_task=40 -S build_locally=false -Sspack_spec=babelstream%nvhpc@23.9 +thrust implementation=cuda cuda_arch=70 backend=cuda'
[2023-11-27T11:04:55] info: reframe:   launched by:       br-kolgu@login-02
[2023-11-27T11:04:55] info: reframe:   working directory: '/lustre/home/br-kolgu/excalibur-tests'
[2023-11-27T11:04:55] info: reframe:   settings files:    '<builtin>', '/home/br-kolgu/excalibur-tests/benchmarks/reframe_config.py'
[2023-11-27T11:04:55] info: reframe:   check search path: '/lustre/home/br-kolgu/excalibur-tests/benchmarks/apps/babelstream'
[2023-11-27T11:04:55] info: reframe:   stage directory:   '/lustre/home/br-kolgu/excalibur-tests/stage'
[2023-11-27T11:04:55] info: reframe:   output directory:  '/lustre/home/br-kolgu/excalibur-tests/output'
[2023-11-27T11:04:55] info: reframe:   log files:         '/tmp/rfm-te9208tf.log'
[2023-11-27T11:04:55] info: reframe: 
[2023-11-27T11:04:55] debug: reframe: Looking for tests in '/lustre/home/br-kolgu/excalibur-tests/benchmarks/apps/babelstream'
[2023-11-27T11:04:55] debug: reframe: Validating '/lustre/home/br-kolgu/excalibur-tests/benchmarks/apps/babelstream/babelstream.py': OK
[2023-11-27T11:04:55] debug: reframe:   > Loaded 25 test(s)
[2023-11-27T11:04:55] verbose: reframe: Loaded 25 test(s)
[2023-11-27T11:04:55] verbose: reframe: Generated 11 test case(s)
[2023-11-27T11:04:55] verbose: reframe: Filtering test cases(s) by name: 11 remaining
[2023-11-27T11:04:55] verbose: reframe: Filtering test cases(s) by tags: 1 remaining
[2023-11-27T11:04:55] verbose: reframe: Filtering test cases(s) by other attributes: 1 remaining
[2023-11-27T11:04:55] debug: reframe: Building and validating the full test DAG
[2023-11-27T11:04:55] debug: reframe: Full test DAG:
[2023-11-27T11:04:55] debug: reframe:   ('STDINDICESBenchmark_NVIDIA', 'isambard-macs:volta', 'default') -> []
  ('STDRANGESBenchmark_NVIDIA', 'isambard-macs:volta', 'default') -> []
  ('STDDATABenchmark_NVIDIA', 'isambard-macs:volta', 'default') -> []
  ('OMPBenchmark_NVIDIA', 'isambard-macs:volta', 'default') -> []
  ('RAJABenchmark_GPU', 'isambard-macs:volta', 'default') -> []
  ('THRUSTBenchmark_NVIDIA', 'isambard-macs:volta', 'default') -> []
  ('KOKKOSBenchmark_GPU', 'isambard-macs:volta', 'default') -> []
  ('OCLBenchmark_GPU', 'isambard-macs:volta', 'default') -> []
  ('CUDABenchmark', 'isambard-macs:volta', 'default') -> []
  ('ACCBenchmark_GPU', 'isambard-macs:volta', 'default') -> []
  ('BabelstreamBenchmarkBase', 'isambard-macs:volta', 'default') -> []
[2023-11-27T11:04:55] debug: reframe: Pruned test DAG
[2023-11-27T11:04:55] debug: reframe:   ('THRUSTBenchmark_NVIDIA', 'isambard-macs:volta', 'default') -> []
[2023-11-27T11:04:55] verbose: reframe: Final number of test cases: 1
[2023-11-27T11:04:55] debug: reframe: Loading environment for current system
[2023-11-27T11:04:55] debug: reframe: (Un)using module paths from command line
[2023-11-27T11:04:55] debug: reframe: Loading user modules from command line
[2023-11-27T11:04:55] info: reframe: [==========] Running 1 check(s)
[2023-11-27T11:04:55] info: reframe: [==========] Started on Mon Nov 27 11:04:55 2023 
[2023-11-27T11:04:55] info: reframe: 
[2023-11-27T11:04:55] info: reframe: [----------] start processing checks
[2023-11-27T11:04:55] info: reframe: [ RUN      ] THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default
[2023-11-27T11:04:55] debug: THRUSTBenchmark_NVIDIA /72bd159a: Entering stage: setup
[2023-11-27T11:04:55] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Setting up test paths
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Created stage directory '/lustre/home/br-kolgu/excalibur-tests/stage/isambard-macs/volta/default/THRUSTBenchmark_NVIDIA' [clean_stagedir: True]
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Created output directory '/lustre/home/br-kolgu/excalibur-tests/output/isambard-macs/volta/default/THRUSTBenchmark_NVIDIA'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Setting up <class 'type'> job for test 'THRUSTBenchmark_NVIDIA' (scheduler: 'pbs', launcher: 'mpirun')
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Setting up <class 'type'> job for test 'THRUSTBenchmark_NVIDIA' (scheduler: 'pbs', launcher: 'mpirun')
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: [CMD] 'qsub -o rfm_build.out -e rfm_build.err rfm_build.sh'
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:56] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:56] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:57] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:57] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:57] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:57] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:57] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:57] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:57] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:57] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:57] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:58] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:58] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:58] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:58] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:58] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:58] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:59] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:59] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:04:59] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:04:59] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:00] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:00] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:00] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:00] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:01] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:01] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:01] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:01] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:02] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:02] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:02] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:02] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:03] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:03] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:04] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:04] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:05] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:05] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:06] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:06] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:07] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:07] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:08] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:08] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:10] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:10] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:11] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:11] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:13] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:13] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:14] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:14] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:16] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:16] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:19] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:19] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:21] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:21] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:23] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:23] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:26] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:26] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:29] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:29] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:33] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:33] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:37] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:37] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:41] debug: reframe: [CMD] 'qstat -f 81960.gw4head'
[2023-11-27T11:05:41] debug: reframe: [CMD] 'qstat -xf 81960.gw4head'
[2023-11-27T11:05:41] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile
[2023-11-27T11:05:41] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: Entering stage: compile_wait
[2023-11-27T11:05:41] debug: THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default: caught reframe.core.exceptions.BuildError: stdout: 'rfm_build.out', stderr: 'rfm_build.err'
--- rfm_build.err (first 10 lines) ---
==> Warning: duplicate found for gcc@=12.1.0 on rhel8/any. Edit your compilers.yaml configuration to remove it.
==> Error: ProcessError: Command exited with status 77:
    '/var/tmp/pbs.81960.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-src/configure' '--prefix=/lustre/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/nvhpc-23.9/gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui' '--without-guile' '--disable-nls' '--disable-dependency-tracking'

2 errors found in build log:
     6     checking for gawk... gawk
     7     checking whether make sets $(MAKE)... yes
     8     checking whether make supports nested variables... yes
     9     checking whether make supports the include directive... yes (GNU sty
           le)
--- rfm_build.err --- 
[2023-11-27T11:05:41] info: reframe: [     FAIL ] (1/1) THRUSTBenchmark_NVIDIA /72bd159a @isambard-macs:volta+default
[2023-11-27T11:05:41] info: reframe: ==> test failed during 'compile_wait': test staged in '/lustre/home/br-kolgu/excalibur-tests/stage/isambard-macs/volta/default/THRUSTBenchmark_NVIDIA'
[2023-11-27T11:05:41] verbose: reframe: ==> setup: 0.129s compile: 45.248s run: n/a sanity: n/a performance: n/a total: 45.381s
[2023-11-27T11:05:41] info: reframe: [----------] all spawned checks have finished

[2023-11-27T11:05:41] info: reframe: [  FAILED  ] Ran 1/1 test case(s) from 1 check(s) (1 failure(s), 0 skipped, 0 aborted)
[2023-11-27T11:05:41] info: reframe: [==========] Finished on Mon Nov 27 11:05:41 2023 
[2023-11-27T11:05:41] info: reframe: ====================================================================================================================================================================================
[2023-11-27T11:05:41] info: reframe: SUMMARY OF FAILURES
[2023-11-27T11:05:41] info: reframe: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[2023-11-27T11:05:41] info: reframe: FAILURE INFO for THRUSTBenchmark_NVIDIA (run: 1/1)
[2023-11-27T11:05:41] info: reframe:   * Description: Build BabelStream with Spack Build System
[2023-11-27T11:05:41] info: reframe:   * System partition: isambard-macs:volta
[2023-11-27T11:05:41] info: reframe:   * Environment: default
[2023-11-27T11:05:41] info: reframe:   * Stage directory: /lustre/home/br-kolgu/excalibur-tests/stage/isambard-macs/volta/default/THRUSTBenchmark_NVIDIA
[2023-11-27T11:05:41] info: reframe:   * Node list: 
[2023-11-27T11:05:41] info: reframe:   * Job type: batch job (id=None)
[2023-11-27T11:05:41] info: reframe:   * Dependencies (conceptual): []
[2023-11-27T11:05:41] info: reframe:   * Dependencies (actual): []
[2023-11-27T11:05:41] info: reframe:   * Maintainers: []
[2023-11-27T11:05:41] info: reframe:   * Failing phase: compile_wait
[2023-11-27T11:05:41] info: reframe:   * Rerun with '-n /72bd159a -p default --system isambard-macs:volta -r'
[2023-11-27T11:05:41] info: reframe:   * Reason: build error: stdout: 'rfm_build.out', stderr: 'rfm_build.err'
--- rfm_build.err (first 10 lines) ---
==> Warning: duplicate found for gcc@=12.1.0 on rhel8/any. Edit your compilers.yaml configuration to remove it.
==> Error: ProcessError: Command exited with status 77:
    '/var/tmp/pbs.81960.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-src/configure' '--prefix=/lustre/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/nvhpc-23.9/gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui' '--without-guile' '--disable-nls' '--disable-dependency-tracking'

2 errors found in build log:
     6     checking for gawk... gawk
     7     checking whether make sets $(MAKE)... yes
     8     checking whether make supports nested variables... yes
     9     checking whether make supports the include directive... yes (GNU sty
           le)
--- rfm_build.err --- 
[2023-11-27T11:05:41] verbose: reframe: Traceback (most recent call last):
  File "/lustre/home/br-kolgu/macs_venv_2/lib/python3.10/site-packages/reframe/frontend/executors/__init__.py", line 317, in _safe_call
    return fn(*args, **kwargs)
  File "/lustre/home/br-kolgu/macs_venv_2/lib/python3.10/site-packages/reframe/core/hooks.py", line 109, in _fn
    func(obj, *args, **kwargs)
  File "/lustre/home/br-kolgu/macs_venv_2/lib/python3.10/site-packages/reframe/core/pipeline.py", line 1909, in compile_wait
    raise BuildError(self._build_job.stdout,
reframe.core.exceptions.BuildError: stdout: 'rfm_build.out', stderr: 'rfm_build.err'
--- rfm_build.err (first 10 lines) ---
==> Warning: duplicate found for gcc@=12.1.0 on rhel8/any. Edit your compilers.yaml configuration to remove it.
==> Error: ProcessError: Command exited with status 77:
    '/var/tmp/pbs.81960.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-src/configure' '--prefix=/lustre/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/nvhpc-23.9/gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui' '--without-guile' '--disable-nls' '--disable-dependency-tracking'

2 errors found in build log:
     6     checking for gawk... gawk
     7     checking whether make sets $(MAKE)... yes
     8     checking whether make supports nested variables... yes
     9     checking whether make supports the include directive... yes (GNU sty
           le)
--- rfm_build.err --- 

[2023-11-27T11:05:41] info: reframe: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[2023-11-27T11:05:41] info: reframe: Log file(s) saved in '/tmp/rfm-te9208tf.log'
[2023-11-27T11:05:41] debug: reframe: >>> profiler report [start] <<<
[2023-11-27T11:05:41] debug: reframe: main: 46.036931 s
[2023-11-27T11:05:41] debug: reframe:     test processing: 45.950674 s
[2023-11-27T11:05:41] debug: reframe:         RegressionCheckLoader.load_all: 0.430691 s
[2023-11-27T11:05:41] debug: reframe:             TestRegistry.instantiate_all: 0.329500 s
[2023-11-27T11:05:41] debug: reframe:         generate_testcases: 0.000412 s
[2023-11-27T11:05:41] debug: reframe:         main.<locals>._sort_testcases: 0.000019 s
[2023-11-27T11:05:41] debug: reframe:         build_deps: 0.000276 s
[2023-11-27T11:05:41] debug: reframe:         validate_deps: 0.000204 s
[2023-11-27T11:05:41] debug: reframe:         prune_deps: 0.000029 s
[2023-11-27T11:05:41] debug: reframe:         toposort: 0.000055 s
[2023-11-27T11:05:41] debug: reframe:         Runner.runall: 45.496150 s
[2023-11-27T11:05:41] debug: reframe:             RegressionTask.setup: 0.136647 s
[2023-11-27T11:05:41] debug: reframe:             RegressionTask.compile: 0.059361 s
[2023-11-27T11:05:41] debug: reframe:             RegressionTask.compile_complete: 0.161942 s
[2023-11-27T11:05:41] debug: reframe:             RegressionTask.compile_wait: 0.056470 s
[2023-11-27T11:05:41] debug: reframe: >>> profiler report [ end ] <<<
giordano commented 10 months ago

The most important file which contains the reason why the test is failing is config.log, it should be somewhere around /var/tmp/pbs.81951.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-qfhzizskwnrobnf4s7eqplfqaam3ppui/spack-src/config.log or so

kaanolgu commented 10 months ago

when I try to locate the /var/tmp/pbs.81951.gw4head/ it seems dissappeared. Is it possible to keep it or move it to somewhere else during the execution ? @giordano

Edit :

(macs_venv_2) [br-kolgu@login-02 excalibur-tests]$ /var/tmp/
dnf-br-kolgu-frolaxtm/                                                   systemd-private-28f3e23c5a0e47359ead0dd089578770-chronyd.service-vEmA26/
.flexlm/                                                                 systemd-private-28f3e23c5a0e47359ead0dd089578770-postfix.service-qN4qNh/

Edit 2 : 

/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/gcc-9.2.0/gmake-4.4.1-iqnx5igwzteyyhglbhphz5xkgw5kn4nx/.spack/spack-build-out.txt

I have access to these build reports but not config.log file to be present in that folder

tkoskela commented 10 months ago

Is there anything useful in spack-build-out.txt?

kaanolgu commented 10 months ago

@tkoskela 

==> gmake: Executing phase: 'install'
==> [2023-11-27-01:03:40.992599] '/var/tmp/pbs.81919.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-iqnx5igwzteyyhglbhphz5xkgw5kn4nx/spack-src/configure' '--prefix=/lustre/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/gcc-9.2.0/gmake-4.4.1-iqnx5igwzteyyhglbhphz5xkgw5kn4nx' '--without-guile' '--disable-nls' '--disable-dependency-tracking'
checking for a BSD-compatible install... /bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc accepts -g... yes
checking for /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc option to enable C11 features... none needed
checking whether /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc understands -c and -o together... yes
checking whether the compiler is clang... no
checking for compiler option needed when checking for declarations... none
checking dependency style of /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc... none
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for wchar.h... yes
checking for minix/config.h... no
checking for sys/param.h... yes
checking for stdbool.h... yes
checking for vfork.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether _XOPEN_SOURCE should be defined... no
checking for gcc... (cached) /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc
checking whether the compiler supports GNU C... (cached) yes
checking whether /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc accepts -g... (cached) yes
checking for /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc option to enable C11 features... (cached) none needed
checking whether /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc understands -c and -o together... (cached) yes
checking whether the compiler is clang... (cached) no
checking for compiler option needed when checking for declarations... (cached) none
checking dependency style of /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc... (cached) none
checking whether the compiler supports GNU C++... yes
checking whether /lustre/home/br-kolgu/spack/lib/spack/env/gcc/g++ accepts -g... yes
checking for /lustre/home/br-kolgu/spack/lib/spack/env/gcc/g++ option to enable C++11 features... none needed
checking dependency style of /lustre/home/br-kolgu/spack/lib/spack/env/gcc/g++... none
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to run the C preprocessor... /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for Minix Amsterdam compiler... no
checking for ar... ar
checking for ranlib... ranlib
checking for /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc option to enable large file support... none needed
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for eaccess... yes
checking for /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc options needed to detect all undeclared functions... none needed
checking host CPU and C ABI... x86_64
checking for C compiler option to allow warnings... -Wno-error
checking for alloca as a compiler built-in... yes
checking for getloadavg... yes
checking for sys/loadavg.h... no
checking whether getloadavg is declared... yes
checking if system libc has working GNU glob... yes
checking whether system uses MSDOS-style paths... no
checking for bool, true, false... no
checking for ranlib... (cached) ranlib
checking how to run the C preprocessor... /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc -E
checking for ar... ar
checking for perl... perl
checking whether byte ordering is bigendian... no
checking for a sed that does not truncate output... /bin/sed
checking whether NLS is requested... no
checking for msgfmt... /bin/msgfmt
checking for gmsgfmt... /bin/msgfmt
checking for xgettext... /bin/xgettext
checking for msgmerge... /bin/msgmerge
checking for ld used by /lustre/home/br-kolgu/spack/lib/spack/env/gcc/gcc... /lustre/home/br-kolgu/spack/lib/spack/env/ld
checking if the linker (/lustre/home/br-kolgu/spack/lib/spack/env/ld) is GNU ld... yes
checking for shared library run path origin... done
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking whether to use NLS... no
checking for library containing strerror... none required
checking for library containing getpwnam... none required
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking whether stat file-mode macros are broken... no
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for strings.h... (cached) yes
checking for locale.h... yes
checking for unistd.h... (cached) yes
checking for limits.h... yes
checking for memory.h... yes
checking for sys/param.h... (cached) yes
checking for sys/resource.h... yes
checking for sys/timeb.h... yes
checking for sys/time.h... yes
checking for sys/select.h... yes
checking for sys/file.h... yes
checking for fcntl.h... yes
checking for spawn.h... yes
checking for an ANSI C-conforming const... yes
checking for uid_t in sys/types.h... yes
checking for pid_t... yes
checking for off_t... yes
checking for size_t... (cached) yes
checking for ssize_t... yes
checking for unsigned long long int... yes
checking for long long int... yes
checking for intmax_t... yes
checking for uintmax_t... yes
checking for sig_atomic_t... yes
checking for nanoseconds field of struct stat... st_mtim.tv_nsec
checking whether to use high resolution file timestamps... yes
checking for library containing clock_gettime... none required
checking for standard gettimeofday... yes
checking for strtoll... yes
checking for strdup... yes
checking for strndup... yes
checking for stpcpy... yes
checking for memrchr... yes
checking for mempcpy... yes
checking for umask... yes
checking for mkstemp... yes
checking for mktemp... yes
checking for fdopen... yes
checking for dup... yes
checking for dup2... yes
checking for getcwd... yes
checking for realpath... yes
checking for sigsetmask... yes
checking for sigaction... yes
checking for getgroups... yes
checking for seteuid... yes
checking for setegid... yes
checking for setlinebuf... yes
checking for setreuid... yes
checking for setregid... yes
checking for mkfifo... yes
checking for getrlimit... yes
checking for setrlimit... yes
checking for setvbuf... yes
checking for pipe... yes
checking for strerror... yes
checking for strsignal... yes
checking for lstat... yes
checking for readlink... yes
checking for atexit... yes
checking for isatty... yes
checking for ttyname... yes
checking for pselect... yes
checking for posix_spawn... yes
checking for posix_spawnattr_setsigmask... yes
checking whether bsd_signal is declared... no
checking for fork... yes
checking for vfork... yes
checking for working fork... yes
checking for working vfork... (cached) yes
checking for strcasecmp... yes
checking for strncasecmp... yes
checking for strcmpi... no
checking for strncmpi... no
checking for stricmp... no
checking for strnicmp... no
checking for working strcoll... yes
checking whether closedir returns void... no
checking for struct dirent.d_type... yes
checking for pkg-config... /bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking whether sys_siglist is declared... yes
checking whether _sys_siglist is declared... yes
checking whether __sys_siglist is declared... no
checking for sys/wait.h... yes
checking for waitpid... yes
checking for wait3... yes
checking for union wait... no
checking for SA_RESTART... yes
checking whether dlopen is declared... yes
checking whether dlsym is declared... yes
checking whether dlerror is declared... yes
checking for library containing dlopen... -ldl
checking if the linker accepts -Wl,--export-dynamic... yes
checking for posix_spawn that fails synchronously... yes
checking for location of SCCS get command... get
checking if malloc debugging is wanted... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating build.cfg
config.status: creating tests/config-flags.pm
config.status: creating Makefile
config.status: creating lib/Makefile
config.status: creating doc/Makefile
config.status: creating po/Makefile.in
config.status: creating src/config.h
config.status: linking /var/tmp/pbs.81919.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-iqnx5igwzteyyhglbhphz5xkgw5kn4nx/spack-src/build.sh to build.sh
config.status: executing depfiles commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile
==> [2023-11-27-01:03:48.702615] '/var/tmp/pbs.81919.gw4head/br-kolgu/spack-stage/spack-stage-gmake-4.4.1-iqnx5igwzteyyhglbhphz5xkgw5kn4nx/spack-src/build.sh'
config.status: creating ./lib/alloca.h__
compiling lib/concat-filename.c...
compiling lib/findprog-in.c...
creating libgnu.a...
compiling src/ar.c...
compiling src/arscan.c...
compiling src/commands.c...
compiling src/default.c...
compiling src/dir.c...
compiling src/expand.c...
compiling src/file.c...
compiling src/function.c...
compiling src/getopt.c...
compiling src/getopt1.c...
compiling src/guile.c...
compiling src/hash.c...
compiling src/implicit.c...
compiling src/job.c...
compiling src/load.c...
compiling src/loadapi.c...
compiling src/main.c...
compiling src/misc.c...
compiling src/output.c...
compiling src/read.c...
compiling src/remake.c...
compiling src/rule.c...
compiling src/shuffle.c...
compiling src/signame.c...
compiling src/strcache.c...
compiling src/variable.c...
compiling src/version.c...
compiling src/vpath.c...
compiling src/posixos.c...
compiling src/remote-stub.c...
linking make...
done.
==> [2023-11-27-01:03:54.352447] Installing make to /lustre/home/br-kolgu/excalibur-tests/benchmarks/spack/isambard-macs/volta/opt/cray-rhel8-cascadelake/gcc-9.2.0/gmake-4.4.1-iqnx5igwzteyyhglbhphz5xkgw5kn4nx/bin
tkoskela commented 10 months ago

Hi Kaan, are you still having this issue?

kaanolgu commented 10 months ago

Hi Tuomas, yes still persistent

tkoskela commented 6 months ago

My undersatnding was that this was an issue with the nvhpc spack package, is that right? Can we close this issue, or is there something that could be done on our side?

kaanolgu commented 6 months ago

It was the issue with MACS on Isambard so no longer an issue closing it