search

SCOREC / core

parallel finite element unstructured meshes
Other
179 stars 63 forks source link

Spack test fails: "gmi failed: could not open model file" #370

Closed SamBosch77 closed 1 year ago

SamBosch77 commented 2 years ago

Summary:

Post-install with spack, the program fails the internal spack tests at runtime. Let me know if you need any additional information I could provide to help with this process.

Error Details:

Console output

rl5jdun
==> Error: TestFailure: 2 tests failed.

Command exited with status -6:
    './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
gmi failed: could not open model file

1 error found in test log:
  >> 5    FAILED: Command exited with status -6:
     6        './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
     7    gmi failed: could not open model file
     8
     9      File "/home/users/sbosch/spack/bin/spack", line 98, in <module>
     10       sys.exit(spack.main.main())
     11     File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 893, in main

Command exited with status 134:
    '/storage/users/sbosch/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.4.0/mpich-4.0.2-rxj536cii5nmlozrnc5p3anskgw5fwgc/bin/mpiexec' '-n' '2' 'split' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_2_.smb' '2'

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 4097551 RUNNING AT instinct
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
gmi failed: could not open model file
gmi failed: could not open model file

1 error found in test log:
     58    ===================================================================================
     59    YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
     60    This typically refers to a problem with your application.
     61    Please see the FAQ page for debugging suggestions
     62    gmi failed: could not open model file
     63    gmi failed: could not open model file
  >> 64    FAILED: Command exited with status 134:
     65        '/storage/users/sbosch/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.4.0/mpich-4.0.2-rxj536cii5nmlozrnc5p3anskg
           w5fwgc/bin/mpiexec' '-n' '2' 'split' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_2_.smb' '2'
     66
     67    ===================================================================================
     68    =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
     69    =   PID 4097551 RUNNING AT instinct
     70    =   EXIT CODE: 134

/storage/users/sbosch/spack/lib/spack/spack/build_environment.py:1076, in _setup_pkg_and_run:
       1073        tb_string = traceback.format_exc()
       1074
       1075        # build up some context from the offending package so we can
  >>   1076        # show that, too.
       1077        package_context = get_package_context(tb)
       1078
       1079        logfile = None

See test log for details:
  /home/users/sbosch/.spack/test/ahmtzvwalket6vutjhqvd55pjmdtqgsa/pumi-2.2.7-rl5jdun-test-out.txt

==> Error: 1 test(s) in the suite failed.

--- ==> Spack test ahmtzvwalket6vutjhqvd55pjmdtqgsa
==> Testing package pumi-2.2.7-rl5jdun
======================== 1 failed, 0 passed of 1 specs ========================= ---

pumi-test-out.txt 

==> Testing package pumi-2.2.7-rl5jdun
==> [2022-08-29-11:27:20.022131] testing pumi uniform mesh refinement
==> [2022-08-29-11:27:20.022406] './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
gmi failed: could not open model file
FAILED: Command exited with status -6:
    './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
gmi failed: could not open model file

  File "/home/users/sbosch/spack/bin/spack", line 98, in <module>
    sys.exit(spack.main.main())
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 893, in main
    return _main(argv)
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 848, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 876, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 533, in _invoke_command
    return_val = command(parser, args)
  File "/storage/users/sbosch/spack/lib/spack/spack/cmd/test.py", line 439, in test
    globals()['test_%s' % args.test_command](args)
  File "/storage/users/sbosch/spack/lib/spack/spack/cmd/test.py", line 208, in test_run
    test_suite(remove_directory=not args.keep_stage,
  File "/storage/users/sbosch/spack/lib/spack/spack/install_test.py", line 164, in __call__
    spec.package.do_test(dirty=dirty, externals=externals)
  File "/storage/users/sbosch/spack/lib/spack/spack/package.py", line 2006, in do_test
    spack.build_environment.start_build_process(
  File "/storage/users/sbosch/spack/lib/spack/spack/build_environment.py", line 1171, in start_build_process
    p.start()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/storage/users/sbosch/spack/lib/spack/spack/build_environment.py", line 1060, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/storage/users/sbosch/spack/lib/spack/spack/package.py", line 2905, in test_process
    test_fn(pkg)
  File "/storage/users/sbosch/spack/var/spack/repos/builtin/packages/pumi/package.py", line 107, in test
    self.run_test(exe, options, expected, purpose=description,
==> [2022-08-29-11:27:20.779361] testing pumi mesh partitioning
==> [2022-08-29-11:27:20.779744] '/storage/users/sbosch/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.4.0/mpich-4.0.2-rxj536cii5nmlozrnc5p3anskgw5fwgc/bin/mpiexec' '-n' '2' 'split' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_2_.smb' '2'

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 4097551 RUNNING AT instinct
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
gmi failed: could not open model file
gmi failed: could not open model file
FAILED: Command exited with status 134:
    '/storage/users/sbosch/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.4.0/mpich-4.0.2-rxj536cii5nmlozrnc5p3anskgw5fwgc/bin/mpiexec' '-n' '2' 'split' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_2_.smb' '2'

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 4097551 RUNNING AT instinct
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
gmi failed: could not open model file
gmi failed: could not open model file

  File "/home/users/sbosch/spack/bin/spack", line 98, in <module>
    sys.exit(spack.main.main())
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 893, in main
    return _main(argv)
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 848, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 876, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/storage/users/sbosch/spack/lib/spack/spack/main.py", line 533, in _invoke_command
    return_val = command(parser, args)
  File "/storage/users/sbosch/spack/lib/spack/spack/cmd/test.py", line 439, in test
    globals()['test_%s' % args.test_command](args)
  File "/storage/users/sbosch/spack/lib/spack/spack/cmd/test.py", line 208, in test_run
    test_suite(remove_directory=not args.keep_stage,
  File "/storage/users/sbosch/spack/lib/spack/spack/install_test.py", line 164, in __call__
    spec.package.do_test(dirty=dirty, externals=externals)
  File "/storage/users/sbosch/spack/lib/spack/spack/package.py", line 2006, in do_test
    spack.build_environment.start_build_process(
  File "/storage/users/sbosch/spack/lib/spack/spack/build_environment.py", line 1171, in start_build_process
    p.start()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/storage/users/sbosch/spack/lib/spack/spack/build_environment.py", line 1060, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/storage/users/sbosch/spack/lib/spack/spack/package.py", line 2905, in test_process
    test_fn(pkg)
  File "/storage/users/sbosch/spack/var/spack/repos/builtin/packages/pumi/package.py", line 116, in test
    self.run_test(mpiexec, mpiopt + exe + options, expected,

System and Build Info:

spack debug report

  • Spack: 0.18.0 (c09bf37ff690c29779a342670cf8a171ad1b9233)
  • Python: 3.8.10
  • Platform: linux-ubuntu20.04-broadwell
  • Concretizer: clingo

spack find -dvl pumi

==> 1 installed package
-- linux-ubuntu20.04-x86_64 / gcc@9.4.0 -------------------------
rl5jdun pumi@2.2.7~fortran~int64~ipo~shared+simmodsuite_version_check~testing~zoltan build_type=RelWithDebInfo simmodsuite=none
y3d2cji     cmake@3.23.1~doc+ncurses+ownlibs~qt build_type=Release
7ofkj5g         ncurses@6.2~symlinks+termlib abi=none
tgufarn             pkgconf@1.8.0
del3f6o         openssl@1.1.1o~docs~shared certs=system
xjse77e             perl@5.34.1+cpanm+shared+threads
bmmadng                 berkeley-db@18.1.40+cxx~docs+stl patches=b231fcc
ehf5627                 bzip2@1.0.8~debug~pic+shared
p5ebbuy                     diffutils@3.8
qnzwa5k                         libiconv@1.16 libs=shared,static
ovdf2pq                 gdbm@1.19
hse57kg                     readline@8.1
cvc7wjl                 zlib@1.2.12+optimize+pic+shared patches=0d38234
rxj536c     mpich@4.0.2~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~vci~verbs~wrapperrpath datatype-engine=auto device=ch4 netmod=ofi pmi=pmi
icrjbgb         findutils@4.9.0 patches=440b954
c3rwxom         hwloc@2.7.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared
iwj5wuv             libpciaccess@0.16
tueb4ra                 libtool@2.4.7
kcj4z36                     m4@1.4.19+sigsegv patches=9dc5fbd,bfdffa7
c4hxteu                         libsigsegv@2.13
f3hyg4w                 util-macros@1.19.3
x6fwxii             libxml2@2.9.13~python
22jgjfa                 xz@5.2.5+pic libs=shared,static
xwapafs         libfabric@1.14.1~debug~kdreg fabrics=rxm,sockets,tcp,udp
qaanq7e         yaksa@0.2~cuda~rocm
d33hbbr             autoconf@2.69 patches=35c4492,7793209,a49dd5b
kc2f75p             automake@1.16.5
eqvhpoq             python@3.8.13+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060
poaz426                 expat@2.4.8+libbsd
cv3ofog                     libbsd@0.11.5
deksstr                         libmd@1.0.4
rc7lfsx                 gettext@0.21+bzip2+curses+git~libunistring+libxml2+tar+xz
mkwelay                     tar@1.34 zip=pigz
5v6qbpo                         pigz@2.7
iz2z5mz                         zstd@1.5.2+programs compression=none libs=shared,static
dpfg22g                 libffi@3.4.2
nam6ocr                 sqlite@3.38.5+column_metadata+dynamic_extensions+fts~functions+rtree
j4isjhe                 util-linux-uuid@2.37.4

Spack Maintainers and Others:

@cwsmith @wspear

cwsmith commented 2 years ago

@SamBosch77 Thank you. Is the command to run spack post-install tests (to reproduce the failures) spack test ?

SamBosch77 commented 2 years ago

Yes, I believe that should reproduce the issue. I can't remember if there's a difference between spackTestRun <hash> and spack test run <hash>, but from my brief investigation they seem to yield the same output.

cwsmith commented 2 years ago

@SamBosch77 This draft PR to Spack fixes the issue for me: https://github.com/spack/spack/pull/32446

If it works for you I'll mark the PR as active/ready for review.

SamBosch77 commented 2 years ago

Looks like this fixed this issue on my end! Thanks!