search

multiphenics / multiphenicsx

multiphenicsx - easy prototyping of multiphysics problems in FEniCSx
https://multiphenics.github.io/
GNU Lesser General Public License v3.0
45 stars 9 forks source link

Conda install often crashes on import of DofMapRestriction #15

Closed hermanmakhm closed 1 year ago

hermanmakhm commented 1 year ago

I am using multiphenicsx with dolfinx via conda with python 3.11 and when I call from multiphenicsx.fem import DofMapRestriction I am often (but not always) presented with the following error. This happens only with multiple cores. I don't remember having this happen with one single process, but with parallel launches this happens very often but not consistently. Launching the same script can give this error some % of the time (it seems the more the processes the higher the % chance of seeing this error). Moreover this never happens with my spack install (but I have other reasons that force me to use conda for now).

  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/fenicsx_init.py", line 46, in <module>
    from multiphenicsx.fem import DofMapRestriction
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/fem/__init__.py", line 9, in <module>
    from multiphenicsx.fem.dofmap_restriction import DofMapRestriction
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/fem/dofmap_restriction.py", line 15, in <module>
    from multiphenicsx.cpp import cpp_library as mcpp
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/__init__.py", line 15, in <module>
    cpp_library = compile_package(
                  ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 85, in mpi_jit
    output = local_jit(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/compile_package.py", line 85, in compile_package
    return compile_code(package_name, package_file, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/compile_code.py", line 100, in compile_code
    return cppimport.imp(package_name_with_hash)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 50, in imp
    return imp_from_filepath(filepath, fullname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 88, in imp_from_filepath
    load_module(module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 104, in load_module
    _actually_load_module(module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 91, in _actually_load_module
    module_data["module"] = importlib.import_module(module_data["fullname"])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: /stck/hmmak/.cache/real-fenics/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN22multiphenicsx_wrappers2laERN8pybind117module_E
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.cpp -o /tmp/tmpq37xczez/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/real-fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.cpp -o /tmp/tmp_zjbwf_8/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.o -std=c++11 -fvisibility=hidden -std=c++20

There are also other INFO:root things mixed in the error but I'm not sure about the order of the error since there are so many processes printing these errors.

Moreover, actually since I started using my conda install instead of my spack install I have had some INFO:root things that used to appear that look like this whenever I switch between real and complex environments. I've "fixed" this myself by changing the directory jit.py in dolfinx points towards for cache to something unique for each environment.

INFO:root:running build_ext
INFO:root:building 'multiphenicsx_85a06d4525c3410c54a93869d159d912' extension
INFO:root:creating /tmp/tmp2yg5jzgi/stck
INFO:root:creating /tmp/tmp2yg5jzgi/stck/hmmak
INFO:root:creating /tmp/tmp2yg5jzgi/stck/hmmak/.cache
INFO:root:creating /tmp/tmp2yg5jzgi/stck/hmmak/.cache/fenics
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la
INFO:root:creating /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /stck/hmmak/.cache/fenics/.rendered.multiphenicsx_85a06d4525c3410c54a93869d159d912.cpp -o /tmp/tmp2yg5jzgi/stck/hmmak/.cache/fenics/.rendered.multiphenicsx_85a06d4525c3410c54a93869d159d912.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/petsc4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/stck/hmmak/.cache/fenics -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.cpp -o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.o -std=c++11 -fvisibility=hidden -std=c++20
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmp2yg5jzgi/stck/hmmak/.cache/fenics/.rendered.multiphenicsx_85a06d4525c3410c54a93869d159d912.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/DofMapRestriction.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/petsc.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/sparsitybuild.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/fem/utils.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/la/petsc.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/fem.o /tmp/tmp2yg5jzgi/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/multiphenicsx/wrappers/la.o -o /tmp/tmp2yg5jzgi/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmp2yg5jzgi/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so -> /stck/hmmak/.cache/fenics
hermanmakhm commented 1 year ago

After trying some more I realised I get not only that error but other times I get this traceback error as well

  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/fenicsx_init.py", line 46, in <module>
    from multiphenicsx.fem import DofMapRestriction
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/fem/__init__.py", line 9, in <module>
    from multiphenicsx.fem.dofmap_restriction import DofMapRestriction
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/fem/dofmap_restriction.py", line 15, in <module>
    from multiphenicsx.cpp import cpp_library as mcpp
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/__init__.py", line 15, in <module>
    cpp_library = compile_package(
                  ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 91, in mpi_jit
    raise RuntimeError(f"Failed just-in-time compilation of form: {error_msg}")
RuntimeError: Failed just-in-time compilation of form: Compilation failed on root node.

As well as one of the processes printing instead of the above runtime error.

RuntimeError: Failed just-in-time compilation of form: /stck/hmmak/.cache/real-fenics/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN22multiphenicsx_wrappers2laERN8pybind117module_E

Currently I'm not sure what triggers this error as opposed to the other error.

francesco-ballarin commented 1 year ago

RuntimeError: Failed just-in-time compilation of form: /stck/hmmak/.cache/real-fenics/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN22multiphenicsx_wrappers2laERN8pybind117module_E

This is error is certainly more helpful than the other.

Now that you have separated the cache for real and complex builds (I have a similar patch in jit.py myself) I suggest to start from a clean cache and verify if the error appears again. It surely is something specific of the conda environment, because on my system I have a python 3.11 installation and I do not have any random crash.

hermanmakhm commented 1 year ago

I have tried rm -rf with /.cache/ folder many times in the past month and both errors (in particular the first one that ends with return _bootstrap._gcd_import(name[level:], package, level)) occur often.

francesco-ballarin commented 1 year ago

I am not a conda user, so I need some help in setting up the conda environment up. Can you send me detailed instructions on how you created your environment?

hermanmakhm commented 1 year ago

I have a shell script that pip installs all the necessary packages of dolfinx and multiphenicsx after installation and activation of conda (which I installed with these instructions). Here it is:

# git URLs
GITURL_BASIX='https://github.com/FEniCS/basix'
GITURL_UFL='https://github.com/FEniCS/ufl'
GITURL_FFCX='https://github.com/FEniCS/ffcx'
GITURL_DOLFINX='https://github.com/FEniCS/dolfinx'
GITURL_MULTIPHENICSX='https://github.com/multiphenics/multiphenicsx'

# exit on error
set -e

# scratch path
SCRATCH_PATH=$HOME

# OPTIONS_______________________________________________________________________
echo
# with complex petscs/slepcs?
read -p ' - complex linear algebra? (y/n): ' complex
if [ $complex == y ]; then
    ENV_NAME=fx-cplx
else
    ENV_NAME=fx-real
fi

# default env location
ENV_PATH=$SCRATCH_PATH/conda/envs/$ENV_NAME

# default env location
BUILD_PATH=$SCRATCH_PATH/build-tmp

# flag to build silently
read -p ' - install silently [suppress conda confirmations]? (y/n): ' answer
if [ $answer == y ]; then
    CONDA_SILENT_FLAG='-y'
else
    CONDA_SILENT_FLAG=''
fi

# required packages
PACKAGES_GENERAL='cxx-compiler cmake python'
PACKAGES_BASIX='xtensor xtl numpy pybind11'
PACKAGES_DOLFINX='boost mpich hdf5=*=mpi*'
if [ $complex == y ]; then
    PACKAGES_DOLFINX=$PACKAGES_DOLFINX' petsc=*=*complex* slepc=*=*complex*'
else
    PACKAGES_DOLFINX=$PACKAGES_DOLFINX' petsc=*=*real* slepc=*=*real*'
fi
PACKAGES_DOLFINX=$PACKAGES_DOLFINX' mpi4py petsc4py slepc4py pyvista'

echo
echo 'CREATE CONDA ENVIRONMENT___________________________________________________'
conda create $CONDA_SILENT_FLAG --prefix $ENV_PATH

echo ' - activate conda environment'
conda activate $ENV_PATH

echo
echo 'INSTALL GENERAL PACKAGES___________________________________________________'
echo 'Required packages: '$PACKAGES_GENERAL
conda install -c conda-forge $CONDA_SILENT_FLAG $PACKAGES_GENERAL

echo
echo 'CREATE BUILD PATH__________________________________________________________'
echo $BUILD_PATH
mkdir -p $BUILD_PATH

echo
echo 'INSTALL BASIX______________________________________________________________'
echo 'Required packages: '$PACKAGES_BASIX
conda install -c conda-forge $CONDA_SILENT_FLAG $PACKAGES_BASIX
echo 'Get git repository: '$GITURL_BASIX
cd $BUILD_PATH
git clone $GITURL_BASIX basix
cd basix

echo 'Compile basix'
cd cpp
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$ENV_PATH -B build -S .
cmake --build build
cmake --install build
echo 'Install basix'
cd ../python
pip install .

echo
echo 'INSTALL UFL________________________________________________________________'
echo 'Get git repository: '$GITURL_UFL
cd $BUILD_PATH
git clone $GITURL_UFL ufl
cd ufl
# from https://fenics.readthedocs.io/projects/ufl/en/latest/installation.html
echo 'Install ufl'
pip install .

echo
echo 'INSTALL FFCX_______________________________________________________________'
echo 'Get git repository: '$GITURL_FFCX
cd $BUILD_PATH
git clone $GITURL_FFCX ffcx
cd ffcx
# form https://github.com/FEniCS/ffcx
echo 'Install ffcx'
pip install .

echo
echo 'INSTALL DOLFINX____________________________________________________________'
echo 'Required packages: '$PACKAGES_DOLFINX
conda install -c conda-forge $CONDA_SILENT_FLAG $PACKAGES_DOLFINX
echo 'Get git repository: '$GITURL_DOLFINX
cd $BUILD_PATH
git clone $GITURL_DOLFINX dolfinx
cd dolfinx
# from https://docs.fenicsproject.org/dolfinx/main/python/installation
echo 'Compile dolfinx'
cd cpp
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$ENV_PATH ../
make install
echo 'Install dolfinx'
cd ../../python
pip install .

echo
echo 'INSTALL MULTIPHENICSX______________________________________________________'
echo 'Get git repository: '$GITURL_MULTIPHENICSX
cd $BUILD_PATH
git clone $GITURL_MULTIPHENICSX multiphenicsx
cd multiphenicsx
# from https://github.com/multiphenics/multiphenicsx
echo 'Install multiphenicsx'
pip install .

# add PetschBinaryIO.py to path
conda-develop $ENV_PATH/lib/petsc/bin

echo
echo 'FINALISE SCRIPT____________________________________________________________'
echo 'Remove build directory'
rm -rf $BUILD_PATH
echo 'Check env space on disk'
du -h -d 0 $ENV_PATH

echo
echo 'END________________________________________________________________________'
echo
echo '#'
echo '# The environment has been succesfully created at'
echo '#'
echo '#    '$ENV_PATH
echo '#'
echo '# To properly load the environment use:'
echo '#'
echo '#     $ conda activate '$ENV_PATH
echo '#'
echo '# the first time you activate the environment'
echo '#'
echo '# To remove the environment:'
echo '#'
echo '#     $ conda env remove --prefix '$ENV_PATH
echo

I have also tried conda install -c conda-forge fenics-dolfinx mpich pyvista as recommended on the readme of https://github.com/FEniCS/dolfinx but the same errors occur.

francesco-ballarin commented 1 year ago

I cannot replicate in a clean ubuntu container created with docker run -it --rm ubuntu, and installing conda, dolfinx and multiphenicsx as follows

apt update
apt install wget vim git
wget https://repo.anaconda.com/archive/Anaconda3-2023.09-0-Linux-x86_64.sh
bash Anaconda3-2023.09-0-Linux-x86_64.sh 
eval "$(/root/anaconda3/bin/conda shell.bash hook)"
conda init
conda create -n fenicsx-env
conda activate fenicsx-env
conda install -c conda-forge fenics-dolfinx mpich pyvista
git clone https://github.com/multiphenics/multiphenicsx.git
cd multiphenicsx/
pip install .[tests,tutorials]
cd /tmp/
python3 -c 'import multiphenicsx.cpp'
# ... manually repeat the import 10 times, all of them pass
hermanmakhm commented 1 year ago

Unfortunately I can't use docker nor ubuntu (I'm on CentOS). Moreover, the error doesn't occur with serial running and rather only when I launch with mpirun with nodes>1. Moreover, the more processors I add the more the likelihood of the error.

francesco-ballarin commented 1 year ago

I am definitely not suggesting that you should use docker or change your OS! However, I need a way to reproduce the issue to investigate it, as until now it seems to be specific to your system.

francesco-ballarin commented 1 year ago

only when I launch with mpirun with nodes>1. Moreover, the more processors I add the more the likelihood of the error.

For reference, I now did try importing multiphenicsx.cpp under mpirun using the same container as in https://github.com/multiphenics/multiphenicsx/issues/15#issuecomment-1777040223, even with

for i in $(seq 1 30); do echo $i; mpirun -n $i python3 -c 'import multiphenicsx.cpp' || break ; done; echo $?

which massively oversubscribes the number of cores I have on my machine. Still, compilation and import give no error in those 30 runs.

hermanmakhm commented 1 year ago

So what I just tried is removing all traces of multiphenicsx from my .cache and running mpirun -n 1 python -c 'import multiphenicsx.cpp' with successively increasing number of processors by replacing 1 manually with [1, 10, 20, 100, 150, 200, 220]

At -n 1 the INFO:root stuff appeared and then did not appear again until -n 200 but there were no errors. At -n 220 there was the original error and afterwards I tried again and the same error even when I do mpirun -n 1 python -c 'import multiphenicsx.cpp' until I delete my .cache again.

Edit: I should note this is on an ssh to an HPC which is why I'm able to subscribe to that many nodes.

francesco-ballarin commented 1 year ago

What happens if you run the following test.py file

import pathlib
import sys
import os
import tempfile

import cppimport

import dolfinx
import dolfinx.pkgconfig
from dolfinx import wrappers
from dolfinx.jit import mpi_jit_decorator
from dolfinx.mesh import create_unit_square

import mpi4py
from mpi4py import MPI

def test_mpi_comm_wrapper_cppimport(dir):  # noqa: F811
    """Test MPICommWrapper <-> mpi4py.MPI.Comm conversion for code compiled with cppimport"""

    dolfinx_pc = dolfinx.pkgconfig.parse("dolfinx")

    @mpi_jit_decorator
    def compile_module():
        cpp_code_header = f"""
/*
<%
setup_pybind11(cfg)
cfg['compiler_args'] = ['-std=c++20']
cfg['include_dirs'] += {dolfinx_pc["include_dirs"]
                        + [mpi4py.get_include()]
                        + [str(wrappers.get_include_path())]}
%>
*/
"""

        cpp_code = """
#include <pybind11/pybind11.h>
#include <caster_mpi.h>

dolfinx_wrappers::MPICommWrapper
test_comm_passing(const dolfinx_wrappers::MPICommWrapper comm)
{
    MPI_Comm c = comm.get();
    return dolfinx_wrappers::MPICommWrapper(c);
}

PYBIND11_MODULE(mpi_comm_wrapper, m)
{
    m.def("test_comm_passing", &test_comm_passing);
}
"""
        path = pathlib.Path(dir)
        open(pathlib.Path(dir, "mpi_comm_wrapper.cpp"), "w").write(cpp_code + cpp_code_header)
        rel_path = path.relative_to(pathlib.Path(__file__).parent)
        p = str(rel_path).replace("/", ".") + ".mpi_comm_wrapper"
        return cppimport.imp(p)

    module = compile_module(MPI.COMM_WORLD)

    w1 = MPI.COMM_WORLD
    w2 = module.test_comm_passing(w1)

    assert isinstance(w1, MPI.Comm)
    assert isinstance(w2, MPI.Comm)
    assert w1 == w2

assert len(sys.argv) == 2
if sys.argv[1] == "fixed":
    path = os.path.join(os.getcwd(), "a_fixed_directory")
    if MPI.COMM_WORLD.rank == 0:
        print("Using fixed directory")
        if not os.path.exists(path):
            os.mkdir(path)
    test_mpi_comm_wrapper_cppimport(path)
else:
    if MPI.COMM_WORLD.rank == 0:
        path = tempfile.mkdtemp(dir=os.getcwd())
        print("Using temporary directory", path)
    else:
        path = ""
    path = MPI.COMM_WORLD.bcast(path, 0)
    test_mpi_comm_wrapper_cppimport(path)

as follows:

python3 test.py fixed
mpirun -n 220 python3 test.py fixed
python3 test.py temp
mpirun -n 220 python3 test.py temp

This test is a slightly modified version of https://github.com/FEniCS/dolfinx/blob/main/python/test/unit/common/test_mpi.py

francesco-ballarin commented 1 year ago

As a temporary workaround, you may want to try this, starting from a clean cache

python3 -c 'import multiphenicsx.cpp'

export DOLFINX_CACHE=$(python3 -c 'import dolfinx.jit; print(dolfinx.jit.get_options()["cache_dir"])')
MULTIPHENICSX_SO=$(echo $DOLFINX_CACHE/multiphenicsx_*.so)
echo $MULTIPHENICSX_SO

MULTIPHENICSX_CPP_INIT=$(python3 -c 'import multiphenicsx.cpp; import os; print(os.path.join(os.path.dirname(multiphenicsx.__file__), "cpp", "__init__.py"))')
echo $MULTIPHENICSX_CPP_INIT

cat > $MULTIPHENICSX_CPP_INIT << EOL
import importlib
import sys
import os

sys.path.append("$DOLFINX_CACHE")
cpp_library = importlib.import_module(os.path.basename("$MULTIPHENICSX_SO").split('.')[0])
sys.path.remove("$DOLFINX_CACHE")
EOL

python3 -c 'import multiphenicsx.cpp'

With the first line, we compile the multiphenicsx c++ part. The remaining lines are used to hardcode the path of the generated shared library inside the multiphenicsx/cpp/__init__.py file. Notice that you will loose the ability to recompile again the shared file (i.e., if you happen to delete the cache, you will have to go back to the original multiphenicsx/cpp/__init__.py file).

If this workaround enables you to work with 220 cores, I will close this. I am not willing to put much more effort in this because this part will need to be rewritten anyways (in particular, dropping cppimport, which seems to give you problems) after https://github.com/FEniCS/dolfinx/pull/2820 will get merged upstream.

hermanmakhm commented 1 year ago

So far I've only tested test.py. I will try the workaround. Here is the output:

(fx-real) [hmmak@sator5 EL8 /tmp_user/sator/hmmak/fenicsx-scripts/3D]$ python3 test.py fixed
Using fixed directory
INFO:root:running build_ext
INFO:root:building 'a_fixed_directory.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmpvr0s4u2n/tmp_user
INFO:root:creating /tmp/tmpvr0s4u2n/tmp_user/sator
INFO:root:creating /tmp/tmpvr0s4u2n/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmpvr0s4u2n/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpvr0s4u2n/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpvr0s4u2n/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpvr0s4u2n/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.o -std=c++20
INFO:root:creating /tmp/tmpvr0s4u2n/a_fixed_directory
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmpvr0s4u2n/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.o -o /tmp/tmpvr0s4u2n/a_fixed_directory/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmpvr0s4u2n/a_fixed_directory/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory
(fx-real) [hmmak@sator5 EL8 /tmp_user/sator/hmmak/fenicsx-scripts/3D]$ mpirun -n 220 python3 test.py fixed
Using fixed directory
(fx-real) [hmmak@sator5 EL8 /tmp_user/sator/hmmak/fenicsx-scripts/3D]$ python3 test.py temp
Using temporary directory /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf
INFO:root:running build_ext
INFO:root:building 'tmp496pjsdf.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmpmkb8dk6g/tmp_user
INFO:root:creating /tmp/tmpmkb8dk6g/tmp_user/sator
INFO:root:creating /tmp/tmpmkb8dk6g/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmpmkb8dk6g/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpmkb8dk6g/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpmkb8dk6g/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpmkb8dk6g/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf/.rendered.mpi_comm_wrapper.o -std=c++20
INFO:root:creating /tmp/tmpmkb8dk6g/tmp496pjsdf
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmpmkb8dk6g/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf/.rendered.mpi_comm_wrapper.o -o /tmp/tmpmkb8dk6g/tmp496pjsdf/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmpmkb8dk6g/tmp496pjsdf/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp496pjsdf
(fx-real) [hmmak@sator5 EL8 /tmp_user/sator/hmmak/fenicsx-scripts/3D]$ mpirun -n 220 python3 test.py temp
INFO:root:running build_ext
INFO:root:building 'tmp_j59l56h.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmp5pkmywt9/tmp_user
INFO:root:creating /tmp/tmp5pkmywt9/tmp_user/sator
INFO:root:creating /tmp/tmp5pkmywt9/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmp5pkmywt9/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmp5pkmywt9/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmp5pkmywt9/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmp5pkmywt9/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h/.rendered.mpi_comm_wrapper.o -std=c++20
INFO:root:creating /tmp/tmp5pkmywt9/tmp_j59l56h
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmp5pkmywt9/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h/.rendered.mpi_comm_wrapper.o -o /tmp/tmp5pkmywt9/tmp_j59l56h/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmp5pkmywt9/tmp_j59l56h/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h
Using temporary directory /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp_j59l56h

I then ran mpirun -n 220 python -c "import multiphenicsx.cpp" just to check and the first error that ends in

File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/compile_package.py", line 85, in compile_package
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dynamic module does not define module export function (PyInit_multiphenicsx_85a06d4525c3410c54a93869d159d912)

occurred again.

I also tried submitting it as an sbatch shell script but I get 

Using fixed directory
Using fixed directory
INFO:root:running build_ext
INFO:root:running build_ext
INFO:root:building 'a_fixed_directory.mpi_comm_wrapper' extension
INFO:root:building 'a_fixed_directory.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmpjpu_3tcv/tmp_user
INFO:root:creating /tmp/tmpjbr5px2d/tmp_user
INFO:root:creating /tmp/tmpjpu_3tcv/tmp_user/sator
INFO:root:creating /tmp/tmpjbr5px2d/tmp_user/sator
INFO:root:creating /tmp/tmpjpu_3tcv/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmpjpu_3tcv/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpjpu_3tcv/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpjpu_3tcv/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory
INFO:root:creating /tmp/tmpjbr5px2d/tmp_user/sator/hmmak
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpjpu_3tcv/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.o
INFO:root:creating /tmp/tmpjbr5px2d/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpjbr5px2d/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpjbr5px2d/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpjbr5px2d/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.o
INFO:root:creating /tmp/tmpjpu_3tcv/a_fixed_directory
INFO:root:creating /tmp/tmpjbr5px2d/a_fixed_directory
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmpjbr5px2d/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.o -o /tmp/tmpjbr5px2d/a_fixed_directory/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmpjpu_3tcv/tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory/.rendered.mpi_comm_wrapper.o -o /tmp/tmpjpu_3tcv/a_fixed_directory/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmpjpu_3tcv/a_fixed_directory/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory
INFO:root:copying /tmp/tmpjbr5px2d/a_fixed_directory/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/a_fixed_directory
INFO:cppimport.checksum:Failed to find compiled extension; rebuilding.
Traceback (most recent call last):
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 74, in <module>
    test_mpi_comm_wrapper_cppimport(path)
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 58, in test_mpi_comm_wrapper_cppimport
    module = compile_module(MPI.COMM_WORLD)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 85, in mpi_jit
    output = local_jit(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 56, in compile_module
    return cppimport.imp(p)
           ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 50, in imp
    return imp_from_filepath(filepath, fullname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 87, in imp_from_filepath
    build_safely(filepath, module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 52, in build_safely
    raise Exception(
Exception: Could not compile binary as lock already taken and timed out. Try increasing the timeout setting if the build time is longer (pid 275883).
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 88, in imp_from_filepath
    load_module(module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 104, in load_module
    _actually_load_module(module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 91, in _actually_load_module
    module_data["module"] = importlib.import_module(module_data["fullname"])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: dynamic module does not define module export function (PyInit_mpi_comm_wrapper)

for fixed and for temp

Using temporary directory /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmpsztf87_8
INFO:root:running build_ext
INFO:root:building 'tmp2gk04k0w.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmpniu6jdcf/tmp_user
INFO:root:creating /tmp/tmpniu6jdcf/tmp_user/sator
INFO:root:creating /tmp/tmpniu6jdcf/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmpniu6jdcf/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpniu6jdcf/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpniu6jdcf/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpniu6jdcf/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.o -std=c++20
INFO:root:creating /tmp/tmpniu6jdcf/tmp2gk04k0w
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmpniu6jdcf/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.o -o /tmp/tmpniu6jdcf/tmp2gk04k0w/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmpniu6jdcf/tmp2gk04k0w/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
Using temporary directory /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
INFO:root:running build_ext
INFO:root:building 'tmp2gk04k0w.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmpyhgg7z95/tmp_user
INFO:root:creating /tmp/tmpyhgg7z95/tmp_user/sator
INFO:root:creating /tmp/tmpyhgg7z95/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmpyhgg7z95/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpyhgg7z95/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpyhgg7z95/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpyhgg7z95/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.o
Traceback (most recent call last):
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 82, in <module>
    test_mpi_comm_wrapper_cppimport(path)
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 58, in test_mpi_comm_wrapper_cppimport
    module = compile_module(MPI.COMM_WORLD)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 85, in mpi_jit
    output = local_jit(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 56, in compile_module
    return cppimport.imp(p)
           ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 50, in imp
    return imp_from_filepath(filepath, fullname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 87, in imp_from_filepath
    build_safely(filepath, module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 52, in build_safely
    raise Exception(
Exception: Could not compile binary as lock already taken and timed out. Try increasing the timeout setting if the build time is longer (pid 3202884).
Traceback (most recent call last):
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 82, in <module>
    test_mpi_comm_wrapper_cppimport(path)
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 58, in test_mpi_comm_wrapper_cppimport
    module = compile_module(MPI.COMM_WORLD)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 85, in mpi_jit
    output = local_jit(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 56, in compile_module
    return cppimport.imp(p)
           ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 50, in imp
    return imp_from_filepath(filepath, fullname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 87, in imp_from_filepath
    build_safely(filepath, module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 52, in build_safely
    raise Exception(
Exception: Could not compile binary as lock already taken and timed out. Try increasing the timeout setting if the build time is longer (pid 3202877).

INFO:root:running build_ext
INFO:root:building 'tmp2gk04k0w.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmp_jnmb_b8/tmp_user
INFO:root:creating /tmp/tmp_jnmb_b8/tmp_user/sator
INFO:root:creating /tmp/tmp_jnmb_b8/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmp_jnmb_b8/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmp_jnmb_b8/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmp_jnmb_b8/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/pybind11/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/mpi4py/include -I/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/wrappers -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmp_jnmb_b8/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.o -std=c++20
INFO:root:running build_ext
INFO:root:building 'tmp2gk04k0w.mpi_comm_wrapper' extension
INFO:root:creating /tmp/tmpfwbfe_51/tmp_user
INFO:root:creating /tmp/tmpfwbfe_51/tmp_user/sator
INFO:root:creating /tmp/tmpfwbfe_51/tmp_user/sator/hmmak
INFO:root:creating /tmp/tmpfwbfe_51/tmp_user/sator/hmmak/fenicsx-scripts
INFO:root:creating /tmp/tmpfwbfe_51/tmp_user/sator/hmmak/fenicsx-scripts/3D
INFO:root:creating /tmp/tmpfwbfe_51/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -fPIC -I/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w -I/tmp_user/sator/hmmak/conda/envs/fx-real/include/python3.11 -c /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.cpp -o /tmp/tmpfwbfe_51/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.o
INFO:root:creating /tmp/tmpfwbfe_51/tmp2gk04k0w
INFO:root:/tmp_user/sator/hmmak/conda/envs/fx-real/bin/x86_64-conda-linux-gnu-c++ -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -Wl,-rpath-link,/tmp_user/sator/hmmak/conda/envs/fx-real/lib -L/tmp_user/sator/hmmak/conda/envs/fx-real/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp_user/sator/hmmak/conda/envs/fx-real/include /tmp/tmpfwbfe_51/tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/.rendered.mpi_comm_wrapper.o -o /tmp/tmpfwbfe_51/tmp2gk04k0w/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so
INFO:root:copying /tmp/tmpfwbfe_51/tmp2gk04k0w/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so -> /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w
Traceback (most recent call last):
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 82, in <module>
    test_mpi_comm_wrapper_cppimport(path)
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 58, in test_mpi_comm_wrapper_cppimport
    module = compile_module(MPI.COMM_WORLD)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 85, in mpi_jit
    output = local_jit(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 56, in compile_module
    return cppimport.imp(p)
           ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 50, in imp
    return imp_from_filepath(filepath, fullname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 88, in imp_from_filepath
    load_module(module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 104, in load_module
    _actually_load_module(module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 91, in _actually_load_module
    module_data["module"] = importlib.import_module(module_data["fullname"])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /tmp_user/sator/hmmak/fenicsx-scripts/3D/tmp2gk04k0w/mpi_comm_wrapper.cpython-311-x86_64-linux-gnu.so: invalid ELF header
Traceback (most recent call last):
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 82, in <module>
    test_mpi_comm_wrapper_cppimport(path)
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 58, in test_mpi_comm_wrapper_cppimport
    module = compile_module(MPI.COMM_WORLD)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/dolfinx/jit.py", line 85, in mpi_jit
    output = local_jit(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/fenicsx-scripts/3D/test.py", line 56, in compile_module
    return cppimport.imp(p)
           ^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 50, in imp
    return imp_from_filepath(filepath, fullname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/__init__.py", line 87, in imp_from_filepath
    build_safely(filepath, module_data)
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/cppimport/importer.py", line 52, in build_safely
    raise Exception(
Exception: Could not compile binary as lock already taken and timed out. Try increasing the timeout setting if the build time is longer (pid 3202878).
hermanmakhm commented 1 year ago

I think I might have done something wrong with your workaround as now running python3 -c 'import multiphenicsx.cpp' yields

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/site-packages/multiphenicsx/cpp/__init__.py", line 6, in <module>
    cpp_library = importlib.import_module(os.path.basename("/stck/hmmak/.cache/real-fenics/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so").split('.')[0])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp_user/sator/hmmak/conda/envs/fx-real/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: /stck/hmmak/.cache/real-fenics/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZNK7dolfinx6common8IndexMap10size_localEv
francesco-ballarin commented 1 year ago

https://github.com/multiphenics/multiphenicsx/issues/15#issuecomment-1781167460 tells me that this is a cppimport issue, most likely when it interfaces with the filesystem

https://github.com/multiphenics/multiphenicsx/issues/15#issuecomment-1781211840 makes me worry that you have several versions of dolfinx around. Please try:

hermanmakhm commented 1 year ago

ldd gives me

ldd /stck/hmmak/.cache/real-fenics/multiphenicsx_85a06d4525c3410c54a93869d159d912.cpython-311-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffdea2af000)
    libstdc++.so.6 => /tmp_user/sator/hmmak/conda/envs/fx-real/lib/libstdc++.so.6 (0x000014ab0054b000)
    libgcc_s.so.1 => /tmp_user/sator/hmmak/conda/envs/fx-real/lib/libgcc_s.so.1 (0x000014ab008d5000)
    libc.so.6 => /lib64/libc.so.6 (0x000014ab00188000)
    libm.so.6 => /lib64/libm.so.6 (0x000014aaffe06000)
    librt.so.1 => /lib64/librt.so.1 (0x000014aaffbfe000)
    /lib64/ld-linux-x86-64.so.2 (0x000014ab0072e000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x000014aaff9de000)

I'm not sure how to interpret this.

However adding import dolfinx.cpp seems to have fixed the workaround. I have been consistently able to run at >220 processors since that addition to the workaround. I think this can be closed; if further testing indicates otherwise I will raise a new issue.

francesco-ballarin commented 1 year ago

Great!