search

conda-forge / fenics-feedstock

A conda-smithy repository for fenics.
BSD 3-Clause "New" or "Revised" License
13 stars 12 forks source link

Issue with scotch v7 ordering of real functionspace in parallel #209

Open sblauth opened 3 months ago

sblauth commented 3 months ago

Solution to issue cannot be found in the documentation.

Issue

Hi everyone,

I found an issue with the graph re-ordering using scotch version 7 in parallel. Consider the following MWE

from fenics import *

mesh = UnitSquareMesh(8, 8)
R = FunctionSpace(mesh, "R", 0)

assert True

The script runs fine in serial, but when running this in paralell, I receive the following error

*** -------------------------------------------------------------------------
*** DOLFIN encountered an error. If you are not able to resolve this issue
*** using the information listed below, you can ask for help at
***
***     fenics-support@googlegroups.com
***
*** Remember to include the error message listed below and, if possible,
*** include a *minimal* running example to reproduce the error.
***
*** -------------------------------------------------------------------------
*** Error:   Unable to re-order graph using SCOTCH.
*** Reason:  Error during re-ordering.
*** Where:   This error was encountered inside SCOTCH.cpp.
*** Process: 0
*** 
*** DOLFIN version: 2019.1.0
*** Git changeset:  ea4ec9db9a0ab5757694c6a6ff308b135231a4dd
*** -------------------------------------------------------------------------

This behavior occurs both with MPICH and OpenMPI. Moreover, it seems to be related to scotch, as using

parameters["dof_ordering_library"] = "Boost"

before defining the mesh fixes the problem. With scotch v6.0.9 (the older builds), everything worked fine.

I have tested this problem with all four builds that I have given in #208, only the ones using scotch 7 (or libscotch 7) are the problematic ones. Below, I will only attach one problematic build.

Installed packages

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h69a702a_16    conda-forge
alsa-lib                  1.2.12               h4ab18f5_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
binutils_impl_linux-64    2.40                 ha1999f0_7    conda-forge
binutils_linux-64         2.40                 hb3c18ed_9    conda-forge
blosc                     1.21.6               hef167b5_0    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.32.1               h4bc722e_0    conda-forge
ca-certificates           2024.7.4             hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.18.0               hbb29018_2    conda-forge
certifi                   2024.7.4           pyhd8ed1ab_0    conda-forge
cftime                    1.6.4           py311h18e1886_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.1           py311h9547e67_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
eigen                     3.4.0                h00ab1b0_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h59595ed_0    conda-forge
fenics                    2019.1.0            h8783aee_48    conda-forge
fenics-dijitso            2019.1.0          pyhd8ed1ab_39    conda-forge
fenics-dolfin             2019.1.0        py311h57ea513_48    conda-forge
fenics-ffc                2019.1.0          pyhd8ed1ab_39    conda-forge
fenics-fiat               2019.1.0          pyhd8ed1ab_39    conda-forge
fenics-libdolfin          2019.1.0            h894a989_48    conda-forge
fenics-ufl                2019.1.0        py311h38be061_38    conda-forge
fftw                      3.3.10          mpi_openmpi_h4a81ba8_9    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.53.1          py311h61187de_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
gcc                       12.3.0              h915e2ae_13    conda-forge
gcc_impl_linux-64         12.3.0              h58ffeeb_13    conda-forge
gcc_linux-64              12.3.0               h9528a6a_9    conda-forge
gettext                   0.22.5               h59595ed_2    conda-forge
gettext-tools             0.22.5               h59595ed_2    conda-forge
glib                      2.80.3               h8a4344b_1    conda-forge
glib-tools                2.80.3               h73ef956_1    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
gmpy2                     2.1.5           py311hc4f1f91_1    conda-forge
graphite2                 1.3.13            h59595ed_1003    conda-forge
gst-plugins-base          1.24.5               hbaaba92_0    conda-forge
gstreamer                 1.24.5               haf2f30d_0    conda-forge
gxx                       12.3.0              h915e2ae_13    conda-forge
gxx_impl_linux-64         12.3.0              h2a574ab_13    conda-forge
gxx_linux-64              12.3.0               ha28b414_9    conda-forge
h5py                      3.11.0          nompi_py311h439e445_102    conda-forge
harfbuzz                  8.5.0                hfac3d4d_0    conda-forge
hdf4                      4.2.15               h2a13503_7    conda-forge
hdf5                      1.14.3          mpi_openmpi_hf5a2fd3_4    conda-forge
hypre                     2.31.0          mpi_openmpi_h900e7ac_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
importlib-metadata        8.0.0              pyha770c72_0    conda-forge
importlib_metadata        8.0.0                hd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipython                   8.26.0             pyh707e725_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
kernel-headers_linux-64   3.10.0              h4a8ded7_16    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py311h9547e67_1    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libasprintf               0.22.5               h661eb56_2    conda-forge
libasprintf-devel         0.22.5               h661eb56_2    conda-forge
libblas                   3.9.0           22_linux64_openblas    conda-forge
libboost                  1.84.0               hba137d9_3    conda-forge
libboost-headers          1.85.0               ha770c72_2    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcap                    2.69                 h0f662aa_0    conda-forge
libcblas                  3.9.0           22_linux64_openblas    conda-forge
libclang-cpp15            15.0.7          default_h127d8a8_5    conda-forge
libclang13                18.1.8          default_h6ae225f_0    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.8.0                hca28451_1    conda-forge
libdeflate                1.20                 hd590300_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-devel_linux-64     12.3.0             h6b66f73_113    conda-forge
libgcc-ng                 14.1.0               h77fa898_0    conda-forge
libgcrypt                 1.11.0               h4ab18f5_0    conda-forge
libgettextpo              0.22.5               h59595ed_2    conda-forge
libgettextpo-devel        0.22.5               h59595ed_2    conda-forge
libgfortran-ng            14.1.0               h69a702a_0    conda-forge
libgfortran5              14.1.0               hc5f4f2c_0    conda-forge
libglib                   2.80.3               h8a4344b_1    conda-forge
libgomp                   14.1.0               h77fa898_0    conda-forge
libgpg-error              1.50                 h4f305b6_0    conda-forge
libhwloc                  2.11.1          default_hecaa2ac_1000    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           22_linux64_openblas    conda-forge
libllvm15                 15.0.7               hb3ce162_4    conda-forge
libllvm18                 18.1.8               hc9dba70_0    conda-forge
libnetcdf                 4.9.2           nompi_h135f659_114    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libogg                    1.3.5                h4ab18f5_0    conda-forge
libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libpq                     16.3                 ha72fbe1_0    conda-forge
libptscotch               7.0.4                h4eaa18c_2    conda-forge
libsanitizer              12.3.0              hb8811af_13    conda-forge
libscotch                 7.0.4                h0c89965_2    conda-forge
libsndfile                1.2.2                hc60ed4a_1    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-devel_linux-64  12.3.0             h6b66f73_113    conda-forge
libstdcxx-ng              14.1.0               hc0a3c3a_0    conda-forge
libsystemd0               255                  h3516f8a_1    conda-forge
libtiff                   4.6.0                h1dd3fc0_3    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.16                 hd590300_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxkbcommon              1.7.0                h2c5496b_1    conda-forge
libxml2                   2.12.7               h4c95cb1_3    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
matplotlib                3.9.1           py311h38be061_0    conda-forge
matplotlib-base           3.9.1           py311hffb96ce_0    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
meshio                    5.3.5              pyhd8ed1ab_0    conda-forge
metis                     5.1.0             h59595ed_1007    conda-forge
mpc                       1.3.1                hfe3b2da_0    conda-forge
mpfr                      4.2.1                h9458935_1    conda-forge
mpg123                    1.32.6               h59595ed_0    conda-forge
mpi                       1.0                     openmpi    conda-forge
mpi4py                    3.1.6           py311h4267d7f_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
mumps-include             5.7.1                ha770c72_0    conda-forge
mumps-mpi                 5.7.1                hcb8ae40_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.3.0                hf1915f5_4    conda-forge
mysql-libs                8.3.0                hca2cd23_4    conda-forge
ncurses                   6.5                  h59595ed_0    conda-forge
netcdf4                   1.7.1           nompi_py311h25b3b55_101    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.102                h593d115_0    conda-forge
numpy                     1.26.4          py311h64a7726_0    conda-forge
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openmpi                   4.1.6              hc5af2df_101    conda-forge
openssl                   3.3.1                h4ab18f5_1    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
parmetis                  4.0.3             he9a3056_1005    conda-forge
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
pcre2                     10.44                h0f59acf_0    conda-forge
petsc                     3.21.2          real_hd0ea2e5_100    conda-forge
petsc4py                  3.21.2          py311he6cfc66_0    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.4.0          py311h82a398c_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pixman                    0.43.2               h59595ed_0    conda-forge
pkg-config                0.29.2            h4bc722e_1009    conda-forge
pkgconfig                 1.5.5              pyhd8ed1ab_4    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
ply                       3.11               pyhd8ed1ab_2    conda-forge
prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptscotch                  7.0.4                hd53efc5_2    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio-client         17.0                 hb77b528_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pybind11                  2.12.0          py311h9547e67_0    conda-forge
pybind11-global           2.12.0          py311h9547e67_0    conda-forge
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.9          py311hf0fb5b6_5    conda-forge
pyqt5-sip                 12.12.2         py311hb755f60_5    conda-forge
pytest                    8.2.2              pyhd8ed1ab_0    conda-forge
python                    3.11.9          hb806964_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
qhull                     2020.2               h434a139_5    conda-forge
qt-main                   5.15.8              ha2b5568_22    conda-forge
readline                  8.2                  h8228510_1    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
scalapack                 2.2.0                h67de57e_1    conda-forge
scipy                     1.14.0          py311h517d4fd_1    conda-forge
scotch                    7.0.4                hd53efc5_2    conda-forge
setuptools                70.3.0             pyhd8ed1ab_0    conda-forge
sip                       6.7.12          py311hb755f60_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slepc                     3.21.1          real_h4e5f841_301    conda-forge
slepc4py                  3.21.1          py311hc1ef5fc_101    conda-forge
snappy                    1.2.1                ha2e4443_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
suitesparse               7.7.0                hf4753ba_1    conda-forge
superlu                   5.2.2                h00795ac_0    conda-forge
superlu_dist              8.2.1                h511944b_1    conda-forge
sympy                     1.12.1          pypyh2585a3b_103    conda-forge
sysroot_linux-64          2.17                h4a8ded7_16    conda-forge
tbb                       2021.12.0            h434a139_3    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.4.1           py311h331c9d8_0    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
xcb-util                  0.4.1                hb711507_2    conda-forge
xcb-util-image            0.4.0                hb711507_2    conda-forge
xcb-util-keysyms          0.4.1                hb711507_0    conda-forge
xcb-util-renderutil       0.3.10               hb711507_0    conda-forge
xcb-util-wm               0.4.2                hb711507_0    conda-forge
xkeyboard-config          2.42                 h4ab18f5_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.9                hb711507_1    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xf86vidmodeproto     2.3.1             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.19.2             pyhd8ed1ab_0    conda-forge
zlib                      1.3.1                h4ab18f5_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Environment info

active environment : openmpinew
    active env location : /p/tv/local/miniconda3_blauths/envs/openmpinew
            shell level : 2
       user config file : /u/b/blauths/.condarc
 populated config files : /u/b/blauths/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.8.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.4=0
                          __glibc=2.34=0
                          __linux=5.14.0=0
                          __unix=0=0
       base environment : /p/tv/local/miniconda3_blauths  (writable)
      conda av data dir : /p/tv/local/miniconda3_blauths/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /p/tv/local/miniconda3_blauths/pkgs
                          /u/b/blauths/.conda/pkgs
       envs directories : /p/tv/local/miniconda3_blauths/envs
                          /u/b/blauths/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.8 Linux/5.14.0-362.24.1.el9_3.x86_64 rhel/9.4 glibc/2.34
                UID:GID : 132041:513
             netrc file : None
           offline mode : False