Segmentation fault (core dumped)

[Kerro-junior]

Test Error

 $ pytest --cov==gninatorch
Segmentation fault (core dumped)

System Details

CentOS7 Linux server Anaconda environment (python3.9)

I have created this environment and install all the libraries manually.

(It takes hours in the 'solving' process when running conda env create -f devtools/conda-envs/gninatorch.yaml, so I quit and created a environment 'gninatorch' and install libraries one by one as the yml shows)

After all dependencies have been installed (I additionally installed scikit-learn-intelex as the scikit-learn suggests), I run python -m pip install . the gninatorch 0.0.2+3.ge46cb51 successfully installed. BUT It could not success at $ pytest --cov==gninatorch.

-------------conda list---------------

_libgcc_mutex             0.1                        main    anaconda
_openmp_mutex             5.1                       1_gnu    anaconda
abseil-cpp                20211102.0           h27087fc_1    conda-forge
alembic                   1.8.1              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyhd3eb1b0_0    anaconda
arrow-cpp                 8.0.0            py39h60b952e_0    anaconda
asttokens                 2.0.5              pyhd3eb1b0_0    anaconda
attrs                     22.1.0           py39h06a4308_0    anaconda
aws-c-common              0.4.57               he6710b0_1    anaconda
aws-c-event-stream        0.1.6                h2531618_5    anaconda
aws-checksums             0.1.9                he6710b0_0    anaconda
aws-sdk-cpp               1.8.185              hce553d0_0    anaconda
backcall                  0.2.0              pyhd3eb1b0_0    anaconda
bcrypt                    3.2.2            py39hb9d737c_0    conda-forge
black                     22.6.0           py39h06a4308_0    anaconda
blas                      1.0                         mkl    anaconda
blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
boost-cpp                 1.78.0               he72f1d9_0    conda-forge
bottleneck                1.3.5            py39h7deecbd_0    anaconda
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py39h27cfd23_1003    anaconda
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f8727e_0    anaconda
ca-certificates           2023.01.10           h06a4308_0    anaconda
certifi                   2022.12.7        py39h06a4308_0    anaconda
cffi                      1.15.1           py39h5eee18b_3    anaconda
cfgv                      3.3.1            py39h06a4308_0    anaconda
charset-normalizer        2.0.4              pyhd3eb1b0_0    anaconda
click                     8.1.6           unix_pyh707e725_0    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
codecov                   2.1.11             pyhd3eb1b0_0    anaconda
configparser              5.3.0              pyhd8ed1ab_0    conda-forge
coverage                  6.3.2            py39h7f8727e_0    anaconda
cryptography              38.0.4           py39h9ce1e76_0    anaconda
cudatoolkit               11.3.1               h2bc3f7f_2    anaconda
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
daal4py                   2023.1.1         py39h79cecc1_0    defaults
dal                       2023.1.1         hdb19cb5_48679    defaults
databricks-cli            0.17.7             pyhd8ed1ab_1    conda-forge
decorator                 5.1.1              pyhd3eb1b0_0    anaconda
distlib                   0.3.6            py39h06a4308_0    anaconda
docker-py                 6.1.3              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.2                    pypi_0    pypi
executing                 0.8.3              pyhd3eb1b0_0    anaconda
fftw                      3.3.9                h27cfd23_1    anaconda
filelock                  3.9.0            py39h06a4308_0    anaconda
flake8                    6.0.0            py39h06a4308_0    anaconda
flask                     2.3.2              pyhd8ed1ab_0    conda-forge
flit-core                 3.6.0              pyhd3eb1b0_0    anaconda
freetype                  2.12.1               h4a9f257_0    anaconda
gflags                    2.2.2             he1b5a44_1004    conda-forge
giflib                    5.2.1                h5eee18b_1    anaconda
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.32             pyhd8ed1ab_0    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
gninatorch                0.0.2+3.ge46cb51          pypi_0    pypi
greenlet                  2.0.1            py39h6a678d5_0    anaconda
grpc-cpp                  1.46.1               h33aed49_1    anaconda
gunicorn                  20.1.0           py39hf3d152e_3    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
identify                  2.5.5            py39h06a4308_0    anaconda
idna                      3.4              py39h06a4308_0    anaconda
ignite                    0.4.12                     py_0    pytorch
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_resources       6.0.0              pyhd8ed1ab_1    conda-forge
iniconfig                 2.0.0                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561    anaconda
ipython                   8.8.0            py39h06a4308_0    anaconda
isort                     5.9.3              pyhd3eb1b0_0    anaconda
itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
jedi                      0.18.1           py39h06a4308_1    anaconda
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.0              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h7f8727e_0    anaconda
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py39h6a678d5_0    anaconda
krb5                      1.19.3               h3790be6_0    conda-forge
lcms2                     2.12                 h3be6417_0    anaconda
ld_impl_linux-64          2.38                 h1181459_1    anaconda
lerc                      3.0                  h295c915_0    anaconda
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcurl                   7.87.0               h91b91d3_0    anaconda
libdeflate                1.8                  h7f8727e_5    anaconda
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h6a678d5_6    anaconda
libgcc-ng                 11.2.0               h1234567_1    anaconda
libgfortran-ng            11.2.0               h00389a5_1    anaconda
libgfortran5              11.2.0               h1234567_1    anaconda
libgomp                   11.2.0               h1234567_1    anaconda
libnghttp2                1.46.0               hce63b2e_0    anaconda
libpng                    1.6.37               hbc83047_0    anaconda
libprotobuf               3.20.3               he621ea3_0    anaconda
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libssh2                   1.10.0               h8f2d780_0    anaconda
libstdcxx-ng              11.2.0               h1234567_1    anaconda
libthrift                 0.15.0               he6d91bd_0    conda-forge
libtiff                   4.5.0                h6a678d5_1    anaconda
libuv                     1.40.0               h7b6447c_0    anaconda
libwebp                   1.2.4                h11a3e52_0    anaconda
libwebp-base              1.2.4                h5eee18b_0    anaconda
lz4-c                     1.9.4                h6a678d5_0    anaconda
mako                      1.2.4              pyhd8ed1ab_0    conda-forge
markdown                  3.4.3              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py39h7f8727e_0    anaconda
matplotlib-base           3.4.3            py39h2fa2bec_2    conda-forge
matplotlib-inline         0.1.6            py39h06a4308_0    anaconda
mccabe                    0.7.0              pyhd3eb1b0_0    anaconda
mkl                       2021.4.0           h06a4308_640    anaconda
mkl-service               2.4.0            py39h7f8727e_0    anaconda
mkl_fft                   1.3.1            py39hd3c417c_0    anaconda
mkl_random                1.2.2            py39h51133e4_0    anaconda
mlflow                    2.5.0            py39ha39b057_0    conda-forge
molgrid                   0.5.3                    pypi_0    pypi
mpi                       1.0                       mpich    anaconda
mpich                     4.1.1                hbae89fd_0    defaults
mypy                      0.981            py39h06a4308_0    anaconda
mypy_extensions           0.4.3            py39h06a4308_1    anaconda
ncurses                   6.4                  h6a678d5_0    anaconda
nodeenv                   1.7.0            py39h06a4308_0    anaconda
numexpr                   2.8.4            py39he184ba9_0    anaconda
numpy                     1.23.5           py39h14f4228_0    anaconda
numpy-base                1.23.5           py39h31eccc5_0    anaconda
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1s               h7f8727e_0    anaconda
orc                       1.7.4                h07ed6aa_0    anaconda
packaging                 22.0             py39h06a4308_0    anaconda
pandas                    1.5.2            py39h417a72b_0    anaconda
paramiko                  3.2.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd3eb1b0_0    anaconda
pathspec                  0.10.3           py39h06a4308_0    anaconda
pexpect                   4.8.0              pyhd3eb1b0_3    anaconda
pickleshare               0.7.5           pyhd3eb1b0_1003    anaconda
pillow                    9.3.0            py39h6a678d5_2    anaconda
pip                       22.3.1           py39h06a4308_0    anaconda
platformdirs              2.5.2            py39h06a4308_0    anaconda
pluggy                    1.2.0                    pypi_0    pypi
pooch                     1.4.0              pyhd3eb1b0_0    anaconda
pre-commit                2.20.0           py39h06a4308_0    anaconda
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prometheus_flask_exporter 0.22.4             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.36           py39h06a4308_0    anaconda
protobuf                  3.20.3           py39h6a678d5_0    anaconda
psutil                    5.9.0            py39h5eee18b_0    anaconda
ptyprocess                0.7.0              pyhd3eb1b0_2    anaconda
pure_eval                 0.2.2              pyhd3eb1b0_0    anaconda
py                        1.11.0             pyhd3eb1b0_0    anaconda
pyarrow                   8.0.0            py39h992f0b0_0    anaconda
pycodestyle               2.10.0           py39h06a4308_0    anaconda
pycparser                 2.21               pyhd3eb1b0_0    anaconda
pyflakes                  3.0.1            py39h06a4308_0    anaconda
pygments                  2.11.2             pyhd3eb1b0_0    anaconda
pyjwt                     2.8.0              pyhd8ed1ab_0    conda-forge
pynacl                    1.5.0            py39hb9d737c_1    conda-forge
pyopenssl                 22.0.0             pyhd3eb1b0_0    anaconda
pyparsing                 3.1.0              pyhd8ed1ab_0    conda-forge
pyquaternion              0.9.9                    pypi_0    pypi
pysocks                   1.7.1            py39h06a4308_0    anaconda
pytest                    7.4.0                    pypi_0    pypi
pytest-cov                3.0.0              pyhd3eb1b0_0    anaconda
python                    3.9.16               h7a1cb2a_0    anaconda
python-dateutil           2.8.2              pyhd3eb1b0_0    anaconda
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.7           py39h06a4308_0    anaconda
pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
pyyaml                    6.0              py39hb9d737c_4    conda-forge
querystring_parser        1.2.4                      py_0    conda-forge
re2                       2022.04.01           h27087fc_0    conda-forge
readline                  8.2                  h5eee18b_0    anaconda
requests                  2.28.1           py39h06a4308_0    anaconda
scikit-learn              1.2.0            py39h6a678d5_0    anaconda
scikit-learn-intelex      2023.1.1         py39h06a4308_0    defaults
scipy                     1.10.0           py39h14f4228_0    anaconda
setuptools                65.6.3           py39h06a4308_0    anaconda
six                       1.16.0             pyhd3eb1b0_1    anaconda
smmap                     3.0.5              pyh44b312d_0    conda-forge
snappy                    1.1.9                h295c915_0    anaconda
sqlalchemy                1.4.39           py39h5eee18b_0    anaconda
sqlite                    3.40.1               h5082296_0    anaconda
sqlparse                  0.4.4              pyhd8ed1ab_0    conda-forge
stack_data                0.2.0              pyhd3eb1b0_0    anaconda
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tbb                       2021.6.0             hdb19cb5_1    anaconda
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tk                        8.6.12               h1ccaba5_0    anaconda
toml                      0.10.2             pyhd3eb1b0_0    anaconda
tomli                     2.0.1            py39h06a4308_0    anaconda
torchvision               0.13.1          cpu_py39h164cc8f_0    anaconda
tornado                   6.1              py39hb9d737c_3    conda-forge
tqdm                      4.64.1           py39h06a4308_0    anaconda
traitlets                 5.7.1            py39h06a4308_0    anaconda
typing_extensions         4.4.0            py39h06a4308_0    anaconda
tzdata                    2022a                hda174b7_0    anaconda
ukkonen                   1.0.1            py39hdb19cb5_0    anaconda
urllib3                   1.26.14          py39h06a4308_0    anaconda
utf8proc                  2.6.1                h27cfd23_0    anaconda
virtualenv                20.17.1          py39h06a4308_0    anaconda
wcwidth                   0.2.5              pyhd3eb1b0_0    anaconda
websocket-client          1.6.1              pyhd8ed1ab_0    conda-forge
werkzeug                  2.3.6              pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd3eb1b0_0    anaconda
xz                        5.2.10               h5eee18b_1    anaconda
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h5eee18b_0    anaconda
zstd                      1.5.2                ha4553b6_0    anaconda

[RMeli]

Hi @Kerro-junior. Unfortunately I can't reproduce this problem. Can you pleas try to run pytest --nogpu? This might give us some additional clues about the problem.

It takes hours in the 'solving' process when running conda env create -f devtools/conda-envs/gninatorch.yaml`

It is known that conda can be very slow in solving an environment. I'd strongly recommend to use mamba instead. For me it took less than two minutes to create the environment with mamba env create -f devtools/conda-envs/gninatorch.yaml.

I have created this environment and install all the libraries manually.

How did you install the libraries manually? With conda? There are known incompatibilities with libmolgrid and openbabel (see https://github.com/gnina/libmolgrid/issues/62), therefore you should be careful about how you install those.

---

PS: I took the freedom to re-format your issue. Please use code blocks and inline code blocks to make things a bit more readable than plain text.

[Kerro-junior]

For your questions, I manually installed the libraries by the consequence you provided in yml file with conda.(but libmolgrid with pip).

I try use $pytest --nogpu but it also leads to Segmentation fault (core dumped)

$ ls
build         CODE_OF_CONDUCT.md  docs      gninatorch           LICENSE      __pycache__  README.md        setup.cfg  tests
CITATION.cff  devtools            examples  gninatorch.egg-info  MANIFEST.in  pytest.ini   readthedocs.yml  setup.py   versioneer.py
$ pytest --nogpu
Segmentation fault (core dumped)

I will look into the incompatibilities between libmolgrid and openbabel later, and tried use mamba to create environment.

[Kerro-junior]

sorry to bother you, but I still cannot use mamba to install the gnina-torch, actually I do not know why I can't install mamba in conda, it always solving the imcompatibility.

do you have any idea of the reason of

$ pytest --nogpu
Segmentation fault (core dumped) 

[RMeli]

Unfortunately I can't reproduce the problem on my end, therefore it is a bit difficult to know what's going on.

do you have any idea

Since nothing is printed before the segmentation fault (not even the initial pytest stuff), I'd suspect the failure is happening within conftest.py. This would suggest there might be some issues with your molgrid installation.

it always solving the imcompatibility

As mentioned above, conda is notoriously slow in these situations. In my experience, this was happening when you try to install new things in an already bloated environment (an environment with many different packages, or the base environment if used for everything). But I haven't seen this issue since I switched to mamba.

why I can't install mamba in conda

If you look at the mamba installation instructions, you will see that installing mamba with conda is not recommended. Therefore, I'd suggest to try a fresh install from Mambaforge.

[Kerro-junior]

Thanks for your advice, I have successfully installed mambaforge into my base environment, and I have successfully used mamba to create gninatorch environment( I notive this env uses python3.11), then use python -m pip install . successfully installed gninatorch package.

BUT, the pytest still has troubles:

......
Building wheels for collected packages: gninatorch
  Building wheel for gninatorch (setup.py) ... done
  Created wheel for gninatorch: filename=gninatorch-0.0.2+3.ge46cb51-py3-none-any.whl size=37053758 sha256=76adaede79bdab7484c927344773e4a6612ce7ae60fe89e4da09f792cadaf349
  Stored in directory: /tmp/pip-ephem-wheel-cache-p64x6wx4/wheels/af/ee/f2/cba9368ce892555bc0147841fd0a14502c3d17893e4b99760d
Successfully built gninatorch
Installing collected packages: pyquaternion, molgrid, gninatorch
  Attempting uninstall: gninatorch
    Found existing installation: gninatorch 0.0.2+3.ge46cb51
    Can't uninstall 'gninatorch'. No files were found to uninstall.
Successfully installed gninatorch-0.0.2+3.ge46cb51 molgrid-0.5.3 pyquaternion-0.9.9

(gninatorch) [/home/user/gnina-torch]$ pytest --cov=gninatorch
-bash: /data2/rjli/bin/pytest: /data2/rjli/anaconda3/bin/python3.10: bad interpreter: No such file or directory

Then I check some basic information about python:

(gninatorch) [/home/user/gnina-torch]$ python
Python 3.11.3 (main, Apr 19 2023, 23:54:32) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

then I tried base environment to do pytest, still error

(gninatorch) [/home/user/gnina-torch]$ mamba activate base

(base) [/home/user/gnina-torch]$ pytest --cov=gninatorch
-bash: /data2/rjli/bin/pytest: /data2/rjli/anaconda3/bin/python3.10: bad interpreter: No such file or directory

[RMeli]

/data2/rjli/anaconda3/bin/python3.10

There is clearly something off with your configuration, since it is picking up Python not from the environment.

• Have you re-started your shell after installing mamba? Try to re-start the shell and re-activate the gina-torch environment.
• Was pytest correctly installed in your environment? Try to use python -m pytest --cov=gninatorch to make sure.

I have successfully installed mambaforge into my base environment

As mentioned previously, this is not recommended by the mamba community, so it might also be part of the problem.

[Kerro-junior]

Thanks, I have re-active the gninatorch environment, and it finally reports the test information:

$ python -m pytest  --cov=gninatorch
ImportError while loading conftest '/data2/rjli/gnina-torch/tests/conftest.py'.
tests/conftest.py:3: in <module>
    import molgrid
../anaconda3/lib/python3.11/site-packages/molgrid/__init__.py:1: in <module>
    from .molgrid import *
E   SystemError: type Boost.Python.enum has the Py_TPFLAGS_HAVE_GC flag but has no traverse function

Sorry to bother you, but I have no idea how to handle this libmolgrid import error...

[RMeli]

Can you please try to build an environment with Python 3.10 and see if something changes?

[Kerro-junior]

I firstly use mamba update python=3.10 to change the gninatorch environment's python, then I run the pytest and got:

(gninatorch)$ python -m pytest --cov=gninatorch
 version `GLIBCXX_3.4.30' not found (required by ~/mambaforge/envs/gninatorch/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
Failed to import torch.
ImportError while loading conftest 'gnina-torch/tests/conftest.py'.
tests/conftest.py:5: in <module>
    import torch
../mambaforge/envs/gninatorch/lib/python3.10/site-packages/torch/__init__.py:199: in <module>
    from torch._C import *  # noqa: F403

So I delete the specific gcc I defined in the ~/.bashrc, and run the pytest again:

$ python -m pytest --cov=gninatorch
ImportError while loading conftest '/home/user/gnina-torch/tests/conftest.py'.
tests/conftest.py:3: in <module>
    import molgrid
../anaconda3/lib/python3.11/site-packages/molgrid/__init__.py:1: in <module>
    from .molgrid import *
E   SystemError: type Boost.Python.enum has the Py_TPFLAGS_HAVE_GC flag but has no traverse function

HOW STRANGE it is! the python of gninatorch turn into python3.11 again?I tried again 'mamba update python=3.10' to update python into 3.10 but failed.SO I remove the env and modify the .yamlfile in /devtools/conda-envs as 'python=3.9' (I saw a mamba warning about molgrid require python3.9 specifically), and thenpython -m pip install .

Then I conda activate gninatorch, the python is still 3.11, Later I realized I need to use 'mamba activate gninatorch' to use the python3.9 installed in gninatorch. If I use 'conda activate gninatorch' the python will always point to /anaconda3/bin/python3.11

So finally after I installed the gninatorch with mamba, BUT it constantly report errors:

ImportError while loading conftest '/data2/rjli/gnina-torch/tests/conftest.py'.
tests/conftest.py:5: in <module>
    import torch
../mambaforge/envs/gninatorch/lib/python3.9/site-packages/torch/__init__.py:199: in <module>
    from torch._C import *  # noqa: F403
E   ImportError: /home/software/support/gcc/950/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/user/mambaforge/envs/gninatorch/lib/python3.9/site-packages/torch/lib/libtorch_python.so)

[Kerro-junior]

I learned that it's the problem of my gcc, FYI my gcc is assigned by adding source /home/software/ENV/support/gcc/gcc-9.5.0.env , and when I run 'which gcc' it echos: /home/software/support/gcc/950/bin/gcc

I would be grateful if you can teach how to solve the problem ImportError: /home/software/support/gcc/950/lib64/libstdc++.so.6: versionGLIBCXX_3.4.30' not found`, casue I know little about these stuff...

[RMeli]

@Kerro-junior thanks for checking. Since this is a problem with how your setup is (mis-)configured, I can't help much. You will have to fix how things are setup on your end.

What I can suggest is to try to install a newer version of GCC from coda-forge within your gninatorch environment:

mamba install -c condo-forge gcc=12.1.0