kpeeters / cadabra2

A field-theory motivated approach to computer algebra.
https://cadabra.science/
GNU General Public License v3.0
216 stars 37 forks source link

Cadabra2 young_project_product`and young_project_tensor crashed on segmentation fault #131

Closed mzavadil closed 5 years ago

mzavadil commented 5 years ago

Windows version of Cadabra2 crashed for any expression on young_project_product or young_project_tensor with modulo_monoterm=True (if this parameter set to False computation ok).

I tried to build three versions from source code: current master from 14.2.2019, version with tag 2.2.5 and 2.2.4. The same problem for each of them.

Windows version:

Microsoft Windows 10 Pro: 10.0.17763 Build 17763

conda info

     active environment : base
    active env location : C:\tools\Anaconda3
            shell level : 1
       user config file : C:\Users\bigfo\.condarc
 populated config files : C:\Users\bigfo\.condarc
          conda version : 4.6.4
    conda-build version : 3.17.8
         python version : 3.7.2.final.0
       base environment : C:\Tools\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Tools\Anaconda3\pkgs
                          C:\Users\bigfo\.conda\pkgs
                          C:\Users\bigfo\AppData\Local\conda\conda\pkgs
       envs directories : C:\Tools\Anaconda3\envs
                          C:\Users\bigfo\.conda\envs
                          C:\Users\bigfo\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.4 requests/2.21.0 CPython/3.7.2 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False

conda list

# packages in environment at C:\tools\Anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0
alabaster                 0.7.12                   py37_0
anaconda                  custom                   py37_0
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.6                    py37_0
anaconda-project          0.8.2                    py37_0
asn1crypto                0.24.0                   py37_0
astroid                   2.1.0                    py37_0
astropy                   3.1.1            py37he774522_0
atomicwrites              1.3.0                      py_0
attrs                     18.2.0           py37h28b3542_0
babel                     2.6.0                    py37_0
backcall                  0.1.0                    py37_0
backports                 1.0                      py37_1
backports.os              0.1.1                    py37_0
backports.shutil_get_terminal_size 1.0.0                    py37_2
beautifulsoup4            4.7.1                    py37_1
bitarray                  0.8.3            py37hfa6e2cd_0
bkcharts                  0.2                      py37_0
blas                      1.0                         mkl
blaze                     0.11.3                   py37_0
bleach                    3.1.0                    py37_0
blosc                     1.15.0               h7bd577a_0
bokeh                     1.0.4                    py37_0
boto                      2.49.0                   py37_0
bottleneck                1.2.1            py37h452e1ab_1
bzip2                     1.0.6                hfa6e2cd_5
ca-certificates           2019.1.23                     0
certifi                   2018.11.29               py37_0
cffi                      1.12.0                   pypi_0    pypi
chardet                   3.0.4                    py37_1
click                     7.0                      py37_0
cloudpickle               0.8.0                    pypi_0    pypi
clyent                    1.2.2                    py37_1
colorama                  0.4.1                    pypi_0    pypi
comtypes                  1.1.7                    py37_0
conda                     4.6.4                    py37_0
conda-build               3.17.8                   py37_0
conda-env                 2.6.0                         1
conda-verify              3.1.1                    py37_0
console_shortcut          0.1.1                         3
contextlib2               0.5.5                    py37_0
cryptography              2.5              py37h7a1dbc1_0
curl                      7.63.0            h2a8f88b_1000
cycler                    0.10.0                   py37_0
cython                    0.29.5                   pypi_0    pypi
cytoolz                   0.9.0.1          py37hfa6e2cd_1
dask                      1.1.1                      py_0
dask-core                 1.1.1                      py_0
datashape                 0.5.4                    py37_1
decorator                 4.3.2                    py37_0
defusedxml                0.5.0                    py37_1
distributed               1.25.3                   py37_0
docutils                  0.14                     py37_0
entrypoints               0.3                      py37_0
et_xmlfile                1.0.1                    py37_0
fastcache                 1.0.2            py37hfa6e2cd_2
filelock                  3.0.10                   py37_0
flask                     1.0.2                    py37_1
flask-cors                3.0.7                    py37_0
freetype                  2.9.1                ha9979f8_1
future                    0.17.1                   py37_0
get_terminal_size         1.0.0                h38e98db_0
gevent                    1.4.0            py37he774522_0
glob2                     0.6                      py37_1
greenlet                  0.4.15           py37hfa6e2cd_0
h5py                      2.9.0            py37h5e291fa_0
hdf5                      1.10.4               h7ebc959_0
heapdict                  1.0.0                    py37_2
html5lib                  1.0.1                    py37_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
idna                      2.8                      py37_0
imageio                   2.4.1                    py37_0
imagesize                 1.1.0                    py37_0
importlib-metadata        0.8                      pypi_0    pypi
intel-openmp              2019.1                      144
ipykernel                 5.1.0            py37h39e3cac_0
ipython                   7.2.0            py37h39e3cac_0
ipython-genutils          0.2.0                    pypi_0    pypi
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.4.2                    py37_0
isort                     4.3.4                    py37_0
itsdangerous              1.1.0                    py37_0
jdcal                     1.4                      py37_0
jedi                      0.13.2                   pypi_0    pypi
jinja2                    2.10                     py37_0
jpeg                      9b                   hb83a4c4_2
jsonschema                2.6.0                    py37_0
jupyter                   1.0.0                    py37_7
jupyter_client            5.2.4                    py37_0
jupyter_console           6.0.0                    py37_0
jupyter_core              4.4.0                    py37_0
jupyterlab                0.35.4                   pypi_0    pypi
jupyterlab_server         0.2.0                    py37_0
keyring                   18.0.0                   pypi_0    pypi
kiwisolver                1.0.1            py37h6538335_0
krb5                      1.16.1               hc04afaa_7
lazy-object-proxy         1.3.1            py37hfa6e2cd_2
libarchive                3.3.3                h0643e63_5
libcurl                   7.63.0            h2a8f88b_1000
libiconv                  1.15                 h1df5818_7
liblief                   0.9.0                ha925a31_2
libpng                    1.6.36               h2a8f88b_0
libsodium                 1.0.16               h9d3ae62_0
libssh2                   1.8.0                h7a1dbc1_4
libtiff                   4.0.10               hb898794_2
libxml2                   2.9.9                h464c3ec_0
libxslt                   1.1.33               h579f668_0
llvmlite                  0.27.0           py37ha925a31_0
locket                    0.2.0                    py37_1
lxml                      4.3.1                    pypi_0    pypi
lz4-c                     1.8.1.2              h2fa13f4_0
lzo                       2.10                 h6df0209_2
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markupsafe                1.1.0            py37he774522_0
matplotlib                3.0.2            py37hc8f65d3_0
mccabe                    0.6.1                    py37_1
menuinst                  1.4.14           py37hfa6e2cd_0
mistune                   0.8.4            py37he774522_0
mkl                       2019.1                      144
mkl-service               1.1.2            py37hb782905_5
mkl_fft                   1.0.10           py37h14836fe_0
mkl_random                1.0.2            py37h343c172_0
more-itertools            6.0.0                    pypi_0    pypi
mpmath                    1.1.0                    py37_0
msgpack-python            0.6.1            py37h74a9793_1
msys2-conda-epoch         20160418                      1
multipledispatch          0.6.0                    py37_0
mypy                      0.670                    pypi_0    pypi
mypy-extensions           0.4.1                    pypi_0    pypi
navigator-updater         0.2.1                    py37_0
nbconvert                 5.3.1                    py37_0
nbformat                  4.4.0                    py37_0
networkx                  2.2                      py37_1
nltk                      3.4                      py37_1
nose                      1.3.7                    py37_2
notebook                  5.7.4                    py37_0
numba                     0.42.0           py37hf9181ef_0
numexpr                   2.6.9            py37hdce8814_0
numpy                     1.15.4                   pypi_0    pypi
numpydoc                  0.8.0                    py37_0
odo                       0.5.1                    py37_0
olefile                   0.46                     py37_0
openpyxl                  2.6.0                    pypi_0    pypi
openssl                   1.1.1a               he774522_0
packaging                 19.0                     py37_0
pandas                    0.24.1           py37ha925a31_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py37_1
parso                     0.3.4                    pypi_0    pypi
partd                     0.3.9                    py37_0
path.py                   11.5.0                   py37_0
pathlib2                  2.3.3                    py37_0
patsy                     0.5.1                    py37_0
pep8                      1.7.1                    py37_0
pickleshare               0.7.5                    pypi_0    pypi
pillow                    5.4.1            py37hdc69c19_0
pip                       19.0.2                   pypi_0    pypi
pkginfo                   1.5.0.1                  py37_0
plotly                    3.5.0                    py37_0
pluggy                    0.8.1                    py37_0
ply                       3.11                     py37_0
prometheus_client         0.5.0                    py37_0
prompt_toolkit            2.0.8                      py_0
psutil                    5.5.1                    pypi_0    pypi
py                        1.7.0                    py37_0
py-lief                   0.9.0            py37ha925a31_2
pycodestyle               2.5.0                    py37_0
pycosat                   0.6.3            py37hfa6e2cd_0
pycparser                 2.19                     py37_0
pycrypto                  2.6.1            py37hfa6e2cd_9
pycurl                    7.43.0.2         py37h7a1dbc1_0
pyflakes                  2.1.0                    py37_0
pygments                  2.3.1                    pypi_0    pypi
pylint                    2.2.2                    py37_0
pyodbc                    4.0.25           py37ha925a31_0
pyopenssl                 19.0.0                   py37_0
pyparsing                 2.3.1                    py37_0
pyqt                      5.9.2            py37h6538335_2
pyreadline                2.1                      py37_1
pysocks                   1.6.8                    py37_0
pytables                  3.4.4            py37h1da0976_0
pytest                    4.2.1                    pypi_0    pypi
pytest-arraydiff          0.3              py37h39e3cac_0
pytest-astropy            0.5.0                    py37_0
pytest-doctestplus        0.2.0                    py37_0
pytest-openfiles          0.3.2                    py37_0
pytest-remotedata         0.3.1                    py37_0
python                    3.7.2                h8c8aaf0_2
python-dateutil           2.8.0                    pypi_0    pypi
python-libarchive-c       2.8                      py37_6
pytz                      2018.9                   py37_0
pywavelets                1.0.1            py37h8c2d366_0
pywin32                   224                      pypi_0    pypi
pywin32-ctypes            0.2.0                    pypi_0    pypi
pywinpty                  0.5.5                 py37_1000
pyyaml                    3.13             py37hfa6e2cd_0
pyzmq                     17.1.2           py37ha925a31_2
qt                        5.9.7            vc14h73c81de_0  [vc14]
qtawesome                 0.5.6                      py_0
qtconsole                 4.4.3                    py37_0
qtpy                      1.6.0                      py_0
requests                  2.21.0                   py37_0
retrying                  1.3.3                    py37_2
rope                      0.12.0                   pypi_0    pypi
ruamel_yaml               0.15.46          py37hfa6e2cd_0
scikit-image              0.14.2                   pypi_0    pypi
scikit-learn              0.20.2           py37h343c172_0
scipy                     1.2.1                    pypi_0    pypi
seaborn                   0.9.0                    py37_0
send2trash                1.5.0                    py37_0
setuptools                40.8.0                   pypi_0    pypi
simplegeneric             0.8.1                    py37_2
singledispatch            3.4.0.3                  py37_0
sip                       4.19.8           py37h6538335_0
six                       1.12.0                   pypi_0    pypi
snappy                    1.1.7                h777316e_3
snowballstemmer           1.2.1                    py37_0
sortedcollections         1.1.2                    py37_0
sortedcontainers          2.1.0                    py37_0
soupsieve                 1.7.3                    pypi_0    pypi
sphinx                    1.8.4                    py37_0
sphinxcontrib             1.0                      py37_1
sphinxcontrib-websupport  1.1.0                    py37_1
spyder                    3.3.3                    py37_0
spyder-kernels            0.4.2                    py37_0
sqlalchemy                1.2.18                   pypi_0    pypi
sqlite                    3.26.0               he774522_0
statsmodels               0.9.0            py37h452e1ab_0
sympy                     1.3                      py37_0
tblib                     1.3.2                    py37_0
terminado                 0.8.1                    py37_1
testpath                  0.4.2                    py37_0
tk                        8.6.8                hfa6e2cd_0
toolz                     0.9.0                    py37_0
tornado                   5.1.1            py37hfa6e2cd_0
tqdm                      4.31.1                   pypi_0    pypi
traitlets                 4.3.2                    py37_0
typed-ast                 1.3.1                    pypi_0    pypi
unicodecsv                0.14.1                   py37_0
urllib3                   1.24.1                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_0
wcwidth                   0.1.7                    pypi_0    pypi
webencodings              0.5.1                    py37_1
werkzeug                  0.14.1                   py37_0
wheel                     0.33.0                   pypi_0    pypi
widgetsnbextension        3.4.2                    py37_0
win_inet_pton             1.0.1                    py37_1
win_unicode_console       0.5                      py37_0
wincertstore              0.2                      py37_0
winpty                    0.4.3                         4
wrapt                     1.11.1           py37he774522_0
xlrd                      1.2.0                    py37_0
xlsxwriter                1.1.4                    pypi_0    pypi
xlwings                   0.15.2                   py37_0
xlwt                      1.3.0                    py37_0
xz                        5.2.4                h2fa13f4_4
yaml                      0.1.7                hc54c509_2
zeromq                    4.3.1                h33f27b4_3
zict                      0.1.3                    py37_0
zipp                      0.3.3                    pypi_0    pypi
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0

MikTeX version: 2.9.6972

kpeeters commented 5 years ago

Are you able to provide me with a backtrace when the crash happens?

mzavadil commented 5 years ago

I did a little debugging in VS2017:

cababra2

Cadabra crashes in DataCell.cc during eval python script by PyRun_String:

# -*- coding: utf-8 -*-
R=Ex(r'R_{a b c d}')
RiemannTensor(R, Ex(r''))
A=Ex(r'A^{a b c}')
AntiSymmetric(A, Ex(r''))
ex=A*R
young_project_tensor(ex, modulo_monoterm = True)

on null pointer dereference in cadabra2.pyd module if script contains line: young_project_tensor(ex, modulo_monoterm = True)

Note: this code part is not at git repository, I added it for debugging only.

PyObject* ex = PyErr_Occurred();
        if (ex) {
            PyObject *type, *value, *traceback;
            PyErr_Fetch(&type, &value, &traceback);
            //pybind11_fail("Python script error: " + value.c_str());
        }

For more information, there is necessary debug module cadabra2.pyd at method young_project_tensor , but I do not know exactly how. If you tell me the concrete steps, I can try to debug myself.

mzavadil commented 5 years ago

I tried to run it in WinDBG:

windbg

And the null pointer dereference problem seems to be in indexsort.cc line 41 where pointer tb is null.

In DEBUG mode is all OK because method indexsort::can_apply() checks a null of tb pointer resp. sets it by assert macro at young_project_tensor.cc:

if(modulo_monoterm) { // still necessary for column exchange
  indexsort isort(kernel, rep);
  assert(isort.can_apply(newtensor)); // to set tb
  isort.apply(newtensor);
}

But assert macro is deactivated in RELEASE mode when NDEBUG is defined, see: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/assert-macro-assert-wassert?view=vs-2017

Solution is in either project do not use assert macro for business logic (only checks which can be in release mode safely neglected) or do not build with NDEBUG directive.

kpeeters commented 5 years ago

Thanks for tracking this down; linux compilers typically leave asserts in release code so that's why I never noticed it. I'll do a grep through the source, there may be other cases of this problem.

kpeeters commented 5 years ago

I have moved (hopefully all) content of assert macros which has side effects out of the assert, so it will generate the correct code even with NDEBUG; 8e837207bb9e417e1deb4a1754d649a0762e009a . Can you test this for me please?

mzavadil commented 5 years ago

I tried to build the last commit, but with compilation error: build_error.txt Probably an invalid syntax of generic iterator do_subtree in header file Functional.hh occurred from the previous commit.

After cherry-picked your fix https://github.com/kpeeters/cadabra2/commit/8e837207bb9e417e1deb4a1754d649a0762e009a on commit https://github.com/kpeeters/cadabra2/commit/a4f9fa61d82a6902fa52230a92a2d3a5c9378176 is build successful except tests builds:

"C:\Users\bigfo\Workspace\dev\cadabra2\build\install.vcxproj" (default target) (1) ->
"C:\Users\bigfo\Workspace\dev\cadabra2\build\ALL_BUILD.vcxproj" (default target) (3) ->
"C:\Users\bigfo\Workspace\dev\cadabra2\build\tests\beginners_test.vcxproj" (default target) (4) ->
(CustomBuild target) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 3. [C:\Users\bigfo\Workspace\dev\cadabra2\build\tests\beginners_test.vcxproj]

The problem is probably in unix-like path to echo utility in tests/CMakeLists.txt.

add_custom_target("${NBTEST}_test" ALL /bin/echo -n DEPENDS ${CDBOUT}/${NBTEST}.cdb)

I changed it to: (internal cmd command echo has not support for the parameter -n)

add_custom_target("${NBTEST}_test" ALL echo DEPENDS ${CDBOUT}/${NBTEST}.cdb)

and build is complete successful.

Only ctest command don't passed for this "echo" manipulations :(


      Start  1: quickstart
 1/43 Test  #1: quickstart .......................***Failed    0.24 sec
      Start  2: beginners
 2/43 Test  #2: beginners ........................***Failed    0.12 sec
      Start  3: tensor_monomials
 3/43 Test  #3: tensor_monomials .................***Failed    0.12 sec
      Start  4: for_previous_users
 4/43 Test  #4: for_previous_users ...............***Failed    0.13 sec
      Start  5: converge```

But reported bug of this issue is fixed - windows version of Cadabra2 don't crashes anymore.
mzavadil commented 5 years ago

Unfortunately your fix of young_project_tensor don't work for statements contains Derivative:

# -*- coding: utf-8 -*-
from cadabra2 import *
print("start...")
print(Indices(Ex(r"{a,b}"), Ex(r"")))
print(AntiSymmetric(Ex(r"F^{a b}"), Ex(r"")))
print(Derivative(Ex(r"\nabla{#}"), Ex(r"")))
ex = Ex(r"\nabla_{b}{ F^{a b} }")
print(ex)
_ = young_project_tensor(ex)

The problem is again in null pointer dereference of tb, this time in the module core\properties\derivative.cc:

derive_crash

Note: It was built from last commint https://github.com/kpeeters/cadabra2/commit/8e837207bb9e417e1deb4a1754d649a0762e009a with excluded build-invalid commit https://github.com/kpeeters/cadabra2/commit/c4a590bf7a17d8272c9455fdb75d1e2c6cc6bdec.

kpeeters commented 5 years ago

That's a different bug, thanks for reporting it. Now fixed with commit 466b4e68b7e492b2d77fb36f5d20b9df157585d7 .

kpeeters commented 5 years ago

Your build error should now be fixed on the master branch, please let me know whether it works on your end.

mzavadil commented 5 years ago

I confirm, build error related to syntax problem in Functional.hh has been fixed, but there is still the problem with the echo utility in tests/CMakeLists.txt. Without its repair, whole build of Cadabra2 fails on

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 3. [C:\Users\bigfo\Workspace\dev\cadabra2\build\tests\beginners_test.vcxproj]

Then there is one more specific problem with cmake. I will create new issue for it.

kpeeters commented 5 years ago

Have replaced /bin/echo with echo.