10 errors in tests

[francispoulin]

I have installed shenfun using conda, as requested and that seemed to go well. However, when I try the tests, as suggested I get 10 errors. Maybe my conda install wasn't as good as I thought it was?

`$ python -m pytest tests/ =============================== test session starts =============================== platform linux -- Python 3.8.2, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 rootdir: /home/fpoulin/software/shenfun collected 1 item / 10 errors

===================================== ERRORS ====================================== __ ERROR collecting tests/test_bases.py _ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_bases.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_bases.py:3: in from shenfun import * shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMASymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ERROR collecting tests/test_curl.py _ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_curl.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_curl.py:4: in from shenfun import inner, div, curl, TestFunction, TrialFunction, Function, \ shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ____ ERROR collecting tests/test_forms.py ___ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_forms.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_forms.py:4: in import shenfun shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMASymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ERROR collecting tests/test_fourier.py __ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_fourier.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_fourier.py:2: in from shenfun import shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ____ ERROR collecting tests/test_io.py ____ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_io.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_io.py:6: in from shenfun import shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ____ ERROR collecting tests/test_la.py ____ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_la.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_la.py:4: in from shenfun.chebyshev.la import PDMA shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' _ ERROR collecting tests/test_lagrangianparticles.py ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_lagrangian_particles.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_lagrangian_particles.py:4: in from shenfun import * shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMASymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ____ ERROR collecting tests/testmatrices.py ____ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_matrices.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_matrices.py:9: in import shenfun shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ____ ERROR collecting tests/test_tensorproductspace.py ____ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_tensorproductspace.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_tensorproductspace.py:8: in from shenfun.fourier import bases as fbases shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' ____ ERROR collecting tests/test_transforms.py ____ ImportError while importing test module '/home/fpoulin/software/shenfun/tests/test_transforms.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: tests/test_transforms.py:7: in import shenfun shenfun/init.py:33: in from . import chebyshev shenfun/chebyshev/init.py:2: in from . import la shenfun/chebyshev/la.py:6: in from shenfun.optimization.cython import la shenfun/optimization/cython/init.py:1: in from .la import TDMA_SymLU, TDMA_SymSolve, PDMA_SymLU, PDMA_SymSolve, \ E ModuleNotFoundError: No module named 'shenfun.optimization.cython.la' !!!!!!!!!!!!!!!!!!!! Interrupted: 10 errors during collection !!!!!!!!!!!!!!!!!!!!! =============================== 10 errors in 0.66s ================================`

[mikaem]

This is an installation issue. My guess is that you have both installed shenfun from conda AND cloned it from github. And now that you're running the tests it is trying to use the version you cloned from github. It is not a problem to have both these versions installed, Python will use whichever it finds first. The problem is that you need to run install on the cloned version, because the cython code there needs to be compiled. Run setup.py with

python setup.py build_ext -i

This should build the cloned version as well. Note that the version installed from conda already has the cython code built and is ready to go out of the box.

[francispoulin]

You are correct. I installed shenfun using conda, as I now see it's the simplest way to go, but then I also cloned the repo just to be able to access the tests. Maybe that's a bad idea but I thought this was the easiest way to access the demos and tests.

I built it as you suggested and that worked fine. Then I ran it and received the same error. However, when I moved just out of the directory and tried to run the tests using shenfun/tests/ it gave me a different error. Since I don't have shenfun in my path it must only be using conda, which is what i preferred. I've copied the error below but it is essentially a core dump.

`$ python -m pytest shenfun/tests/ =============================== test session starts =============================== platform linux -- Python 3.8.2, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 rootdir: /home/fpoulin/software/shenfun collecting ... Fatal Python error: Illegal instruction

Current thread 0x00007f0c1e19e740 (most recent call first): File "", line 219 in _call_with_frames_removed File "", line 1109 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/optimization/cython/init.py", line 1 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/chebyshev/la.py", line 6 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1042 in _handle_fromlist File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/chebyshev/init.py", line 2 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1042 in _handle_fromlist File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/init.py", line 33 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/home/fpoulin/software/shenfun/tests/test_bases.py", line 3 in File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 143 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/py/_path/local.py", line 701 in pyimport File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/python.py", line 493 in _importtestmodule File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/python.py", line 425 in _getobj File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/python.py", line 249 in obj File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/python.py", line 441 in _inject_setup_module_fixture File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/python.py", line 428 in collect File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/runner.py", line 257 in File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/runner.py", line 237 in from_call File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/runner.py", line 257 in pytest_make_collect_report File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/manager.py", line 84 in File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in call File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/runner.py", line 379 in collect_one_node File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 721 in genitems File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 496 in _perform_collect File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 458 in perform_collect File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 256 in pytest_collection File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/manager.py", line 84 in File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in call File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 246 in _main File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 197 in wrap_session File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/manager.py", line 84 in File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in call File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/_pytest/config/init.py", line 92 in main File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/pytest/main.py", line 7 in File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/runpy.py", line 86 in _run_code File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/runpy.py", line 193 in _run_module_as_main Illegal instruction (core dumped) `

[mikaem]

I usually have both conda-forge shenfun and a local version available, and then I set the PYTHONPATH if I want to use the local one. So there should be no problem there. Not sure really what is going on. Could you please enter the demo folder and try some of the demos? For example

cd shenfun/demo
python dirichlet_poisson2D.py 24 chebyshev

Mayber there's something wrong with pytest?

[francispoulin]

Good idea. Unfortunately, even the demo gives me a core dump. See output below. I also copied this file outside of the cloned repo and it still gave me the same error.

(shenfun) fpoulin@amlt21:~/software/shenfun/demo$ python dirichlet_poisson2D.py 24 chebyshev Illegal instruction (core dumped)

[mikaem]

Does anything with Python work? Or is it just shenfun?

[francispoulin]

I tried to import numpy and mpi4py and they both seem to work but have not done extensive testing.

I will delete this env and reinstall it to see if the problem persists. Maybe I did something silly in the installation. I will get back to you if the problem persists.

[francispoulin]

I removed and reinstalled shenfun and unfortunately am getting the same errors with the pytest (which was installed by pip) and also when I try the example you mentioned before.

Also, when I am in the shenfun env and try importing shenfun I get a core dump. Nothing with shenfun seems to be working for me, even the most basic command.

[mikaem]

Aha, but did you install pip from conda in the shenfun environment? If you did not that could be it. You have to be careful when mixing conda and pip

which pip

will tell you which pip you are using. Again, conda is not mastered in a day:-) Lots of traps that have little to do with shenfun.

See the test installation chapter

[francispoulin]

Good question. I removed shenfun and reinstalled it using the command below. I am not actually using pytest but when I open python from shenfun and try and import shenfun it fails with a core dump. I tried the tests and they also fail as before.

conda create --name shenfun -c conda-forge shenfun mpich nomkl matplotlib pip pytest

[mikaem]

You can skip the nomkl, but other than that it should be sufficient. I'm sure this has something to do with your environment, because shenfun is tested on each commit using both linux and osx containers with python 2.7, 3.6, 3.7 and 3.8. Could you perhaps print what you get with

conda list

?

[francispoulin]

Should I try installing it without the nomkl? Easy enough to do.

Insert: I did a barebones installation using conda create --name shenfun -c conda-forge shenfun and it still gives a core dump.

From the previous one the conda list I get is the following:

`(shenfun) fpoulin@amlt21:~/software/shenfun$ conda list

packages in environment at /home/fpoulin/software/anaconda3/envs/shenfun:

#

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_llvm conda-forge attrs 19.3.0 py_0 conda-forge bzip2 1.0.8 h516909a_2 conda-forge ca-certificates 2019.11.28 hecc5488_0 conda-forge certifi 2019.11.28 py38_0 conda-forge cftime 1.0.4.2 py38hc1659b7_0 conda-forge curl 7.68.0 hf8cf82a_0 conda-forge cycler 0.10.0 py_2 conda-forge dbus 1.13.6 he372182_0 conda-forge expat 2.2.9 he1b5a44_2 conda-forge fastcache 1.1.0 py38h516909a_0 conda-forge fftw 3.3.8 nompi_h7f3a6c3_1110 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.0 he983fc9_1 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge glib 2.58.3 py38h6f030ca_1002 conda-forge gmp 6.2.0 he1b5a44_2 conda-forge gmpy2 2.1.0b1 py38h04dde30_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_2 conda-forge gstreamer 1.14.5 h36ae1b5_2 conda-forge h5py 2.10.0 mpi_mpich_py38haaae0f6_2 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.5 mpi_mpich_ha7d0aea_1004 conda-forge icu 64.2 he1b5a44_1 conda-forge importlib_metadata 1.5.0 py38_0 conda-forge jpeg 9c h14c3975_1001 conda-forge kiwisolver 1.1.0 py38hc9558a2_0 conda-forge krb5 1.16.4 h2fd8d38_0 conda-forge ld_impl_linux-64 2.33.1 h53a641e_8 conda-forge libblas 3.8.0 15_openblas conda-forge libcblas 3.8.0 15_openblas conda-forge libclang 9.0.1 default_hde54327_0 conda-forge libcurl 7.68.0 hda55be3_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgfortran-ng 7.3.0 hdf63c60_5 conda-forge libiconv 1.15 h516909a_1005 conda-forge liblapack 3.8.0 15_openblas conda-forge libllvm9 9.0.1 hc9558a2_0 conda-forge libnetcdf 4.7.3 mpi_mpich_h755db7c_1 conda-forge libopenblas 0.3.8 h5ec1e0e_0 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxkbcommon 0.10.0 he1b5a44_0 conda-forge libxml2 2.9.10 hee79883_0 conda-forge llvm-openmp 9.0.1 hc9558a2_2 conda-forge matplotlib 3.1.3 py38_0 conda-forge matplotlib-base 3.1.3 py38h250f245_0 conda-forge more-itertools 8.2.0 py_0 conda-forge mpc 1.1.0 h04dde30_1006 conda-forge mpfr 4.0.2 he80fd80_0 conda-forge mpi 1.0 mpich conda-forge mpi4py 3.0.3 py38hcf07815_0 conda-forge mpi4py-fft 2.0.3 py38hc8fccaa_1 conda-forge mpich 3.3.2 hc856adb_0 conda-forge mpmath 1.1.0 py_0 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge netcdf4 1.5.3 mpi_mpich_py38h00fb2e7_2 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge nspr 4.25 he1b5a44_0 conda-forge nss 3.47 he751ad9_0 conda-forge numpy 1.18.1 py38h95a1406_0 conda-forge openssl 1.1.1d h516909a_0 conda-forge packaging 20.1 py_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pip 20.0.2 py_2 conda-forge pluggy 0.13.1 py38_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge py 1.8.1 py_0 conda-forge pyparsing 2.4.6 py_0 conda-forge pyqt 5.12.3 py38hcca6a23_1 conda-forge pyqt5-sip 4.19.18 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pytest 5.3.5 py38_1 conda-forge python 3.8.2 h9d8adfe_1_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge qt 5.12.5 hd8c4c69_1 conda-forge readline 8.0 hf8c457e_0 conda-forge scipy 1.4.1 py38h921218d_0 conda-forge setuptools 45.2.0 py38_0 conda-forge shenfun 2.1.1 py38hc8fccaa_1 conda-forge six 1.14.0 py38_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge sympy 1.5.1 py38_1 conda-forge tk 8.6.10 hed695b0_0 conda-forge tornado 6.0.3 py38h516909a_4 conda-forge wcwidth 0.1.8 py_0 conda-forge wheel 0.34.2 py_1 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xz 5.2.4 h14c3975_1001 conda-forge zipp 3.0.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge `

[mikaem]

And if you do

import shenfun

Does it crash? For both local version and conda-forge version?

[francispoulin]

All I get is a core dump. Is there a way to output more information than that?

I am only using the conda-forge version. I did the build of the local version but nothing more than that. I'm wondering if I should remove that, just in case it's getting in the way.

`(shenfun) fpoulin@amlt21:~/software$ python Python 3.8.2 | packaged by conda-forge | (default, Feb 28 2020, 17:15:22) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import shenfun Illegal instruction (core dumped) (shenfun) fpoulin@amlt21:~/software$ `

[mikaem]

At this point I think you need to try some debugging. Have you ever tried faulthandler? Or something else? It would probably also be a good idea to ask someone with expertise in conda for assistance. Not so easy to be of assistance from a distance.

[francispoulin]

Thanks for the suggestion and agree that debugging is essential. Something I've always wanted to learn and now's the time.

I took the first step and found the following result. This does not make much sense to me but will try and ask for help from conda-forge. If you have any thoughts based on this output please let me know.

`(shenfun) fpoulin@amlt21:~/software$ python -c 'import shenfun' Illegal instruction (core dumped) (shenfun) fpoulin@amlt21:~/software$ python -q -X faulthandler

import shenfun Fatal Python error: Illegal instruction

Current thread 0x00007f55ef76a740 (most recent call first): File "", line 219 in _call_with_frames_removed File "", line 1109 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/optimization/cython/init.py", line 1 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/chebyshev/la.py", line 6 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1042 in _handle_fromlist File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/chebyshev/init.py", line 2 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1042 in _handle_fromlist File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/site-packages/shenfun/init.py", line 33 in File "", line 219 in _call_with_frames_removed File "", line 783 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 1 in Illegal instruction (core dumped) `

[mikaem]

You really need to break this problem down. The code crashes trying to import module la from shenfun.optimization.cython, line 6 in chebyshev/la.py. So you could start by commenting out that line. This requires you to use the local version of shenfun. To do this prepend the path to shenfun to the python path

export PYTHONPATH=/installation path/shenfun/:$PYTHONPATH

And then build using python setup.py build_ext -i

BTW, looking at you error message. Did you clone shenfun from inside the anaconda3 path? In that case, don't. You should not do anything inside that path directly. That is conda's domain.

[francispoulin]

Thanks.

I did not clone shenfun into my anaconda3 path. I just cloned it right outside. I didn't actually do any building or installing so I don't think it should have done much.

Bad news: I tried importing each line from init.py and each one gave the same error. Not looking good for my conda install.

Good news: I trying doing a local install, seemed more complicated hence why I went to conda, and I'm happy to say that I can import shenfun easily. I also tried the tests and even happier to say that only 184 failed and 6849 passed, with 11 warnings. Much better than before. I know that it is going to be a little bit more effort to update but I suppose this is what I have to do.

Do you want me to ask conda-forge for help with the install to see what has gone wrong?

git clone https://github.com/spectralDNS/shenfun.git cd shenfun/ python setup.py build_ext -i export PYTHONPATH=/home/fpoulin/software/shenfun/:$PYTHONPATH pip install .

[mikaem]

Great that the local build works! BTW, what is failing? And why the final pip install .? You don't need to install systemwide if you modify the pythonpath as you do.

I don't think there's any help to be gained from conda-forge. You would have to ask for help on the shenfun-feedstock, and I'm the maintainer there. In general, it is very difficult to get help if the error is not reproducible elsewhere.

[francispoulin]

Ah, thanks for pointing that out. My mistake. I will remove it using pip to fix that up.

There is a lot of read and I don't imagine you want to see all of that but I will copy the last example and then the warnings summary. If you wanted more please let me know how to share the information with you.

`> assert np.allclose(cs[s], cs2[s]) E assert False E + where False = <function allclose at 0x7ff778f83cb0>(array([[[ -0.7461949 -0.7461949j , -0.7461949 -0.7461949j ,\n -0.7461949 -0.7461949j , -0.7461949 -0.746...-27.31219413-27.31219413j, -27.31219413-27.31219413j,\n -27.31219413-27.31219413j, -27.31219413-27.31219413j]]]), array([[[0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],\n [0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],\n [0.+0.j, 0.+0.j, 0.+0.j, 0...[0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],\n [0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],\n [0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j]]])) E + where <function allclose at 0x7ff778f83cb0> = np.allclose

tests/test_transforms.py:407: AssertionError ================================ warnings summary ================================= shenfun/forms/arguments.py:700 /home/fpoulin/software/shenfun/shenfun/forms/arguments.py:700: PytestCollectionWarning: cannot collect test class 'TestFunction' because it has a init constructor (from: tests/test_bases.py) class TestFunction(BasisFunction):

shenfun/forms/arguments.py:700 /home/fpoulin/software/shenfun/shenfun/forms/arguments.py:700: PytestCollectionWarning: cannot collect test class 'TestFunction' because it has a init constructor (from: tests/test_curl.py) class TestFunction(BasisFunction):

shenfun/forms/arguments.py:700 /home/fpoulin/software/shenfun/shenfun/forms/arguments.py:700: PytestCollectionWarning: cannot collect test class 'TestFunction' because it has a init constructor (from: tests/test_fourier.py) class TestFunction(BasisFunction):

shenfun/forms/arguments.py:700 /home/fpoulin/software/shenfun/shenfun/forms/arguments.py:700: PytestCollectionWarning: cannot collect test class 'TestFunction' because it has a init constructor (from: tests/test_io.py) class TestFunction(BasisFunction):

tests/test_bases.py::test_eval_basis_derivative[Basis2] tests/test_bases.py::test_eval_basis_derivative[Basis3] tests/test_bases.py::test_eval_basis_derivative[ShenDirichletBasis2] /home/fpoulin/software/shenfun/shenfun/spectralbase.py:591: UserWarning: Using slow sympy evaluate_basis_derivative warnings.warn('Using slow sympy evaluate_basis_derivative')

tests/test_matrices.py::test_cmatvec[0-2-self-GC-Basis-Basis] tests/test_matrices.py::test_cmatvec[0-2-self-GL-Basis-Basis] tests/test_matrices.py::test_cmatvec[0-3-self-GC-Basis-Basis] tests/test_matrices.py::test_cmatvec[0-3-self-GL-Basis-Basis] /home/fpoulin/software/shenfun/shenfun/chebyshev/matrices.py:369: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result. c[d] = self[0][s]*v[d]

-- Docs: https://docs.pytest.org/en/latest/warnings.html ============ 184 failed, 6849 passed, 11 warnings in 84.30s (0:01:24) ============= `

[mikaem]

Guess what, the original error actually showed up in one of the eight test configurations on azure-pipelines! So then it's not just your environment and I have a better chance of figuring out what goes wrong.

[mikaem]

Fixed it! Was a cython issue that had shown up until now as a warning. Rewrote to get rid of the warning and the segfault disappeared, at least on azure-pipelines.

[francispoulin]

Well, that was a very fast fix. Well done!

I have updated my cloned repo (git update) and still get the same results as before.

Is there a different way of updating it?

[mikaem]

You need to rebuild 'python setup.py build_ext -i'

[mikaem]

And first 'git pull'

[francispoulin]

Of course. Rebuilding it. Yes, I did that now.

Unfortunately, the number of errors still seems to be be 184, and 11 warnings.

Very strange.

[mikaem]

Yes. That's something else. I need to see some error messages to decide what exactly.

[francispoulin]

Ah, I think I see at least one problem. In this build I don't have parallel h5py installed. I have it in my mpi4pi-fft conda installation, but that is removed from here.

Would there be an easy way to build that as well?

[mikaem]

You should activate the shenfun environment, which has all the correct dependencies. With local shenfun path first on the pythonpath, it does not matter that an unused shenfun installation is still around.

[francispoulin]

Good news: I did the conda install and it pass 7033 with 17 warnings and no failures.

I learned that if I try and import shenfun while in the cloned directory it fails, but outside it works fine.

Weird news: when I unset my PYTHONPATH I can now import shenfun.

Maybe it's working fine now, for reasons I can't understand.

Any thoughts?

[mikaem]

Great news😃 I released a bug fix version on conda forge, so if you updated somehow that one should also be fine now👍

[francispoulin]

That is great news. I am glad that we were able to get to the source and you could fix it.

I will do a conda update and presume things will only get better. I am closing this issue since things are now looking good.

[francispoulin]

I have recently installed shenfun on my new ubuntu 20.04 system and when I ran the tests I found there were 11 errors, which reminded me of this old thread.

I went back and saw tha tI needed to run the setup before runnin the tests. Unfortunately, when I try this I get the following error. The good news is that when I tried an example in demos it seemed to work fine but I thought I would point this out in case others might have a similar issue.

    $python setup.py build_ext -i
    running build_ext
    building 'shenfun.optimization.cython.Matvec' extension
    error: unknown file type '.pyx' (from '/home/fpoulin/software/shenfun/shenfun/optimization/cython/Matvec.pyx')

[mikaem]

You need to install cython. Good that you report these errors.

[mikaem]

BTW. What 11 errors? This needs to work out of the box on a new ubuntu 20.04, there should be no need for compiling yourself unless you want to do some developing.

[francispoulin]

Thanks! I feel like i might have done this befor but forgot.

It seems that I can't read properly. Below I had 12 errors and this is the output.

========================== short test summary info ========================== ERROR tests/test_backward.py ERROR tests/test_bases.py ERROR tests/test_curl.py ERROR tests/test_curvilinear.py ERROR tests/test_forms.py ERROR tests/test_fourier.py ERROR tests/test_io.py ERROR tests/test_la.py ERROR tests/test_lagrangian_particles.py ERROR tests/test_matrices.py ERROR tests/test_tensorproductspace.py ERROR tests/test_transforms.py !!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!! ============================ 12 errors in 0.92s =============================

After I compiled cython, there were 3336 passes, 1 skipped and 14 warnings. Do you want me to send you more information?

Are you saying this should have worked without doing the build? I installed cython and built then tried it, but can try something else if you like?

[mikaem]

If you use conda it should work without running a local build. Perhaps you don't have pytest in your shenfun conda environment? That could cause an error with collection I think?

[francispoulin]

Thanks! I recreated my venv and added cython in the list and I believe that the tests are now passing, without having to do any lists.

Now to play with the optimization!

[mikaem]

Ok, great:-) For what it's worth I just downloaded a new ubuntu 20.04 docker container. Installed miniconda and then shenfun+pytest from conda-forge. Cloned this repository and ran all the tests with no mistake.

[francispoulin]

Great to hear it worked so well for you. You didn't even get one skipped? That is the best that I've had so far.

On a different note, I tried using the following and whether I have both or just CYTHON, I get a lot of failures.

export SHENFUN_OPTIMIZATION={CYTHON,NUMBA}

If I don't use this I presume that shenfun will just use cython? If so then I will skip this one. When I try it I get 217 failed, 1 skipped and 54 warnings.

[mikaem]

CYTHON is default, so setting

export SHENFUN_OPTIMIZATION=CYTHON

should do nothing. However

export SHENFUN_OPTIMIZATION=NUMBA

should use numba instead of cython for the optimized routines. You geting errors with numba? Do you have numba installed?

[francispoulin]

I don't have numba installed, but I probably should just in case, but when I set it to CYTHON I get failures where otherwise there are none.

I suppose that's not supposed to happen?

Two examples that fial are test_curl.py and test_demos.py but suspect there are more. I'm rerunning it now to find out.

[mikaem]

I do not get errors setting SHENFUN_OPTIMIZATION=CYTHON. What are the error messages?

[francispoulin]

I think this is a matter of me having bad syntax, I'm sorry.

I agree that when I try this, it works fine.

export SHENFUN_OPTIMIZATION=CYTHON

Following what I saw on the docs, I had a silghtly different RHS, and this it complained about.

export SHENFUN_OPTIMIZATION={CYTHON}

I now see that this is not the right syntax. Sorry for my silly misake and thanks for helping me to figure it out!