search

neuronsimulator / nrn

NEURON Simulator
http://nrn.readthedocs.io
Other
401 stars 116 forks source link

Cython <3 not found #2549

Open SteMasoli opened 1 year ago

SteMasoli commented 1 year ago

Context

Overview of the issue

I'm trying to build the latest NEURON 9 code with RxD, on a brand new installation of Mint 21.2. Cython is 0.29.36 and it is found during the configuration. But during the compilation of RxD is says that there is no module named Cython

Expected result/behavior

Since Cython version is <3 and it is found during the configuration, it should not stop the compilation of RxD.

NEURON setup

cmake .. -DCMAKE_INSTALL_PREFIX=/nrn9_cmake \ -DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DNRN_ENABLE_INTERVIEWS=ON \ -DNRN_ENABLE_MPI=ON \ -DNRN_ENABLE_PYTHON=ON \ -DNRN_ENABLE_RX3D:BOOL=ON \ -DCORENRN_ENABLE_NMODL=OFF \ -DNRN_ENABLE_CORENEURON=OFF make make install


## Logs

> -- Found Cython: .local/bin/cython
> CMake Error at cmake/modules/FindCython.cmake:43 (message):
>   Cython error [1]: Traceback (most recent call last):
> 
>     File ".local/bin/cython", line 5, in <module>
>       from Cython.Compiler.Main import setuptools_main
> 
>   ModuleNotFoundError: No module named 'Cython'.
> Call Stack (most recent call first):
>   CMakeLists.txt:319 (find_package)

If I load the same command "from Cython.Compiler.Main import setuptools_main" in python, there are no errors.
nrnhines commented 1 year ago

I'm installing a mint-21.2 virtualbox guest and will look into this. Will get back to you when I know more.

1uc commented 1 year ago

Something that catches me out often is not deleting the build folder when reconfiguring the build with CMake. Whenever I need to build the docs or RxD, I use a Python virtual environment, because my system Cython is >=3:

python -m venv venv
source venv/bin/activate
pip install -r nrn_requirements.txt
pip install -r external/nmodl/requirements.txt
pip install -r docs/doc_requirements.txt

cmake -DCMAKE_INSTALL_PREFIX=build-2549/install -DCMAKE_BUILD_TYPE=Release -B build-2549 .
cmake --build build-2549 --parallel

(I'm aware that this will likely be of no use to either of you.)

nrnhines commented 1 year ago

@1uc comment may be the hint you need. I see after executing the cmake command in the build folder that

hines@mint21:~/neuron/nrn/build$ grep -r -i cython
CMakeCache.txt: CYTHON_EXECUTABLE:FILEPATH=/usr/local/bin/cython
...
hines@mint21:~/neuron/nrn/build$ /usr/local/bin/cython --version
Cython version 0.29.36

and the succeeding make install was successful.

From

-- Found Cython: .local/bin/cython

I presume CYTHON_EXECUTABLE in CMakeCache is specified as the full path to .local/bin/cython and that version really is 0.29.36.

nrnhines commented 1 year ago

Just for concreteness, after installing mint-21.2, I setup the development environment in preparation for the cmake command (many mistakes elided) using:

sudo apt update
sudo apt install cmake git libncurses-dev libopenmpi-dev libx11-dev libxcomposite-dev openmpi-bin python3-dev
sudo apt install python3-pip
sudo pip3 install scipy numpy 'cython<3' # mistake to use sudo here. But otherwise need to add .local/bin to path.
sudo apt install libreadline-dev # previous cmake told me it was missing
cmake .. -DCMAKE_INSTALL_PREFIX=install  # all your specific args were in fact defaults
make install
install/bin/neurondemo
nrnhines commented 1 year ago

After

$ sudo pip uninstall cython
$ pip install 'cython<3'
$ export PATH=~/.local/bin:$PATH

I observe

hines@mint21:~/neuron/nrn/build$ cmake .. -DCMAKE_INSTALL_PREFIX=install
...
-- Found Cython: /home/hines/.local/bin/cython (found version "0.29.36")

Note the full path for cython. Without the export PATH I observe (starting from empty build folder)

$ cmake .. -DCMAKE_INSTALL_PREFIX=install
...
-- Found Readline: /usr/include  
CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Cython (missing: CYTHON_EXECUTABLE)

I guess I don't understand the circumstances that could generate a

-- Found Cython: .local/bin/cython
1uc commented 1 year ago

In more recent versions of pip this line:

$ pip install 'cython<3'

is equivalent to (at least if one doesn't have write permission for the systemwide location):

$ pip install --user 'cython<3'

and will (I believe) install cython into ${HOME}/.local/bin/cython; and not the systemwide location.

1uc commented 1 year ago

My google-foo is low. The only thing I could find to back up the above is: https://stackoverflow.com/a/39536914

nrnhines commented 1 year ago

@1uc That is what I observe if you mean (precisely) $HOME/.local/bin/cython When I installed via sudo pip install 'cython<3' the module ended up in /usr/local/lib/python3.10/dist-packages and the executable in /usr/local/bin. By the way sudo pip install 'cython<3' is successful but generates the scary message:

Successfully installed cython-0.29.36 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

1uc commented 1 year ago

Short version: Yes, I've updated the comment accordingly; it was misleading.

Long version: This is precisely the difference between pip install --user (equivalent to pip install without sudo rights); and sudo pip install. The former (pip install --user) will install packages in a user site location; and executables in $HOME/.local/bin/cython. The latter (sudo pip install) will install into a system-wide location, which is /usr/local/lib/... for packages and /usr/local/bin for executables (on that VM). Note that it will overwrite packages installed by the system package manager. This will eventually cause issues, because pip and the system package manager will start disagreeing, resulting in broken Python installations. I'd strongly recommend against sudo pip install. The two good alternatives are:

In the near future pip install --user will often be unavailable by default, see https://peps.python.org/pep-0668/#motivation

(Summary: even installing into a user-site can cause packages and Python scripts that have been installed system-wide to fail. This can be unpleasant if that system-wide script is something like the package manager.)

nrnhines commented 1 year ago

Basically, it seems this whole issue boils down to the question, how was it possible for

> -- Found Cython: .local/bin/cython

not to be be a full path. A mere curiosity question is, given cmake's FindCython was satisfied with .local/bin/cython, then what was the current working directory when

> CMake Error at cmake/modules/FindCython.cmake:43 (message):
>   Cython error [1]: Traceback (most recent call last):
> 
>     File ".local/bin/cython", line 5, in <module>
>       from Cython.Compiler.Main import setuptools_main
> 
>   ModuleNotFoundError: No module named 'Cython'.
> Call Stack (most recent call first):
>   CMakeLists.txt:319 (find_package)

occurred. I'm wondering if the underlying problem was merely

export PATH=.local/bin:$PATH

by the user??? instead of

export PATH=$HOME/.local/bin:$PATH.

I'll do that experiment. edit: even with

export PATH=.local/bin:$PATH
cd $HOME
cmake neuron/nrn -DCMAKE_INSTALL_PREFIX=install
...
- Found Cython: /home/hines/.local/bin/cython (found version "0.29.36")

so the puzzle is still there.

1uc commented 1 year ago

Oh, I'm sorry, I didn't spot that part. In that case, I've no idea.

SteMasoli commented 1 year ago

I tried to compile on my usual notebbok with Mint 21.2 but as an upgrade from version 20. Cython is older: 0.29.14, numpy is: 1.23.4, Python is 3.10.12.

Consolidate compiler generated dependencies of target nrniv
Building python module with: /usr/bin/python3 setup.py build
      --cmake-build-dir;/home/SteMasoli/nrn/build;--rx3d-opt-level;0;--without-nrnpython;--build-lib=/home/SteMasoli/nrn/build/lib/python build_ext;--define=USE_PYTHON,NRN_ENABLE_THREADS
[ 95%] Linking CXX executable ../../bin/nrniv
[ 96%] Linking CXX static library ../../../../lib/libunidraw.a
INFO:root:setup.py called with:setup.py build --cmake-build-dir /home/SteMasoli/nrn/build --rx3d-opt-level 0 --without-nrnpython --build-lib=/home/SteMasoli/nrn/build/lib/python build_ext --define=USE_PYTHON,NRN_ENABLE_THREADS
ERROR:root:ERROR: RX3D wheel requires Cython and numpy. Please install beforehand
make[2]: *** [src/nrnpython/CMakeFiles/hoc_module.dir/build.make:83: hoc_module] Error 1
make[1]: *** [CMakeFiles/Makefile2:1083: src/nrnpython/CMakeFiles/hoc_module.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 97%] Built target nrniv
[ 97%] Built target unidraw
make: *** [Makefile:136: all] Error 2
nrnhines commented 1 year ago

I've lost track of your build details. Please remind me of the cmake command you use. Also, it might help me to see the complete transcript of cmake ... after rm CMakeCache.txt

SteMasoli commented 1 year ago

I tried with 3 different computers: 1) Mint 21.2 fresh install - Issue of my first post where configure finds Cython but then it says it does not exist. 2) Mint 21.2 but upgraded from Mint 20. - Issue of my second post where Cython or numpy are "out of date". 3) Ubuntu 20.04 - NEURON 9 and RxD are fine.

I don't have an Ubuntu 22.04 to test but it seams that the issue is related to Mint.

Also, it might help me to see the complete transcript of cmake ... after rm CMakeCache.txt I'll find the files and attach them here.

SteMasoli commented 5 months ago

Sorry for the mega delay. Using V8.2.4, with the same Cmake as the first post:

Building python module with: /usr/bin/python3 setup.py build
      --cmake-build-dir;/home/bremen/nrn/build;--rx3d-opt-level;0;--without-nrnpython;--build-lib=/home/bremen/nrn/build/lib/python build_ext
INFO:root:setup.py called with:setup.py build --cmake-build-dir /home/bremen/nrn/build --rx3d-opt-level 0 --without-nrnpython --build-lib=/home/bremen/nrn/build/lib/python build_ext
ERROR:root:ERROR: RX3D wheel requires Cython and numpy. Please install beforehand
make[2]: *** [src/nrnpython/CMakeFiles/hoc_module.dir/build.make:87: hoc_module] Error 1
make[1]: *** [CMakeFiles/Makefile2:715: src/nrnpython/CMakeFiles/hoc_module.dir/all] Error 2

Cython is the "latest" 0.29.37 and Numpy is 1.23.3. Mint 21.3 brand new with the latest upgrades.

CMakeCache.txt attacched CMakeCache.txt

nrnhines commented 5 months ago

Perhaps we can arrange a zoom meeting and you can share your screen. We can communicate via email to set up a time. Michael.hines@yale.edu

SteMasoli commented 5 months ago

Under Mint 21.3, Cython 0.29.x is called cython3 (even thou it is not) and is not located under .local/bin, like with Ubuntu, but under /usr/bin. I created a symbolic link to cython3 (called cython like in Ubuntu) in .local/bin but it did not changed the result.

Then I checked the exception in setup.py.


if Components.RX3D:
    try:
        from Cython.Distutils import Extension as CyExtension
        from Cython.Distutils import build_ext
        import numpy
    except ImportError:
        logging.error(
            "ERROR: RX3D wheel requires Cython and numpy. Please install beforehand"
        )
        sys.exit(1)

If I manually pass each import to Python3.10.12, everything is fine. 

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from Cython.Distutils import Extension as CyExtension
>>> from Cython.Distutils import build_ext
>>> import numpy
>>> from setuptools.command.build_ext import build_ext
>>>

So numpy and Cython are installed correctly but, for an unknown reason, are not seen during the compile process.

SteMasoli commented 5 months ago

I tried on my other mint installation and, without "sudo make", everything is fine. I tested it with a code the uses RxD and there are no further issues.

Thank you