search

NicolasHug / Surprise

A Python scikit for building and analyzing recommender systems
http://surpriselib.com
BSD 3-Clause "New" or "Revised" License
6.34k stars 1.01k forks source link

Surprise ImportError #399

Open patwanjau opened 2 years ago

patwanjau commented 2 years ago

Description

I am working on a recommender systems project and Surprise has been used as the preferred package in my examples. However, I'm experiencing problems when I use Surprise. There results in an import error attributed to the cpython library. Below is the error encountered:

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/home/user/.local/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File "/home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: /home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

Steps/Code to Reproduce

python -c 'import surprise; print("surprise", surprise.__version__)'

Expected Results

1.1.1

Actual Results

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/home/user/.local/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File "/home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: /home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

Versions

Linux-5.15.12-200.fc35.x86_64-x86_64-with-glibc2.34 Python 3.10.1 (main, Dec 9 2021, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)]

Getting the version for surprise failed when executed in the code for "Steps to reproduce"

ProfHercules commented 2 years ago

What version of Cython are you running?

Seems that the issue stems from a slightly older version of Cython being incompatible with Python 3.10.1, I found reference to it in the Cython repo issues.

If there is an update to your version of Cython available, install that. Otherwise, there are some instructions in the linked issue around building the latest release of Cython from source.

Alternatively, if possible, try using Python < 3.10.1 (maybe 3.9)

patwanjau commented 2 years ago

What version of Cython are you running?

Seems that the issue stems from a slightly older version of Cython being incompatible with Python 3.10.1, I found reference to it in the Cython repo issues.

If there is an update to your version of Cython available, install that. Otherwise, there are some instructions in the linked issue around building the latest release of Cython from source.

Alternatively, if possible, try using Python < 3.10.1 (maybe 3.9)

  1. I had not installed any Cython libraries, therefore, no Cython updates were available.
  2. After your response, I installed the latest version of Cython via pip (version 0.29.26) and the same issue arose

I will attempt to install from source as downgrading Python is not an option at the moment.

ProfHercules commented 2 years ago

I just realised that what you're downloading when you install Surprise may be a pre-compiled wheel, which would have been compiled with an older version of Cython.

Perhaps try this:

For the latest version, you can also clone the repo and build the source (you'll first need Cython and numpy):

pip install numpy cython
git clone https://github.com/NicolasHug/surprise.git
cd surprise
python setup.py install

(from the README)

It should compile Surprise with the new version of Cython you have installed.

mskiptr commented 2 years ago

I hit the same issue. Python: python-3.10.2-1 from Arch Linux

scikit-surprise from pip within a venv:

(.venv) [piotr@andrad ...]$ python -c 'import surprise; print("surprise", surprise.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../.venv/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File ".../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: .../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so:
undefined symbol: _PyGen_Send

Then I did:

(.venv) [piotr@andrad ...]$ pip install numpy cython
(.venv) [piotr@andrad ...]$ git clone https://github.com/NicolasHug/surprise.git
(.venv) [piotr@andrad ...]$ cd surprise
(.venv) [piotr@andrad .../surprise]$ python setup.py install
End of the build log ``` creating dist creating 'dist/scikit_surprise-1.1.1-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing scikit_surprise-1.1.1-py3.10-linux-x86_64.egg creating .../.venv/lib/python3.10/site-packages/scikit_surprise-1.1.1-py3.10-linux-x86_64.egg Extracting scikit_surprise-1.1.1-py3.10-linux-x86_64.egg to .../.venv/lib/python3.10/site-packages Adding scikit-surprise 1.1.1 to easy-install.pth file Installing surprise script to .../.venv/bin Installed .../.venv/lib/python3.10/site-packages/scikit_surprise-1.1.1-py3.10-linux-x86_64.egg Processing dependencies for scikit-surprise==1.1.1 Searching for six==1.16.0 Best match: six 1.16.0 Adding six 1.16.0 to easy-install.pth file Using .../.venv/lib/python3.10/site-packages Searching for scipy==1.8.0 Best match: scipy 1.8.0 Adding scipy 1.8.0 to easy-install.pth file Using .../.venv/lib/python3.10/site-packages Searching for numpy==1.22.3 Best match: numpy 1.22.3 Adding numpy 1.22.3 to easy-install.pth file Installing f2py script to .../.venv/bin Installing f2py3 script to .../.venv/bin Installing f2py3.10 script to .../.venv/bin Using .../.venv/lib/python3.10/site-packages Searching for joblib==1.1.0 Best match: joblib 1.1.0 Adding joblib 1.1.0 to easy-install.pth file Using .../.venv/lib/python3.10/site-packages Finished processing dependencies for scikit-surprise==1.1.1 ```

After that I tried again:

(.venv) [piotr@andrad .../surprise]$ python -c 'import surprise; print("surprise", surprise.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../surprise/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File ".../surprise/surprise/prediction_algorithms/__init__.py", line 23, in <module>
    from .algo_base import AlgoBase
  File ".../surprise/surprise/prediction_algorithms/algo_base.py", line 10, in <module>
    from .. import similarities as sims
ImportError: cannot import name 'similarities' from partially initialized module 'surprise' (most likely due to a circular import) (.../surprise/surprise/__init__.py)
(.venv) [piotr@andrad .../surprise]$ cd ..
(.venv) [piotr@andrad ...]$ python -c 'import surprise; print("surprise", surprise.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../.venv/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File ".../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: .../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so:
undefined symbol: _PyGen_Send

I don't think I understand the problem and I can't really dig deeper into it right now, but maybe someone more knowledgeable can make something out of it. Btw, this issue seems to be quite similar.

Recreating the venv with Python 3.9 (python39-3.9.11-3 from AUR) alleviated the issue.

(.venv) [piotr@andrad ...]$ python -c 'import surprise; print("surprise", surprise.__version__)'
surprise 1.1.1
jaklan commented 2 years ago

We also had a problem with importing surprise, but we discovered the issue was with numpy version:

So it seems the numpy requirement: https://github.com/NicolasHug/Surprise/blob/46b9914995e6c8c7d227b46f2eaeef2d4600580f/requirements.txt#L2 is too loose and it should be more like numpy>=1.22 (or at least it should exclude version range around 1.20)