ValueError: numpy.ndarray size changed, may indicate binary incompatibility.

[HaraldWilhelmi]

When installing the circtools in a Python3 venv, I got a inconsistent numpy setup. That can be seen with that example:

./venv/bin/circtools detect

Expected behavior:

Output folder ./ already exists, reusing
DCC 0.5.0 started
40 CPU cores available, using 2
ERROR: Required file detect is missing, exiting

Actual behavior:

Traceback (most recent call last):
  File "/beegfs/biosw/circtools/1.2.1/venv/bin/circtools", line 18, in <module>
    import circtools
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/circtools/__init__.py", line 2, in <module>
    main()
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/circtools/circtools.py", line 31, in main
    CircTools()
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/circtools/circtools.py", line 77, in __init__
    getattr(self, args.command)()
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/circtools/circtools.py", line 608, in detect
    import DCC
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/DCC/__init__.py", line 3, in <module>
    from .circFilter import Circfilter
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/DCC/circFilter.py", line 5, in <module>
    import HTSeq
  File "/beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages/HTSeq/__init__.py", line 13, in <module>
    from HTSeq._HTSeq import *
  File "src/HTSeq/_HTSeq.pyx", line 1, in init HTSeq._HTSeq
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 80 from PyObject

Some addition facts:

• circtools: 1.2.1
• OS: Debian 10
• Python: 3.9.5
• numpy (at the end of the installation): 1.16.6

Also I did already some research:

• https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp This suggests that the issue was caused by downgrading numpy during the installation. Will check if a workaround can be build from this.

[HaraldWilhelmi]

Uninstalling numpy 1.16.6 before did not help.

Also upgrading numpy is not an option:

$ pip install --upgrade numpy
Requirement already satisfied: numpy in /beegfs/biosw/circtools/1.2.1/venv/lib/python3.9/site-packages (1.16.6)
Collecting numpy
  Using cached numpy-1.22.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.16.6
    Uninstalling numpy-1.16.6:
      Successfully uninstalled numpy-1.16.6
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
circtools 1.2.0 requires biopython>=1.71, which is not installed.
circtools 1.2.0 requires pandas>=0.25.0, which is not installed.
circtools 1.2.0 requires reportlab>=3.3.0, which is not installed.
circtools 1.2.0 requires scipy>=0.19.0, which is not installed.
circtools 1.2.0 requires statsmodels>=0.9.0, which is not installed.
fuchs 0.3.0 requires numpy<1.17.0, but you have numpy 1.22.3 which is incompatible.
Successfully installed numpy-1.22.3

Actually I diverted in two points from the official instructions:

• Upgraded pip
• Installed wheel Will test tomorrow if that makes a difference.

[HaraldWilhelmi]

Using the default pip and running without wheel made no difference.

[HaraldWilhelmi]

After looking closer at the stack backtrace and comparing my setup with an older installation, I decided to to try that as a workaround:

pip install HTSeq==0.13.5

That looks promising but I have to compile from scratch to be certain.

[HaraldWilhelmi]

Suggested workaround works. What would be a good final fix? The requirement for the old numpy version seems to come from FUCHS. Modernizing here might be the most consequent solution. Integrating the workaround into the installation processes of circtools may be cheaper however.

[tjakobi]

Hi Harald,I went through my notes for FUCHS and I don't think there is a reason we need to keep numpy locked for FUCHS. I'll commit a change to the FUCHS repository to use the latest numpy which in turn should also fix the DCC error.

[HaraldWilhelmi]

Forgot about this bug (and was happy with my workaround). After installing circtools again, I noticed that I no longer need the workaround. So this issue is most certainly resolved.