Import error from numpy compilation

[PauBadiaM]

Hi,

First of all I wanted to thank you for this amazing package, this implementation of Fisher's exact test is insanely fast!

I'm developing a package and I would like to include fisher as a dependancy. The problem is, that recently (2022/01/31) my GitHub's Continuous Integration started breaking apart (I found out by updating some documentation, so not changing any dependencies), this is the error:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/main.py", line 147, in parseArgs
    self.createTests()
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/home/runner/work/decoupler-py/decoupler-py/tests/test_methods.py", line 2, in <module>
    import decoupler as dc
  File "/home/runner/work/decoupler-py/decoupler-py/decoupler/__init__.py", line 7, in <module>
    from .method_ora import run_ora
  File "/home/runner/work/decoupler-py/decoupler-py/decoupler/method_ora.py", line 14, in <module>
    from fisher import pvalue_npy
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/fisher/__init__.py", line 3, in <module>
    from .cfisher import *
  File "__init__.pxd", line 242, in init cfisher
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
Error: Process completed with exit code 1.

Basically it breaks when importing fisher. Since the packages are exactly the same version as before, I'm very puzzled on what could be happening. Apparently is something about numpy being compiled in different versions across packages. The dependencies installed are:

Package           Version
----------------- -------
anndata           0.7.8
bleach            3.3.0
decoupler         0.0.1
fisher            0.1.9
h5py              3.6.0
jeepney           0.6.0
joblib            1.1.0
keyring           23.0.1
llvmlite          0.37.0
natsort           8.1.0
numba             0.54.1
numpy             1.20.3
packaging         21.3
pandas            1.4.0
pip               22.0.3
pkginfo           1.7.0
pyparsing         3.0.7
python-dateutil   2.8.2
pytz              2021.3
readme-renderer   29.0
requests-toolbelt 0.9.1
rfc3986           1.4.0
scikit-learn      0.24.2
scikit-misc       0.1.3
scipy             1.8.0
SecretStorage     3.3.1
setuptools        60.7.1
six               1.16.0
sklearn           0.0
skranger          0.7.0
threadpoolctl     3.1.0
tqdm              4.62.3
webencodings      0.5.1
wheel             0.36.2
xlrd              1.2.0

I'm installing the package via pip, could this be a problem? Have you seen something similar, or could you know where the error is coming from?

Thank you for your time

[brentp]

Hi, I just upload v0.1.10 to pypi can you see if that resolves this?

[PauBadiaM]

Thanks for the quick reply @brentp!

Unfortunately the error still persists with version 1.10. What is very puzzling is that installation in several local machines (all different OS) works without a problem, but it fails when running through GitHub Actions. I know this is out out of the scope, but am I doing something wrong here?

This is my setup.py file:

from setuptools import setup, find_packages

setup(
    name="decoupler",
    version="0.0.1",
    install_requires=["numba",
                      "sklearn",
                      "tqdm",
                      "anndata",
                      "skranger",
                      "fisher"
                     ],
    packages=["decoupler"],
    python_requires=">=3.8",
    classifiers=[
        "Programming Language :: Python :: 3",
        "Operating System :: OS Independent",
    ]
)

And my .yml file

name: build

on:
  push:
    branches: [ main, development ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: 3.8
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install wheel
        pip install .
    - name: Test with unnittest
      run: |
        python -m unittest tests/test_*

Maybe the error could be coming from an update in actions/setup-python@v2 or actions/checkout@v2?

[brentp]

Not sure, are you using python 3.8 in your local setup?

[PauBadiaM]

Yes, exactly the same Python and dependencies' versions.

I have created a toy repository to reproduce the error, here: https://github.com/PauBadiaM/test_CI_fisher

Again, in local when I run the test it works fine, but when GitHub's CI tries it, it crashes with the same error: https://github.com/PauBadiaM/test_CI_fisher/runs/5106465412?check_suite_focus=true

I was testing and removing numba seems to work: https://github.com/PauBadiaM/test_CI_fisher/runs/5106492033?check_suite_focus=true

What is weird is that the numba version is the same in my local machines than in GitHub's, so I'm at a loss of what could be happening.

[brentp]

must be some difference in the numpy versions. Can you check that your repo has same numpy as local?

[PauBadiaM]

From a scratch conda environment:

conda create -n test_CI python=3.8
conda activate test_CI

If I add numba as a requirement in setup.py and install it with:

pip install git+https://github.com/PauBadiaM/test_CI_fisher.git

Machine	Package	Version
Local	numba	0.55.1
GitHub's	numba	0.55.1
Local	numpy	1.21.5
GitHub's	numpy	1.21.5
Local	fisher	0.1.9
GitHub's	fisher	0.1.9

As you can see the versions are exactly the same, however on local python -m unittest tests/test.py works but not on GitHub's machine. Interestingly, if I bump fisher's version to 0.1.10 it crashes in my local too.

I also tried different versions of numpy in both local an in GitHub's machine. With a fixed fisher==0.1.9, versions of numpy from 1.21.5 to 1.20.0 (included) work on local but crash in GitHub's. numpy's version 1.19.0 crashed on local so I didn't check any further versions.