Potential bug: Incompatibility with NumPy 2.0 leads to AttributeError

[DaniJonesOcean]

Description

The recent upgrade to NumPy 2.0.0 introduces a breaking change due to the removal of np.sctypes. This update has resulted in deepsensor raising an AttributeError upon import when paired with the new version of NumPy. This indicates that the current deepsensor codebase is not compatible with NumPy 2.0.0's updated API, causing it to break. Indeed, the documentation for NumPy 2.0.0 indicates that np.sctypes has indeed been removed from the namespace:

https://numpy.org/devdocs/numpy_2_0_migration_guide.html#changes-to-namespaces

Here is the full traceback received after trying to import deepsensor.torch:

Traceback (most recent call last):
  File "~/DeepSensor/NumPy-issue/deepsensor_numpy_test.py", line 1, in <module>
    import deepsensor.torch
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/__init__.py", line 20, in <module>
    from .data.processor import DataProcessor
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/__init__.py", line 2, in <module>
    from .loader import TaskLoader
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/loader.py", line 1, in <module>
    from deepsensor.data.task import Task, flatten_X
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/task.py", line 5, in <module>
    import lab as B
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/__init__.py", line 9, in <module>
    from .generic import *
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/generic.py", line 11, in <module>
    from .types import (
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/types.py", line 99, in <module>
    Int = Union[tuple([int, Dimension] + np.sctypes["int"] + np.sctypes["uint"])]
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.sctypes` was removed in the NumPy 2.0 release. Access dtypes explicitly instead.. Did you mean: 'dtypes'?

Sorry to be the bearer of bad news @wesselb, but the error seems to come from the lab package when it tries to access a numpy function. This breaking change, which has also broken lab, could be reported as a separate issue here:

https://github.com/wesselb/lab/issues

This error will affect users who pip install deepsensor using the repository in its current form. One straightforward solution would be pinning the NumPy version in the requirements/requirements*.txt files to the most recent compatible version (i.e. numpy<=1.26.4 or numpy<2).

Reproduction steps

1. Create a new virtual environment for testing (e.g. `deepsensor_numpy_test`, as seen above)
2. pip install `deepsensor` 0.3.6 and its dependencies (including NumPy 2.0.0)
3. Try to import deepsensor code (e.g. `import deepsensor.torch`)
4. Get the Traceback shown above

Version

I am using python 3.10.9. Here is a truncated package list from pip list for completeness:

dask                     2024.6.2
deepsensor               0.3.6
matplotlib               3.9.0
netCDF4                  1.7.1
NeuralProcesses          0.2.6
numpy                    2.0.0
pandas                   2.2.2
pip                      24.1
scikit-learn             1.5.0
scipy                    1.13.1
seaborn                  0.13.2
torch                    2.3.0
xarray                   2024.6.0

The breaking change appears to affect the lab package, which is not pip installed. I found the version number in my virtual environment here: ~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/_version.py, indicating that I'm using the latest version of lab==1.6.5.

OS

Linux

[DaniJonesOcean]

Posted a related issue for the lab package here:

https://github.com/wesselb/lab/issues/20

[wesselb]

This should be fixed in backends==1.6.6.

[DaniJonesOcean]

Thanks, @wesselb. I updated to numpy==2.0.0 and backends==1.6.6, and the AttributeError did not appear again, so that's good news.

Unfortunately, there is a new error when you run import deepsensor.torch:

Traceback (most recent call last):
  File "/home/dannes/DeepSensor/NumPy-issue/deepsensor_numpy_test.py", line 1, in <module>
    import deepsensor.torch
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/__init__.py", l
ine 20, in <module>
    from .data.processor import DataProcessor
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/__init__.p
y", line 2, in <module>
    from .loader import TaskLoader
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/loader.py"
, line 1, in <module>
    from deepsensor.data.task import Task, flatten_X
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/task.py", 
line 5, in <module>
    import lab as B
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/lab/__init__.py", line 9, 
in <module>
    from .generic import *
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/lab/generic.py", line 11, 
in <module>
    from .types import (
  File "/home/dannes/deepsensor_numpy_test/lib/python3.10/site-packages/lab/types.py", line 99, in
 <module>
    Int = Union[tuple([int, Dimension] + np.dtypes["int"] + np.dtypes["uint"])]
TypeError: 'module' object is not subscriptable

I hate to suggest it, but I wonder if migrating to NumPy 2.0 will be a moderately big undertaking? There were a lot of changes. Anything I can do to support this effort?

[wesselb]

@DaniJonesOcean Are you sure that you are running backends==1.6.6? The line that the traceback is showing corresponds to an older version. C.f., these are the current line 99 and surrounding lines:

# Numeric types:
Int = Union[tuple([int, Dimension] + np.core.sctypes["int"] + np.core.sctypes["uint"])]
Int = set_union_alias(Int, "B.Int")
Float = Union[tuple([float] + np.core.sctypes["float"])]

I hate to suggest it, but I wonder if migrating to NumPy 2.0 will be a moderately big undertaking? There were a lot of changes. Anything I can do to support this effort?

Possibly this is the case, yes. :( I'm currently unable to locally run PyTorch 2.3, which is required for NumPy 2, so this is difficult to debug. Any help would be greatly appreciated!

[DaniJonesOcean]

@wesselb When I run pip list, I get the following:

> pip list | grep backends
backends                 1.6.6
backends-matrix          1.3.0

So at least as far as pip is concerned, I'm running backends==1.6.6. (Full disclosure: I'm not really familiar with how packages like lab are bundled together under something like backends. So I might be missing something.)

Possibly this is the case, yes. :( I'm currently unable to locally run PyTorch 2.3, which is required for NumPy 2, so this is difficult to debug. Any help would be greatly appreciated!

I'll do what I can! Maybe other can hop in and assist as well?

[wesselb]

@DaniJonesOcean Could you try reinstalling in a fresh environment? This looks like something has gone wrong. Line 99, which gives the error, isn't what it should be (I double checked the package on PyPI).

I'll do what I can! Maybe other can hop in and assist as well?

Thank you! :) Once a little bit of time frees up for me, I should be able to start digging into this too.

[wesselb]

@DaniJonesOcean, I've digged into this, and I think backends==1.6.6 should be compatible with NumPy 2.0. There is one incompability in a dependency fdm. You can locally hotfix this as follows:

virtualenv -p python3.9 venv
. ./venv/bin/activate
pip install torch neuralprocesses
sed -i 's/np\.math/np/g' ./venv/lib/python3.9/site-packages/fdm/fdm.py
sed -i 's/np\.factorial/math.factorial/g' ./venv/lib/python3.9/site-packages/fdm/fdm.py
sed -i '1s/^/import math\n/' ./venv/lib/python3.9/site-packages/fdm/fdm.py

After this, the following snippet should run correctly:

import numpy as np
import torch

import neuralprocesses.torch as nps

print(np.__version__)  # 2.0.0
print(torch.__version__)  # 2.3.1+cu121

model = nps.construct_convgnp()
pred = model(torch.randn(1, 1, 4), torch.randn(1, 1, 4), torch.randn(1, 1, 10))

# Runs OK!

I will release a fix for fdm soon. For the moment, you can use the above hotfix.

[DaniJonesOcean]

Fantastic, thank you @wesselb!