zarr-developers / zarr-python

An implementation of chunked, compressed, N-dimensional arrays for Python.
http://zarr.readthedocs.io/
MIT License
1.41k stars 267 forks source link

Numpy 2.0 has broken some (admittedly malformed) .tif files in decode_fill_value #1981

Open NewSoupVi opened 3 weeks ago

NewSoupVi commented 3 weeks ago

Zarr version

2.18.2

Numcodecs version

0.12.1

Python Version

3.11

Operating System

Windows

Installation

pip venv

Description

I have a .tif file with dtype uint8, that reports a fill_value of -999999999

gdalinfo output for reference: image

Obviously, this is impossible, meaning the tif file is malformed. But it did "just work" previously. This line just returned 1, and the program would continue running happily.

Numpy 2.0, which came out yesterday, has made a change, though, that makes that line error with:

Traceback (most recent call last):
  File "D:\enpgt-toolbox\venv\Lib\site-packages\zarr\meta.py", line 127, in decode_array_metadata
    fill_value = cls.decode_fill_value(meta["fill_value"], dtype, object_codec)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\enpgt-toolbox\venv\Lib\site-packages\zarr\meta.py", line 260, in decode_fill_value
    return np.array(v, dtype=dtype)[()]
           ^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: Python integer -999999999 out of bounds for uint8

I'm not sure whether this is "your problem" necessarily, but it'd be nice if there were some sort of way around this change. I don't actually care about the fill value, I just want to be able to read this .tif normally, without having to make modifications to it. Which may be an "unreasonable goal", but I figured I'd at least ask / bring it up.

Steps to reproduce

These suck a little bit, because the tif file in question is huge and requires a login to download, and this is through a different package

Additional output

No response

NewSoupVi commented 3 weeks ago

Here are some better reproduction steps.

https://github.com/cgohlke/tifffile/issues/260#issuecomment-2176420795