search

scikit-image / scikit-image

Image processing in Python
https://scikit-image.org
Other
6.05k stars 2.22k forks source link

Test failures on s390x #6994

Open opoplawski opened 1 year ago

opoplawski commented 1 year ago

Description:

Working on updating the Fedora scikit-image package to 0.21.0 I'm getting the following test failures on s390x only:

________________ TestSave.test_imsave_roundtrip[shape1-uint16] _________________
im = <PIL.Image.Image image mode=I;16B size=10x10 at 0x3FF46080AD0>
fp = <_io.BufferedWriter name='/tmp/pytest-of-mockbuild/pytest-0/test_imsave_roundtrip_shape1_u0/roundtrip.png'>
filename = '/tmp/pytest-of-mockbuild/pytest-0/test_imsave_roundtrip_shape1_u0/roundtrip.png'
chunk = <function putchunk at 0x3ff54a3f560>, save_all = False
    def _save(im, fp, filename, chunk=putchunk, save_all=False):
        # save an image to disk (called by the save method)

        if save_all:
            default_image = im.encoderinfo.get(
                "default_image", im.info.get("default_image")
            )
            modes = set()
            append_images = im.encoderinfo.get("append_images", [])
            if default_image:
                chain = itertools.chain(append_images)
            else:
                chain = itertools.chain([im], append_images)
            for im_seq in chain:
                for im_frame in ImageSequence.Iterator(im_seq):
                    modes.add(im_frame.mode)
            for mode in ("RGBA", "RGB", "P"):
                if mode in modes:
                    break
            else:
                mode = modes.pop()
        else:
            mode = im.mode

        if mode == "P":
            #
            # attempt to minimize storage requirements for palette images
            if "bits" in im.encoderinfo:
                # number of bits specified by user
                colors = min(1 << im.encoderinfo["bits"], 256)
            else:
                # check palette contents
                if im.palette:
                    colors = max(min(len(im.palette.getdata()[1]) // 3, 256), 1)
                else:
                    colors = 256

            if colors <= 16:
                if colors <= 2:
                    bits = 1
                elif colors <= 4:
                    bits = 2
                else:
                    bits = 4
                mode = f"{mode};{bits}"

        # encoder options
        im.encoderconfig = (
            im.encoderinfo.get("optimize", False),
            im.encoderinfo.get("compress_level", -1),
            im.encoderinfo.get("compress_type", -1),
            im.encoderinfo.get("dictionary", b""),
        )

        # get the corresponding PNG mode
        try:
>           rawmode, mode = _OUTMODES[mode]
E           KeyError: 'I;16B'
/usr/lib64/python3.11/site-packages/PIL/PngImagePlugin.py:1286: KeyError
The above exception was the direct cause of the following exception:
self = <skimage.io.tests.test_imageio.TestSave object at 0x3ff4d3eee90>
shape = (10, 10), dtype = <class 'numpy.uint16'>
tmp_path = PosixPath('/tmp/pytest-of-mockbuild/pytest-0/test_imsave_roundtrip_shape1_u0')
    @pytest.mark.parametrize(
        "shape,dtype", [
            # float32, float64 can't be saved as PNG and raise
            # uint32 is not roundtripping properly
            ((10, 10), np.uint8),
            ((10, 10), np.uint16),
            ((10, 10, 2), np.uint8),
            ((10, 10, 3), np.uint8),
            ((10, 10, 4), np.uint8),
        ]
    )
    def test_imsave_roundtrip(self, shape, dtype, tmp_path):
        if np.issubdtype(dtype, np.floating):
            min_ = 0
            max_ = 1
        else:
            min_ = 0
            max_ = np.iinfo(dtype).max
        expected = np.linspace(
            min_, max_,
            endpoint=True,
            num=np.prod(shape),
            dtype=dtype
        )
        expected = expected.reshape(shape)
        file_path = tmp_path / "roundtrip.png"
>       imsave(file_path, expected)
skimage/io/tests/test_imageio.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
skimage/io/_io.py:143: in imsave
    return call_plugin('imsave', fname, arr, plugin=plugin, **plugin_args)
skimage/io/manage_plugins.py:205: in call_plugin
    return func(*args, **kwargs)
/usr/lib/python3.11/site-packages/imageio/v3.py:139: in imwrite
    with imopen(
/usr/lib/python3.11/site-packages/imageio/core/v3_plugin_api.py:367: in __exit__
    self.close()
/usr/lib/python3.11/site-packages/imageio/plugins/pillow.py:120: in close
    self._flush_writer()
/usr/lib/python3.11/site-packages/imageio/plugins/pillow.py:428: in _flush_writer
    primary_image.save(self._request.get_file(), **self.save_args)
/usr/lib64/python3.11/site-packages/PIL/Image.py:2432: in save
    save_handler(self, fp, filename)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
im = <PIL.Image.Image image mode=I;16B size=10x10 at 0x3FF46080AD0>
fp = <_io.BufferedWriter name='/tmp/pytest-of-mockbuild/pytest-0/test_imsave_roundtrip_shape1_u0/roundtrip.png'>
filename = '/tmp/pytest-of-mockbuild/pytest-0/test_imsave_roundtrip_shape1_u0/roundtrip.png'
chunk = <function putchunk at 0x3ff54a3f560>, save_all = False
    def _save(im, fp, filename, chunk=putchunk, save_all=False):
        # save an image to disk (called by the save method)

        if save_all:
            default_image = im.encoderinfo.get(
                "default_image", im.info.get("default_image")
            )
            modes = set()
            append_images = im.encoderinfo.get("append_images", [])
            if default_image:
                chain = itertools.chain(append_images)
            else:
                chain = itertools.chain([im], append_images)
            for im_seq in chain:
                for im_frame in ImageSequence.Iterator(im_seq):
                    modes.add(im_frame.mode)
            for mode in ("RGBA", "RGB", "P"):
                if mode in modes:
                    break
            else:
                mode = modes.pop()
        else:
            mode = im.mode

        if mode == "P":
            #
            # attempt to minimize storage requirements for palette images
            if "bits" in im.encoderinfo:
                # number of bits specified by user
                colors = min(1 << im.encoderinfo["bits"], 256)
            else:
                # check palette contents
                if im.palette:
                    colors = max(min(len(im.palette.getdata()[1]) // 3, 256), 1)
                else:
                    colors = 256

            if colors <= 16:
                if colors <= 2:
                    bits = 1
                elif colors <= 4:
                    bits = 2
                else:
                    bits = 4
                mode = f"{mode};{bits}"

        # encoder options
        im.encoderconfig = (
            im.encoderinfo.get("optimize", False),
            im.encoderinfo.get("compress_level", -1),
            im.encoderinfo.get("compress_type", -1),
            im.encoderinfo.get("dictionary", b""),
        )

        # get the corresponding PNG mode
        try:
            rawmode, mode = _OUTMODES[mode]
        except KeyError as e:
            msg = f"cannot write mode {mode} as PNG"
>           raise OSError(msg) from e
E           OSError: cannot write mode I;16B as PNG
/usr/lib64/python3.11/site-packages/PIL/PngImagePlugin.py:1289: OSError
________________________________ test_all_mono _________________________________
    def test_all_mono():
        with expected_warnings(['.* is a boolean image']):
>           mono_check('pil')
skimage/io/tests/test_pil.py:259: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
skimage/_shared/testing.py:194: in mono_check
    testing.assert_allclose(r4, img_as_uint(img4))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
args = (<function assert_allclose.<locals>.compare at 0x3ff463b68e0>, array([[29812, 29812, 31354, ..., 23645, 24672, 24672],...29298, 29298, 29040, ..., 30326, 30326, 30326],
       [29298, 29298, 29040, ..., 30326, 30326, 30326]], dtype=uint16))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}
    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Not equal to tolerance rtol=1e-07, atol=0
E           
E           Mismatched elements: 131976 / 262144 (50.3%)
E           Max absolute difference: 255
E           Max relative difference: 0.33116883
E            x: array([[29812, 29812, 31354, ..., 23645, 24672, 24672],
E                  [29812, 29812, 31354, ..., 23645, 24672, 24672],
E                  [29812, 29812, 31354, ..., 23645, 24672, 24672],...
E            y: array([[29812, 29812, 31354, ..., 23900, 24672, 24672],
E                  [29812, 29812, 31354, ..., 23900, 24672, 24672],
E                  [29812, 29812, 31354, ..., 23900, 24672, 24672],...
/usr/lib64/python3.11/contextlib.py:81: AssertionError
_______________ test_analytical_moments_calculation[3-1-float32] _______________
dtype = <class 'numpy.float32'>, order = 1, ndim = 3
    @pytest.mark.parametrize('dtype', [np.uint8, np.int32, np.float32, np.float64])
    @pytest.mark.parametrize('order', [1, 2, 3, 4])
    @pytest.mark.parametrize('ndim', [2, 3, 4])
    def test_analytical_moments_calculation(dtype, order, ndim):
        if ndim == 2:
            shape = (256, 256)
        elif ndim == 3:
            shape = (64, 64, 64)
        else:
            shape = (16, ) * ndim
        rng = np.random.default_rng(1234)
        if np.dtype(dtype).kind in 'iu':
            x = rng.integers(0, 256, shape, dtype=dtype)
        else:
            x = rng.standard_normal(shape, dtype=dtype)
        # setting center=None will use the analytical expressions
        m1 = moments_central(x, center=None, order=order)
        # providing explicit centroid will bypass the analytical code path
        m2 = moments_central(x, center=centroid(x), order=order)
        # ensure numeric and analytical central moments are close
        thresh = 1e-4 if x.dtype == np.float32 else 1e-9
>       compare_moments(m1, m2, thresh=thresh)
skimage/measure/tests/test_moments.py:237: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
m1 = array([[[140.6073,   0.    ],
        [  0.    ,   0.    ]],
       [[  0.    ,   0.    ],
        [  0.    ,   0.    ]]], dtype=float32)
m2 = array([[[ 1.4060730e+02, -7.3242188e-04],
        [-3.9062500e-03,  0.0000000e+00]],
       [[-1.8066406e-02, -1.1670656e+06],
        [-8.2869025e+05,  1.7116510e+08]]], dtype=float32)
thresh = 0.0001
    def compare_moments(m1, m2, thresh=1e-8):
        """Compare two moments arrays.

        Compares only values in the upper-left triangle of m1, m2 since
        values below the diagonal exceed the specified order and are not computed
        when the analytical computation is used.

        Also, there the first-order central moments will be exactly zero with the
        analytical calculation, but will not be zero due to limited floating point
        precision when using a numerical computation. Here we just specify the
        tolerance as a fraction of the maximum absolute value in the moments array.
        """
        m1 = m1.copy()
        m2 = m2.copy()

        # make sure location of any NaN values match and then ignore the NaN values
        # in the subsequent comparisons
        nan_idx1 = np.where(np.isnan(m1.ravel()))[0]
        nan_idx2 = np.where(np.isnan(m2.ravel()))[0]
        assert len(nan_idx1) == len(nan_idx2)
        assert np.all(nan_idx1 == nan_idx2)
        m1[np.isnan(m1)] = 0
        m2[np.isnan(m2)] = 0

        max_val = np.abs(m1[m1 != 0]).max()
        for orders in itertools.product(*((range(m1.shape[0]),) * m1.ndim)):
            if sum(orders) > m1.shape[0] - 1:
                m1[orders] = 0
                m2[orders] = 0
                continue
            abs_diff = abs(m1[orders] - m2[orders])
            rel_diff = abs_diff / max_val
>           assert rel_diff < thresh
E           assert 0.00012848839 < 0.0001
skimage/measure/tests/test_moments.py:50: AssertionError
=============================== warnings summary ===============================
skimage/filters/tests/test_unsharp_mask.py: 264 warnings
  /builddir/build/BUILDROOT/python-scikit-image-0.21.0-1.fc39.s390x/usr/lib64/python3.11/site-packages/skimage/filters/tests/test_unsharp_mask.py:26: RuntimeWarning: invalid value encountered in cast
    array = ((array + offset) * 128).astype(dtype)
skimage/io/tests/test_io.py::test_imread_http_url
  /usr/lib64/python3.11/typing.py:1345: ResourceWarning: unclosed file <_io.BufferedWriter name='/tmp/pytest-of-mockbuild/pytest-0/test_imsave_roundtrip_shape1_u0/roundtrip.png'>
    self.__args__ = tuple(... if a is _TypingEllipsis else
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED skimage/io/tests/test_imageio.py::TestSave::test_imsave_roundtrip[shape1-uint16]
FAILED skimage/io/tests/test_pil.py::test_all_mono - AssertionError: 
FAILED skimage/measure/tests/test_moments.py::test_analytical_moments_calculation[3-1-float32]

Way to reproduce:

+ cd scikit-image-0.21.0
+ mkdir -p matplotlib
+ touch matplotlib/matplotlibrc
+ export XDG_CACHE_HOME=/builddir/build/BUILD/scikit-image-0.21.0
+ XDG_CACHE_HOME=/builddir/build/BUILD/scikit-image-0.21.0
+ export XDG_CONFIG_HOME=/builddir/build/BUILD/scikit-image-0.21.0
~/build/BUILDROOT/python-scikit-image-0.21.0-1.fc39.s390x/usr/lib64/python3.11/site-packages ~/build/BUILD/scikit-image-0.21.0
+ XDG_CONFIG_HOME=/builddir/build/BUILD/scikit-image-0.21.0
+ export PYTHONDONTWRITEBYTECODE=1
+ PYTHONDONTWRITEBYTECODE=1
+ export 'PYTEST_ADDOPTS=-p no:cacheprovider'
+ PYTEST_ADDOPTS='-p no:cacheprovider'
+ pushd /builddir/build/BUILDROOT/python-scikit-image-0.21.0-1.fc39.s390x//usr/lib64/python3.11/site-packages
+ xvfb-run pytest -v --deselect=skimage/data/tests/test_data.py::test_download_all_with_pooch --deselect=skimage/data/tests/test_data.py::test_eagle --deselect=skimage/data/tests/test_data.py::test_brain_3d --deselect=skimage/data/tests/test_data.py::test_cells_3d --deselect=skimage/data/tests/test_data.py::test_kidney_3d_multichannel --deselect=skimage/data/tests/test_data.py::test_lily_multichannel --deselect=skimage/data/tests/test_data.py::test_skin --deselect=skimage/data/tests/test_data.py::test_vortex --deselect=skimage/measure/tests/test_blur_effect.py::test_blur_effect_3d --deselect=skimage/registration/tests/test_masked_phase_cross_correlation.py::test_masked_registration_3d_contiguous_mask skimage
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 -- /usr/bin/python3
rootdir: /builddir/build/BUILDROOT/python-scikit-image-0.21.0-1.fc39.s390x/usr/lib64/python3.11/site-packages
plugins: localserver-0.7.0
collecting ... collected 8112 items / 9 deselected / 5 skipped / 8103 selected

Version information:

This is from my rawhide dev machines, but the versions should match fairly close

3.11.3 (main, May 24 2023, 00:00:00) [GCC 13.1.1 20230511 (Red Hat 13.1.1-2)]
Linux-6.4.0-0.rc2.23.fc39.x86_64-x86_64-with-glibc2.37.9000
numpy version: 1.24.3
hmaarrfk commented 1 year ago

It seems that we are hitting bugs in Pillow (which we use to dispatch image loading: skimage->imageio->pillow)

as we drill down into this, do you know how to skip certain tests?

Sorry to ask you to read templating code, but you can see what we do for conda-forge releases: https://github.com/conda-forge/scikit-image-feedstock/blob/main/recipe/meta.yaml#L125

We use the -k flag https://docs.pytest.org/en/6.2.x/usage.html#specifying-tests-selecting-tests

I hope this helps move your builds forward while we debug the tests

opoplawski commented 1 year ago

Looks like we have pillow 9.5.0. Yes, I can disable tests if needed to move forward, thanks.

hmaarrfk commented 1 year ago

I'm not sure how motivated you are to get to the bottom of this, but I suspect it will be difficult for us to craft a minimum reproducing example, but I expect it will be rather short. once we reproduce, we could work with pillow to fix it.

Pillow documentation mentions:

I;16B (16-bit big endian unsigned integer pixels)

https://pillow.readthedocs.io/en/stable/handbook/concepts.html

so it seems that we may have an endienness problem in that test

jarrodmillman commented 1 year ago

@opoplawski Would you check whether this test failure is appearing with imageio 2.31.0 or an earlier version of imageio?

FirefoxMetzger commented 1 year ago

Beautiful (and annoying) xD

TestSave.test_imsave_roundtrip[shape1-uint16] is most likely a big-endian vs little-endian issue. You are passing an array of uint16 (big-endian) integers to pillow, and it complains/fails because it's PNG plugin can't save that. What's ironic about this is that PNG internally stores data as big-endian so (in theory) this should be easy to do.

I don't own a system that has a big-endian architecture (nor does GH actions afaik), so it is hard for me to do any concrete investigation beyond looking at the log you've shared. Based on the test name though, my guess is that skimage (via ImageIO) is loading a PNG into machine-native 16-bit and then fails to save it later because pillow doesn't know how to save uint16 big-endian as PNG.

(this is likely a bug in pillow with a simple fix)

test_all_mono appears to have a problem with img_as_uint. Since it only fails on s390x only, I think it could also be worth checking if this is a big/little endian problem. (I don't know the test, but I assume that this is unrelated to pillow)

test_analytical_moments_calculation[3-1-float32] is a numerics problem - again probably not caused by pillow. This could trace back to changes in how moment calculations are implemented, or it could be platform-specific differences in numerical accuracy (I don't know s390x beyond that it is big-endian). If it is related to changes in implementation the test should pass in an earlier version of skimage, if it is platform-specific it will fail in any case and debugging will become a bit more complicated...

olebole commented 1 year ago

I'd just like to mention that we have the same problem in Debian. Current PIL version is 0.10.0, imageio version is 2.31.1. It seems that GH actions can (slowly) run s390x; see the weekly actions in astropy.

github-actions[bot] commented 6 months ago

Hello scikit-image core devs! There hasn't been any activity on this issue for more than 180 days. I have marked it as "dormant" to make it easy to find. To our contributors, thank you for your contribution and apologies if this issue fell through the cracks! Hopefully this ping will help bring some fresh attention to the issue. If you need help, you can always reach out on our forum If you think that this issue is no longer relevant, you may close it, or we may do it at some point (either way, it will be done manually).