search

fractal-analytics-platform / fractal-tasks-core

Main tasks for the Fractal analytics platform
https://fractal-analytics-platform.github.io/fractal-tasks-core/
BSD 3-Clause "New" or "Revised" License
11 stars 5 forks source link

`zenodo_images_multiplex_shifted` fixture fails with Pillow error #674

Closed tcompa closed 2 months ago

tcompa commented 3 months ago

See https://productionresultssa6.blob.core.windows.net/actions-results/7fba96e7-3c3c-4b40-a726-d48a32af5f1d/workflow-job-run-4c309e78-7174-5e1c-e1d6-a7c065a0787a/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-04-02T10%3A22%3A56Z&sig=hJTfwP2lSuZQCK8ICQaUIAgJeNxbrvZt7XdPNDLsR70%3D&sp=r&spr=https&sr=b&st=2024-04-02T10%3A12%3A51Z&sv=2021-12-02.

This was part of the PR #673, but I think the same would happen in main - since we are not pinning Pillow to any specific version.

The first guess here is that this is due to pillow 10.3.0, released yesterday (April 1st).

2024-04-02T09:18:52.0051216Z _______________ ERROR at setup of test_multiplexing_registration _______________
2024-04-02T09:18:52.0052042Z 
2024-04-02T09:18:52.0054472Z zenodo_images_multiplex = ['/home/runner/work/fractal-tasks-core/fractal-tasks-core/tests/data/fake_multiplex/cycle1', '/home/runner/work/fractal-tasks-core/fractal-tasks-core/tests/data/fake_multiplex/cycle2']
2024-04-02T09:18:52.0056293Z testdata_path = PosixPath('/home/runner/work/fractal-tasks-core/fractal-tasks-core/tests/data')
2024-04-02T09:18:52.0056868Z 
2024-04-02T09:18:52.0057008Z     @pytest.fixture(scope="function")
2024-04-02T09:18:52.0057413Z     def zenodo_images_multiplex_shifted(
2024-04-02T09:18:52.0057831Z         zenodo_images_multiplex: list[str],
2024-04-02T09:18:52.0058225Z         testdata_path: Path,
2024-04-02T09:18:52.0058566Z     ) -> list[str]:
2024-04-02T09:18:52.0058818Z         """
2024-04-02T09:18:52.0059080Z         Return a list of strings, like
2024-04-02T09:18:52.0059437Z         ```
2024-04-02T09:18:52.0059662Z         [
2024-04-02T09:18:52.0059957Z             "/some/path/fake_multiplex_shifted/cycle1",
2024-04-02T09:18:52.0060415Z             "/some/path/fake_multiplex_shifted/cycle2"
2024-04-02T09:18:52.0060810Z         ]
2024-04-02T09:18:52.0061032Z         ```
2024-04-02T09:18:52.0061253Z         """
2024-04-02T09:18:52.0061510Z         # Define old and new folders
2024-04-02T09:18:52.0061943Z         old_folder = str(testdata_path / "fake_multiplex")
2024-04-02T09:18:52.0062573Z         new_folder = old_folder.replace("fake_multiplex", "fake_multiplex_shifted")
2024-04-02T09:18:52.0063110Z     
2024-04-02T09:18:52.0063432Z         # Define output folders (one per multiplexing cycle)
2024-04-02T09:18:52.0064013Z         cycle_folders = [f"{new_folder}/cycle{ind}" for ind in (1, 2)]
2024-04-02T09:18:52.0064475Z     
2024-04-02T09:18:52.0064725Z         if os.path.isdir(new_folder):
2024-04-02T09:18:52.0065498Z             # If the shifted-images folder already exists, return immediately
2024-04-02T09:18:52.0066008Z             print(f"{new_folder} already exists")
2024-04-02T09:18:52.0066376Z             return cycle_folders
2024-04-02T09:18:52.0066660Z         else:
2024-04-02T09:18:52.0067228Z             # Copy the fake_multiplex folder into a new one
2024-04-02T09:18:52.0067991Z             shutil.copytree(old_folder, new_folder)
2024-04-02T09:18:52.0068712Z             # Loop over images of cycle2 and apply a shift
2024-04-02T09:18:52.0069541Z             for img_path in glob.glob(f"{cycle_folders[1]}/2020*.png"):
2024-04-02T09:18:52.0070317Z                 print(f"Now shifting {img_path}")
2024-04-02T09:18:52.0070950Z >               _shift_image(str(img_path))
2024-04-02T09:18:52.0071378Z 
2024-04-02T09:18:52.0071585Z tests/tasks/test_registration.py:123: 
2024-04-02T09:18:52.0072321Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-04-02T09:18:52.0073168Z tests/tasks/test_registration.py:88: in _shift_image
2024-04-02T09:18:52.0073920Z     new_img = Image.fromarray(new_array, mode="I")
2024-04-02T09:18:52.0074903Z .venv/lib/python3.10/site-packages/PIL/Image.py:3154: in fromarray
2024-04-02T09:18:52.0075699Z     return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
2024-04-02T09:18:52.0076768Z .venv/lib/python3.10/site-packages/PIL/Image.py:3069: in frombuffer
2024-04-02T09:18:52.0077688Z     return frombytes(mode, size, data, decoder_name, args)
2024-04-02T09:18:52.0078691Z .venv/lib/python3.10/site-packages/PIL/Image.py:3012: in frombytes
2024-04-02T09:18:52.0079498Z     im.frombytes(data, decoder_name, args)
2024-04-02T09:18:52.0080246Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-04-02T09:18:52.0081072Z 
2024-04-02T09:18:52.0081462Z self = <PIL.Image.Image image mode=I size=2560x2160 at 0x7F7221DFDE70>
2024-04-02T09:18:52.0082302Z data = array([[103, 106, 112, ..., 110, 110, 110],
2024-04-02T09:18:52.0082934Z        [105, 100, 104, ..., 110, 110, 110],
2024-04-02T09:18:52.0083535Z        [108, 103,  98, ..., 11... 110, 110, 110],
2024-04-02T09:18:52.0084121Z        [110, 110, 110, ..., 110, 110, 110],
2024-04-02T09:18:52.0084757Z        [110, 110, 110, ..., 110, 110, 110]], dtype=uint16)
2024-04-02T09:18:52.0085498Z decoder_name = 'raw', args = ('I', 0, 1)
2024-04-02T09:18:52.0086245Z d = <ImagingDecoder object at 0x7f7221e7d7d0>, s = (11059200, 0)
2024-04-02T09:18:52.0087044Z msg = 'not enough image data'
2024-04-02T09:18:52.0087382Z 
2024-04-02T09:18:52.0087937Z     def frombytes(self, data: bytes, decoder_name: str = "raw", *args) -> None:
2024-04-02T09:18:52.0088749Z         """
2024-04-02T09:18:52.0089258Z         Loads this image with pixel data from a bytes object.
2024-04-02T09:18:52.0089921Z     
2024-04-02T09:18:52.0090549Z         This method is similar to the :py:func:`~PIL.Image.frombytes` function,
2024-04-02T09:18:52.0091631Z         but loads data into this image instead of creating a new image object.
2024-04-02T09:18:52.0092428Z         """
2024-04-02T09:18:52.0092770Z     
2024-04-02T09:18:52.0093193Z         if self.width == 0 or self.height == 0:
2024-04-02T09:18:52.0093778Z             return
2024-04-02T09:18:52.0094137Z     
2024-04-02T09:18:52.0094551Z         # may pass tuple instead of argument list
2024-04-02T09:18:52.0095324Z         if len(args) == 1 and isinstance(args[0], tuple):
2024-04-02T09:18:52.0095972Z             args = args[0]
2024-04-02T09:18:52.0096394Z     
2024-04-02T09:18:52.0096726Z         # default format
2024-04-02T09:18:52.0097238Z         if decoder_name == "raw" and args == ():
2024-04-02T09:18:52.0097839Z             args = self.mode
2024-04-02T09:18:52.0098277Z     
2024-04-02T09:18:52.0098612Z         # unpack data
2024-04-02T09:18:52.0099129Z         d = _getdecoder(self.mode, decoder_name, args)
2024-04-02T09:18:52.0099764Z         d.setimage(self.im)
2024-04-02T09:18:52.0100229Z         s = d.decode(data)
2024-04-02T09:18:52.0100654Z     
2024-04-02T09:18:52.0100977Z         if s[0] >= 0:
2024-04-02T09:18:52.0101618Z             msg = "not enough image data"
2024-04-02T09:18:52.0102203Z >           raise ValueError(msg)
2024-04-02T09:18:52.0102774Z E           ValueError: not enough image data
2024-04-02T09:18:52.0103214Z 
2024-04-02T09:18:52.0103695Z .venv/lib/python3.10/site-packages/PIL/Image.py:830: ValueError
jluethi commented 2 months ago

I'm switching to using imageio for the file handling, I think that's working fine. Will do further tests to confirm it