search

hipspy / hips

Python library to handle HiPS
https://hips.readthedocs.io
13 stars 16 forks source link

ValueError from flipup call in Tile.to_numpy #112

Open cdeil opened 6 years ago

cdeil commented 6 years ago

Sometimes an error occurs in the np.flipup call here when converting JPEG to numpy array: https://github.com/hipspy/hips/blob/ed7d02d979a5e4966efad509184e942ae487b160/hips/tiles/tile.py#L302

    with Image.open(bio) as image:
                data = np.array(image)
                # Flip tile to be consistent with FITS orientation
                data = np.flipud(data)

ValueError: Input must be >= 1-d.

I saw this here: https://travis-ci.org/hipspy/hips/jobs/288318105#L1611 and also locally (from a different test) here: https://gist.github.com/cdeil/ad3dd0dc6a6c1fe8d58ec22ecdad03d0

It doesn't seem to be deterministic, I re-ran the same test and the fail doesn't show up.

I'm not sure what causes it, it looks like Pillow doesn't read the file properly, but only sometimes!??? For now, I'll just keep this issue open as a reminder, and try one thing: move the np.flipup call out of the with block, to make sure that the file was closed and Pillow was done for sure:

            with Image.open(bio) as image:
                data = np.array(image)
                # Flip tile to be consistent with FITS orientation
            data = np.flipud(data)
cdeil commented 6 years ago

I don't have time to look into this issue now, but wanted to leave a note:

When @adonath and @tboch implemented the HiPS gen, they noticed an issue that FITS I/O doesn't work properly for a Numpy array that's transposed, as the result of calling np.rot90 is: https://github.com/hipspy/hips/pull/123/files#diff-4de2f89531f836afdfd881e73aea8632R43 Probably here where we call np.flipup also a transposed Numpy array is created and from the comments above it seems that also Pillow I/O sometimes doesn't work properly with that.

There is https://github.com/astropy/astropy/issues/2150 describing the issue for Gzipped FITS, but I'm not aware of an open issue in the Astropy tracker without Gzip compression, which I think is what we're using for HiPS.

So basically this issue needs someone to look in detail at how FITS / PNG / JPEG I/O react to passing in transposed Numpy arrays (add test cases for that), and where necessary add a np.copy or np.ascontiguousarray call to make sure a "normal" Numpy array is used in those places. That should be done in the I/O code, not in random code places like here: https://github.com/hipspy/hips/pull/123/files#diff-4de2f89531f836afdfd881e73aea8632R43

@adl1995 or @adonath - If you don't have time for this, I could give it a try in the coming weeks.