search

conda-forge / libtiff-feedstock

A conda-smithy repository for libtiff.
BSD 3-Clause "New" or "Revised" License
1 stars 25 forks source link

Odd behavior between libtiff and numpy #54

Closed djhoese closed 4 years ago

djhoese commented 4 years ago

I'm looking for help debugging some failing CI jobs for my package. Some of the tests use the pylibtiff python package (wrapper around C libtiff) to write numpy arrays to a TIFF image.

Recently our CIs switched from using numpy 1.16.5 on appveyor to numpy 1.17. When this switch happened we started seeing failing tests where a numpy array of all 0s would result in the TIFF having non-zero pixels written to it.

I had seen this a couple days earlier on Travis on Linux because our CIs were accidentally installing numpy 1.18 from PyPI instead of numpy 1.17 from conda-forge, but I assumed this was just a lower-level compatibility issue taking a numpy wheel from PyPI and combining with conda-forge binaries so I ignored it. The above appveyor case is getting both numpy and libtiff from conda-forge.

Does anyone have an idea what might be going on between numpy 1.16 and numpy 1.17 to interfere with libtiff?

Working appveyor job: https://ci.appveyor.com/project/pytroll/satpy/builds/29983559 Failing appveyor job: https://ci.appveyor.com/project/pytroll/satpy/builds/30000631

My head hurts...

djhoese commented 4 years ago

So it looks like this may be a change in how numpy handles my_arr.ctypes.data and the life time of that object. Changing our code from:

tif.WriteEncodedStrip(0, np.ascontiguousarray(data.astype(data_type), data_type).ctypes.data, data.shape[0] * data.shape[1])

to

data = np.ascontiguousarray(data, data_type)
tif.WriteEncodedStrip(0, data.ctypes.data, data.shape[0] * data.shape[1])

Seems to have fixed things. I'm guessing the low-level data buffer doesn't act as a proper python ref-counted object any more...or something. Either way, it doesn't seem to be a libtiff issue.