search

NeurodataWithoutBorders / pynwb

A Python API for working with Neurodata stored in the NWB Format
https://pynwb.readthedocs.io
Other
175 stars 85 forks source link

test_roundtrip: "Member overlaps with another member" errors or segfaults on older Debian/Ubuntu #608

Open yarikoptic opened 5 years ago

yarikoptic commented 5 years ago

1) Bug

Steps to Reproduce

Running tests (using python -m unittest2) while building a backport package for Debian stretch (amd64), test_roundtrip in both tests.integration.ui_write.test_ophys.TestPlaneSegmentation and tests.integration.ui_write.test_ophys.TestRoiResponseSeriesIO error out with

======================================================================
ERROR: test_roundtrip (tests.integration.ui_write.test_ophys.TestPlaneSegmentation)
----------------------------------------------------------------------
Traceback (most recent call last):                           
  File "tests/integration/ui_write/base.py", line 166, in test_roundtrip
    read_container = self.roundtripContainer()               
  File "tests/integration/ui_write/base.py", line 153, in roundtripContainer
    self.writer.write(nwbfile)                               
  File "pynwb/form/utils.py", line 330, in func_call         
    return func(self, **parsed['args'])                      
  File "pynwb/form/backends/hdf5/h5tools.py", line 186, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)      
  File "pynwb/form/utils.py", line 230, in call_docval_func  
    return func(*fargs, **fkwargs)                           
  File "pynwb/form/utils.py", line 330, in func_call         
    return func(self, **parsed['args'])                      
  File "pynwb/form/backends/io.py", line 40, in write        
    self.write_builder(f_builder, **kwargs)                  
  File "pynwb/form/utils.py", line 330, in func_call         
    return func(self, **parsed['args'])                      
  File "pynwb/form/backends/hdf5/h5tools.py", line 367, in write_builder
    self.write_group(self.__file, gbldr)
  File "pynwb/form/utils.py", line 330, in func_call
    return func(self, **parsed['args'])
  File "pynwb/form/backends/hdf5/h5tools.py", line 504, in write_group
    self.write_group(group, sub_builder)
  File "pynwb/form/utils.py", line 330, in func_call
    return func(self, **parsed['args'])
  File "pynwb/form/backends/hdf5/h5tools.py", line 504, in write_group
    self.write_group(group, sub_builder)
  File "pynwb/form/utils.py", line 330, in func_call
    return func(self, **parsed['args'])
  File "pynwb/form/backends/hdf5/h5tools.py", line 504, in write_group
    self.write_group(group, sub_builder)
  File "pynwb/form/utils.py", line 330, in func_call
    return func(self, **parsed['args'])
  File "pynwb/form/backends/hdf5/h5tools.py", line 509, in write_group
    self.write_dataset(group, sub_builder)
  File "pynwb/form/utils.py", line 330, in func_call
    return func(self, **parsed['args'])
  File "pynwb/form/backends/hdf5/h5tools.py", line 615, in write_dataset
    dset = parent.require_dataset(name, shape=(len(data),), dtype=_dtype, **options['io_settings'])
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/group.py", line 129, in require_dataset
    return self.create_dataset(name, *(shape, dtype), **kwds)
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/group.py", line 108, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/dataset.py", line 97, in make_new_dset
    tid = h5t.py_create(dtype, logical=1)
  File "h5py/h5t.pyx", line 1543, in h5py.h5t.py_create (/build/h5py-b1xqNM/h5py-2.7.0/h5py/h5t.c:18114)
  File "h5py/h5t.pyx", line 1565, in h5py.h5t.py_create (/build/h5py-b1xqNM/h5py-2.7.0/h5py/h5t.c:17934)
  File "h5py/h5t.pyx", line 1587, in h5py.h5t.py_create (/build/h5py-b1xqNM/h5py-2.7.0/h5py/h5t.c:17412)
  File "h5py/h5t.pyx", line 1511, in h5py.h5t._c_compound (/build/h5py-b1xqNM/h5py-2.7.0/h5py/h5t.c:16596)
ValueError: Unable to insert member (Member overlaps with another member)

h5py there is 2.7.0-1. This issue doesn't happen on newer Debian/Ubuntu's, and Ubuntu 16.04 (h5py 2.6.0-1) python segfaults at that point:

test_roundtrip (tests.integration.ui_write.test_ophys.TestPlaneSegmentation) ... Segmentation fault (core dumped)

full build logs if you need more context information are available from http://neuro.debian.net/_files/_buildlogs/pynwb/0.4.1+git41-gc3a1bed

Any clues?
It is probably the h5py's problem, but wondered if you encountered and what should I do - not provide package for those platforms, skip this problematic test, or there is a way around? meanwhile I will see if backporting h5py 2.7.1 for them would be easy (seems to not show that problem on ubuntu 17.10)

Environment

Please describe your environment according to the following bullet points.

bendichter commented 5 years ago

Thanks for this report @yarikoptic. It's important for us to understand the limits of our OS support. I would think the issue is probably an HDF5 version problem. What are the HDF5 versions on the two systems? Is HDF5 version linked to h5py version in some way?

yarikoptic commented 5 years ago

on that Debian one:

# ldd /usr/lib/python2.7/dist-packages/h5py/h5t.x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fffd37fa000)
        /usr/lib/cowdancer/libcowdancer.so (0x00007f7b8e10e000)
        libhdf5_serial.so.100 => /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.100 (0x00007f7b8dba6000)
        libhdf5_serial_hl.so.100 => /usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.100 (0x00007f7b8d982000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7b8d765000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7b8d3c6000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7b8d1c2000)
        libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f7b8cf9f000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f7b8cd75000)
        libsz.so.2 => /usr/lib/x86_64-linux-gnu/libsz.so.2 (0x00007f7b8cb72000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7b8c958000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7b8c654000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7b8e580000)
        libaec.so.0 => /usr/lib/x86_64-linux-gnu/libaec.so.0 (0x00007f7b8c44c000)

# dpkg -S /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.100 
libhdf5-100:amd64: /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.100

# apt-cache policy libhdf5-100 
libhdf5-100:
  Installed: 1.10.0-patch1+docs-3+deb9u1
  Candidate: 1.10.0-patch1+docs-3+deb9u1
  Version table:
 *** 1.10.0-patch1+docs-3+deb9u1 500
        500 http://127.0.0.1:3142/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
yarikoptic commented 5 years ago

I would have said you are right that the cause is within the libhdf5 library itself, since upgrade to h5py 2.7.1 build didn't resolve the test failure :-/ but it is the same version of libhdf5-100 as on later Debians: https://packages.debian.org/search?keywords=libhdf5-100

yarikoptic commented 5 years ago

he hey -- I was wrong! I have missed that the failure after "upgrade" of h5py was in python3 run, and I've upgraded only python2 version.... confirming now that upgrade to h5py 2.7.1 resolves the failure there. Unfortunately auto-backporting of 2.7.1 to ubuntu 16.04 fails (errors seems to point to absent linkage to libpython, but probably I will not mess with it ATM), so for now let's conclude that h5py 2.7.1 should be the minimal version to depend on ;)

oruebel commented 5 years ago

Since you are using h5py 2.7 I am wondering whether this is related to the following issue in h5py which was reported after the 2.7 release:

https://github.com/h5py/h5py/issues/862

t-b commented 5 years ago

@oruebel Since 3cc55d40 (Pin specific package versions in requirements, 2017-10-23) we are requiring h5py 2.7.1 in requirements.txt.

oruebel commented 5 years ago

@t-b it might be worth trying to update to h5py 2.8 so see if that fixes the issue