parallel write bug

[bendichter]

1) Bug

It looks like we are having issues with parallel IO in the latest dev branch

Steps to Reproduce

When I run the MPI demo:

OSError: Can't prepare for writing data (Parallel IO does not support writing VL datatypes yet)
Traceback (most recent call last):
  File "test_parallel_pynwb5.py", line 24, in <module>
    io.write(nwbfile)
  File "/home/users/bdichter/code/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/users/bdichter/code/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 219, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)
  File "/home/users/bdichter/code/hdmf/src/hdmf/utils.py", line 281, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/home/users/bdichter/code/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/users/bdichter/code/hdmf/src/hdmf/backends/io.py", line 42, in write
    self.write_builder(f_builder, **kwargs)
  File "/home/users/bdichter/code/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/users/bdichter/code/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 448, in write_builder
    self.write_dataset(self.__file, dbldr, link_data)
  File "/home/users/bdichter/code/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/users/bdichter/code/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 786, in write_dataset
    dset = self.__list_fill__(parent, name, data, options)
  File "/home/users/bdichter/code/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 918, in __list_fill__
    raise e
  File "/home/users/bdichter/code/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 916, in __list_fill__
    dset[:] = data
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/users/bdichter/anaconda3/lib/python3.6/site-packages/h5py-2.9.0.post0-py3.6-linux-x86_64.egg/h5py/_hl/dataset.py", line 708, in __setitem__
    self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 221, in h5py.h5d.DatasetID.write
  File "h5py/_proxy.pyx", line 164, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite

It looks like a problem with vectors of strings. Did we add new vectors of strings recently? Can someone else with mpi4py up and running try to replicate this?

Checklist

• [x] Have you ensured the feature or change was not already reported ?
• [x] Have you included a brief and descriptive title?
• [x] Have you included a clear description of the problem you are trying to solve?
• [x] Have you included a minimal code snippet that reproduces the issue you are encountering?
• [x] Have you checked our Contributing document?

[stamatiad]

I can't reproduce that. I have the latest build: pynwb==1.0.2.post0.dev75. I built from source the mpich 3.2, the hdf5 1.10.5 and h5py with python setup.py configure --mpi, on CentOS (Docker VM). Maybe I'm using some newer lib than you?

[bendichter]

@stamatiad ok thanks I'll keep looking at this