Using h5py and netCDF4 in the same script

[GoogleCodeExporter]

Hei,

Although my script fails in the h5py module (and, yes, I already contacted the 
h5py team), I suspect there is a compatibility issue between python-netCDF4 and 
h5py in the way they open/close access to the HDF5 library.

I am basically reading a h5 file (with h5py), process the data, and write it 
back to netCDF3 (using python netCDF4). This works fine for the first h5 file, 
but crashes for the subsequent files (I am looping through a list of h5 files).

My configuration:
===================
Ubuntu Lucid
python 2.6
h5py 2.2.0 (latest)
python-netCDF4 1.0.7 (latest)
libnetcdf4 4.1.1
libhdf5 1.8.5-patch1
===================

I build python-netCDF4 and h5py from source, and in both cases use appropriate 
environment variables to python setup.py so that both modules will find the 
libraries (that are not in default system directories). I have long experience 
using python-netCDF4, but it is the first time I try to mix h5py and netCDF.

Here is a small script to make it happen:

<code>
import h5py
import netCDF4

g = netCDF.Dataset('test.nc','w')
g.close()
f = h5py.File('file.h5','r')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/thomasl/software/pmw_decode/my_satdata_polar/binsrc/pmw_decode/src/<ipytho
n console> in <module>()

/home/thomasl/software/pmw_decode/my_satdata_polar/binsrc/pmw_decode/mypython/li
b/python2.6/site-packages/h5py/_hl/files.pyc in __init__(self, name, mode, 
driver, libver, userblock_size, **kwds)
    204                 pass
    205
--> 206             fapl = make_fapl(driver, libver, **kwds)
    207             fid = make_fid(name, mode, userblock_size, fapl)
    208

/home/thomasl/software/pmw_decode/my_satdata_polar/binsrc/pmw_decode/mypython/li
b/python2.6/site-packages/h5py/_hl/files.pyc in make_fapl(driver, libver, 
**kwds)
     30     """ Set up a file access property list """
     31     plist = h5p.create(h5p.FILE_ACCESS)
---> 32     plist.set_fclose_degree(h5f.CLOSE_STRONG)
     33
     34     if libver is not None:

AttributeError: 'h5py.h5p.PropLAID' object has no attribute 'set_fclose_degree'
</code>

Do you python-netCDF4 folk have a hint on why this happens? For reference, here 
is a link to the issue I reported to the h5py team:
https://groups.google.com/forum/#!topic/h5py/AZQ30pSy-RI

Cheers,
Thomas

Original issue reported on code.google.com by lavergne...@gmail.com on 10 Dec 2013 at 9:56

[GoogleCodeExporter]

Unfortunately, I have no idea.  The close method simply calls nc_close in the 
netcdf-c library. I don't know how nc_close interacts with the HDF5 library, 
but it's possible not everything is cleaned up properly.  I think the thing to 
do is to try to replicate this in a simple C program, then report this to both 
the Unidata netcdf team and the HDF5 development team.

Original comment by whitaker.jeffrey@gmail.com on 10 Dec 2013 at 3:04

[GoogleCodeExporter]

It also looks like you're using rather old versions of the netcdf and HDF5 
libs. netcdf is now at 4.3.1rc4 and hdf5 at 1.8.12.  Any chance you could 
upgrade to see if this is a problem that has been fixed at the C level?

Original comment by whitaker.jeffrey@gmail.com on 10 Dec 2013 at 4:47

[GoogleCodeExporter]

This script works for me with netcdf 4.3.0 and hdf5 1.8.11

import netCDF4, h5py
for i in range(5): 
    g = netCDF4.Dataset('test%s.nc' % i,'w')
    g.close()
    f = h5py.File('test%s.nc' % i,'r')
    f.close()

It differs from yours in that it look like you are reading a separate hdf5 file 
(file.h5) presumably created by some other application.  I don't know if that's 
import though.  Can you try this version and let us know what happens?  If it 
doesn't work for you, that would confirm that it's an issue that's been fixed 
in the C libs.

Original comment by whitaker.jeffrey@gmail.com on 10 Dec 2013 at 5:59

[GoogleCodeExporter]

Thanks Jeff,

I will look at it first thing tomorrow morning. A colleague of mine reported 
that he was not experiencing such issues on our Ubuntu Precise distributions, 
that have newer versions of the netCDF and hdf5 libs.

I'll let you know asap. Hopefully this is solved by the C libraries.

Thomas

Original comment by lavergne...@gmail.com on 10 Dec 2013 at 6:58

[GoogleCodeExporter]

Jeff,

Good news. The issue seems to disappear when upgrading to our Ubuntu Precise 
distro (python-2.7, hdf5-1.8.4, netcdf-4.1.1, h5py-2.2.0, python-netCDF4-1.0.7).

So it seems this is fixed somewhere in the C library.

Nothing to worry about, then ;)

Cheers, and thanks for your help.
Thomas

Original comment by lavergne...@gmail.com on 11 Dec 2013 at 2:13

[GoogleCodeExporter]

Glad to hear it.  Closing...

Original comment by whitaker.jeffrey@gmail.com on 11 Dec 2013 at 3:28

• Changed state: Verified