search

ARM-DOE / pyart

The Python-ARM Radar Toolkit. A data model driven interactive toolkit for working with weather radar data.
https://arm-doe.github.io/pyart/
Other
516 stars 268 forks source link

BUG: netCDF4 and writing large files #177

Closed kirknorth closed 9 years ago

kirknorth commented 10 years ago

Is anyone aware of an inability of netCDF4 to properly write large files (e.g. writing a large radar object) when format = 'NETCDF4'? Basically I was able to write an X-band file when format = 'NETCDF3_CLASSIC', and read this file after. However, when format = 'NETCDF4', calling pyart.io.write_cfradial finished, but I was getting a segmentation fault when trying to read it after. This file was 200-300 MB in size.

jjhelmus commented 10 years ago

I have not seen anything like this. I do know there were some issue with older versions of the netcdf library that might cause something like this but as of 4.2 they seem to have been cleared up. Since it is only happening with NETCDF4 format files it might be an issue with the hdf5 library. You should be able to check the version of all of these with the following:

python -c "import netCDF4; print netCDF4.__version__"
nc-config --version
h5stat --version

I get the following:

1.0.8
netCDF 4.2.1.1
h5stat: Version 1.8.9

The only other thing I can think of is to make sure that the Python netCDF4 library is linking to the same HDF5 library that the libnetcdf library is:

For me this involved the following:

$ python -c "import netCDF4; print netCDF4.__file__"`
/home/jhelmus/anaconda/lib/python2.7/site-packages/netCDF4.so
$ ldd /home/jhelmus/anaconda/lib/python2.7/site-packages/netCDF4.so
    linux-vdso.so.1 =>  (0x00007fff4adff000)
    libnetcdf.so.7 => /home/jhelmus/anaconda/lib/python2.7/site-packages/../../libnetcdf.so.7 (0x00007f75dffba000)
    libhdf5_hl.so.7 => /home/jhelmus/anaconda/lib/python2.7/site-packages/../../libhdf5_hl.so.7 (0x00007f75dfd8c000)
    libhdf5.so.7 => /home/jhelmus/anaconda/lib/python2.7/site-packages/../../libhdf5.so.7 (0x00007f75df8d3000)
...
$ ldd /home/jhelmus/anaconda/lib/python2.7/site-packages/../../libnetcdf.so.7
linux-vdso.so.1 =>  (0x00007fffd33e2000)
    libhdf5_hl.so.7 => /home/jhelmus/anaconda/lib/python2.7/site-packages/../.././libhdf5_hl.so.7 (0x00007f7b2a819000)
    libhdf5.so.7 => /home/jhelmus/anaconda/lib/python2.7/site-packages/../.././libhdf5.so.7 (0x00007f7b2a35f000)
...

Notice how netCDF4.so is linking to the same libhdf5 libraries as libnetcdf4.so. If this is not true bad things can happen (I think). I've seen this happen where one of the two libraries is linking to the system provided libhdf5 library and the other to the one provided by Anaconda or Canopy.

kirknorth commented 10 years ago

@jjhelmus, here's my output of your commands,

$ python -c 'import netCDF4; print netCDF4.__version__'
1.0.4
$ nc-config --version
netCDF 4.1.1
$ h5stat --version
h5stat: Version 1.8.5-patch1
$ python -c 'import netCDF4; print netCDF4.__file__'
/aos/home/kirk/virtual/nimbus/lib/python2.6/site-packages/netCDF4.so
$ ldd /aos/home/kirk/virtual/nimbus/lib/python2.6/site-packages/netCDF4.so
    linux-vdso.so.1 =>  (0x00007fffb27ff000)
    libnetcdf.so.6 => /usr/lib64/libnetcdf.so.6 (0x00007f679278a000)
    libhdf5_hl.so.6 => /usr/lib64/libhdf5_hl.so.6 (0x00007f6792557000)
    libhdf5.so.6 => /usr/lib64/libhdf5.so.6 (0x00007f6791f6f000)
$ ldd /usr/lib64/libnetcdf.so.6
    linux-vdso.so.1 =>  (0x00007fff737ff000)
    libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003bdb400000)
    libhdf5_hl.so.6 => /usr/lib64/libhdf5_hl.so.6 (0x0000003ff6000000)
    libhdf5.so.6 => /usr/lib64/libhdf5.so.6 (0x0000003ff4800000)

As you can see I'm using a virtualenv.

jjhelmus commented 10 years ago

This might be an issue of NCF-143 which was fixed in netcdf 4.2. I recall running into something along this line before I updated my netcdf library.

jjhelmus commented 10 years ago

@kirknorth Are you still running into this issue or can I close this ticket?

kirknorth commented 10 years ago

I believe the issue still persists because (at the time) I just did the quick fix and set format = 'NETCDF3_CLASSIC'. Please don't close this issue just yet!

jjhelmus commented 10 years ago

Not a problem, I'll keep this open.

jjhelmus commented 9 years ago

@kirknorth Are you still investigating this?

kirknorth commented 9 years ago

@jjhelmus I'm not actively investigating this issue at this time. Give me a couple of days to run some tests. When I created this issue I wsa running Python 2.6 with netCDF4 version 1.0.4, however I now have another VE running Python 2.7 with netCDF4 version 1.1.1 and reason to suspect this won't be an issue in this VE.

jjhelmus commented 9 years ago

@kirknorth Any updates on this issue or can it be closed?

kirknorth commented 9 years ago

@jjhelmus I am no longer running into this issue, likely due to the changes in my VE as stated in my last comment.

jjhelmus commented 9 years ago

Thanks for the update, closing this issue.