netCDF4 "missing libmfhdf.so.0"

[dwhswenson]

There seems to be an incompatibility between current versions of netCDF4 and older versions of hdf4 (specifically 4.2.11-0 does not seem to work).

Steps to reproduce:

conda create --yes --name test_netCDF4 netcdf4
# note that there were two possible resolutions
source activate test_netCDF4
python -c "import netCDF4" # works fine
# switch to the other resolution
conda update --yes jpeg
python -c "import netCDF4" # fails with:
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/home/dwhs/miniconda2/envs/test_netCDF4/lib/python2.7/sitepackages/netCDF4/__init__.py", line 3, in <module>
#     from ._netCDF4 import *
# ImportError: libmfhdf.so.0: cannot open shared object file: No such file or directory

We found this in a test of our project on Travis. Some other part of our build process apparently wanted the more recent jpeg. See https://github.com/openpathsampling/openpathsampling/pull/655#issuecomment-272853914 for more version diffs. We worked around it by pinning numpy to 1.10; that isn't the source of the problem, but it was a quick solution.

[hainm]

I got similar issue..

[jjhelmus]

I was able to work around this by forcing an update of hdf4 to version 4.2.12 in the environment with netcdf4 using, conda install hdf4=4.2.12. This also downgraded the jpeg package from 9d-0 to 8d-2.

[rabernat]

Really important to get these netcdf issues fixed. These errors are tripping up new python geoscience users, causing them deep frustration with python / anaconda as a whole.

[tschoy]

Same issue.

[rabernat]

Is there a way that those of us from the community can help with the fixing of this bug?

Next September I am teaching a python course which will use netcdf extensively. I am already dreading the workarounds I will have to use if conda install netcdf4 does not just work out of the box. While I am not very familiar with conda packaging, I am really motivated to help solve the issue.

Please let me know how I can contribute.

[ArchangeGabriel]

They just need to recompile hdf4 4.2.12 against the latest version of jpeg I think. Because the problem is that hdf4 4.2.11 is OK with latest jpeg (not even sure why…), but 4.2.12 is not, while netcdf4 requires 4.2.12 and does not work with 4.2.11. So you have to pin jpeg as a workaround, which also blocks qt5 btw.

If only they would tell us about actual issue fixing those dependencies… I don’t know how their packaging system works and have other things to do than learn about it just to fix this, but if they have an issue fixing the dep and could tell steps to reproduce, I could take a look and eventually fix it.

[ArchangeGabriel]

That being said, I’ve just noticed https://github.com/conda-forge/conda-forge.github.io/pull/392#issuecomment-304332581. So at least something is happening. ;)

[chryss]

Same here. I couldn't find a good workaround in my Python 3.5 environment. Doing what @jjhelmus suggest forced libgdal to be downgraded to 2.0.0, which has a bug that's critical for my application.

[tschoy]

I just ended up pip install, and it worked.

On Tuesday, June 6, 2017 5:45 AM, Chris Waigl <notifications@github.com> wrote:

Same here. I couldn't find a good workaround in my Python 3.5 environment. Doing what @jjhelmus suggest forced libgdal to be downgraded to 2.0.0, which has a bug that's critical for my application.— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jjhelmus]

A few days ago new builds of hdf4 were added to repo.continuum.io which should address this issue. I've checked on linux-64 and can confirm that these new packages (hdf4-4.2.12-1.tar.bz2) fix the missing libmfhdf.so.0 issue.

Unfortunately on macOS it appears as if netCDF4 is still having issues due to a underpinned requirement on jpeg 8. I'll work on getting a new build of netcdf for macOS to fix this.

~$ conda create -y -n nc python=3.6 netcdf4
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /Users/jhelmus/anaconda/envs/nc:

The following NEW packages will be INSTALLED:

    curl:      7.49.0-1          defaults
    hdf4:      4.2.12-1          defaults
    hdf5:      1.8.17-1          defaults
    jpeg:      9b-0              defaults
    libnetcdf: 4.4.1-0           defaults
    mkl:       2017.0.1-0        defaults
    netcdf4:   1.2.4-np111py36_1 defaults
    numpy:     1.11.3-py36_0     defaults
    openssl:   1.0.2l-0          defaults
    python:    3.6.1-2           defaults
    readline:  6.2-2             defaults
    sqlite:    3.13.0-0          defaults
    tk:        8.5.18-0          defaults
    xz:        5.2.2-1           defaults
    zlib:      1.2.8-3           defaults

#
# To activate this environment, use:
# > source activate nc
#
# To deactivate this environment, use:
# > source deactivate nc
#

~$ source activate nc
~$ python -c "import netCDF4"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jhelmus/anaconda/envs/nc/lib/python3.6/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ImportError: dlopen(/Users/jhelmus/anaconda/envs/nc/lib/python3.6/site-packages/netCDF4/_netCDF4.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libjpeg.8.dylib
  Referenced from: /Users/jhelmus/anaconda/envs/nc/lib/libnetcdf.11.dylib
  Reason: image not found

[ArchangeGabriel]

Hum, linux-64 here, jpeg error too:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-f731da2de255> in <module>()
----> 1 import netCDF4

/local/soft/miniconda2/lib/python2.7/site-packages/netCDF4/__init__.py in <module>()
      1 # init for netCDF4. package
      2 # Docstring comes from extension module _netCDF4.
----> 3 from ._netCDF4 import *
      4 # Need explicit imports for names beginning with underscores
      5 from ._netCDF4 import __doc__, __pdoc__

ImportError: libjpeg.so.8: cannot open shared object file: No such file or directory

[jjhelmus]

@ArchangeGabriel can you provide a conda list of the environment where you are seeing that error?

[ArchangeGabriel]

Sure:

asn1crypto                0.22.0                   py27_0  
backports                 1.0                      py27_0  
cairo                     1.14.8                        0  
cffi                      1.10.0                   py27_0  
colorcet                  0.9.1                    py27_0  
conda                     4.3.21                   py27_0  
conda-env                 2.6.0                         0  
cryptography              1.8.1                    py27_0  
curl                      7.49.0                        1  
cycler                    0.10.0                   py27_0  
dbus                      1.10.10                       0  
decorator                 4.0.11                   py27_0  
enum34                    1.1.6                    py27_0  
expat                     2.1.0                         0  
fontconfig                2.12.1                        3  
freetype                  2.5.5                         2  
functools32               3.2.3.2                  py27_0  
get_terminal_size         1.0.0                    py27_0  
glib                      2.50.2                        1  
gst-plugins-base          1.8.0                         0  
gstreamer                 1.8.0                         0  
h5py                      2.7.0               np111py27_0  
hdf4                      4.2.12                        1  
hdf5                      1.8.17                        1  
icu                       54.1                          0  
idna                      2.5                      py27_0  
ipaddress                 1.0.18                   py27_0  
ipython                   5.3.0                    py27_0  
ipython_genutils          0.2.0                    py27_0  
jpeg                      9b                            0  
libffi                    3.2.1                         1  
libgcc                    5.2.0                         0  
libgfortran               3.0.0                         1  
libiconv                  1.14                          0  
libnetcdf                 4.4.1                         0  
libpng                    1.6.27                        0  
libxcb                    1.12                          1  
libxml2                   2.9.4                         0  
matplotlib                2.0.2               np111py27_0  
mkl                       2017.0.1                      0  
netcdf4                   1.2.4               np111py27_0  
numexpr                   2.6.2               np111py27_0  
numpy                     1.11.3                   py27_0  
openssl                   1.0.2l                        0  
packaging                 16.8                     py27_0  
pandas                    0.20.1              np111py27_0  
path.py                   10.3.1                   py27_0  
pathlib2                  2.2.1                    py27_0  
pcre                      8.39                          1  
pexpect                   4.2.1                    py27_0  
pickleshare               0.7.4                    py27_0  
pip                       9.0.1                    py27_1  
pixman                    0.34.0                        0  
prompt_toolkit            1.0.14                   py27_0  
ptyprocess                0.5.1                    py27_0  
pyasn1                    0.2.3                    py27_0  
pycairo                   1.10.0                   py27_0  
pycosat                   0.6.2                    py27_0  
pycparser                 2.17                     py27_0  
pycrypto                  2.6.1                    py27_6  
pygments                  2.2.0                    py27_0  
pyopenssl                 17.0.0                   py27_0  
pyparsing                 2.1.4                    py27_0  
pyqt                      5.6.0                    py27_2  
pytables                  3.3.0               np111py27_0  
python                    2.7.13                        0  
python-dateutil           2.6.0                    py27_0  
pytz                      2017.2                   py27_0  
qt                        5.6.2                         4  
readline                  6.2                           2  
requests                  2.14.2                   py27_0  
ruamel_yaml               0.11.14                  py27_1  
scandir                   1.5                      py27_0  
scipy                     0.19.0              np111py27_0  
setuptools                27.2.0                   py27_0  
simplegeneric             0.8.1                    py27_1  
sip                       4.18                     py27_0  
six                       1.10.0                   py27_0  
sqlite                    3.13.0                        0  
subprocess32              3.2.7                    py27_0  
tk                        8.5.18                        0  
traitlets                 4.3.2                    py27_0  
wcwidth                   0.1.7                    py27_0  
wheel                     0.29.0                   py27_0  
yaml                      0.1.6                         0  
zlib                      1.2.8                         3  

[jjhelmus]

@ArchangeGabriel Thanks for the details.

I dug further and it looks like the linux-64 libnetcdf 4.4.1-0 package is also broken, it somehow depends on both libjpeg.so.8 and libjpeg.so.9! No my system this oddity happens to work as the system provides libjpeg.so.8 and conda provides libjpeg.so.9. I'll look into getting new builds of libnetcdf for all platforms.

$ ldd  ~/anaconda3/envs/netcdf_py36/lib/libnetcdf.so.11
    linux-vdso.so.1 =>  (0x00007ffc4ade0000)
    libmfhdf.so.0 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libmfhdf.so.0 (0x00007f5088381000)
    libdf.so.0 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libdf.so.0 (0x00007f50880d3000)
    libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f5087e5a000)
    libhdf5_hl.so.10 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libhdf5_hl.so.10 (0x00007f5087c3c000)
    libhdf5.so.10 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libhdf5.so.10 (0x00007f5087777000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5087572000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5087269000)
    libcurl.so.4 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libcurl.so.4 (0x00007f5087006000)
    libssl.so.1.0.0 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libssl.so.1.0.0 (0x00007f5086d8e000)
    libcrypto.so.1.0.0 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libcrypto.so.1.0.0 (0x00007f5086957000)
    libz.so.1 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/./libz.so.1 (0x00007f5086741000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5086538000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f508616f000)
    libjpeg.so.9 => /home/jhelmus/anaconda3/envs/netcdf_py36/lib/././libjpeg.so.9 (0x00007f5085f33000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5085d15000)
    /lib64/ld-linux-x86-64.so.2 (0x0000564b5bbac000)

[ArchangeGabriel]

Hum indeed (on my system, I have /usr/lib64/libjpeg.so.62, so…).

[ArchangeGabriel]

How can I revert to previous situation in the meantime? conda install hdf4=4.2.12-0 doesn’t seems to work, looks like the -0 is not accepted.

EDIT: OK found it, hdf4=4.2.12=0.

[felixmatt]

conda remove netcdf4 pip install netcdf4

worked for me as a workaround for py 3.5.

[jjhelmus]

I new build of libnetcdf, libnetcdf 4.4.1-1, was created and uploaded. This build links against libjpeg.so.9. I believe this should address the final bug in this long journey.

Closing this issue, please comment if there are still outstanding issue related to this and I will re-open.

Thanks all for helping work through this, sorry that the fix took so long.

[nicholasfm]

Hi all,

@jjhelmus I've had this issue coming up having just setup pyart on OS X with libnetcdf 4.4.1-1, anything I'm missing here?

[jjhelmus]

@nicholasfm I'm not able to reproduce any issues with libnetcdf 4.4.1-1 on OS X. Can you describe the issue you are seeing in more detail? The output of conda list would also be helpful.

[nicholasfm]

@jjhelmus it turns out the issue was a python environment issue using the old library despite everything looking in check in conda list. All sorted now though, but thank you all the same.

[Bluno]

I had a similar problem and in my case was because of packages that weren't updated. My main example was numpy and its depedencies. By the command:

conda update numpy

I resolve my issues.