Py2.7 barfs when importing netCDF4

[prjemian]

Jessica (29ID) reports:

(base) bash-4.2$ ipython
Python 2.7.16 |Anaconda custom (64-bit)| (default, Aug 22 2019, 16:00:36) 
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Caching the list of root modules, please wait!
(This will only be done once - type '%rehashx' to reset cache!)

In [1]: import netCDF4
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9588a3d4fb24> in <module>()
----> 1 import netCDF4

/APSshare/anaconda/x86_64/lib/python2.7/site-packages/netCDF4/__init__.py in <module>()
      1 # init for netCDF4. package
      2 # Docstring comes from extension module _netCDF4.
----> 3 from ._netCDF4 import *
      4 # Need explicit imports for names beginning with underscores
      5 from ._netCDF4 import __doc__, __pdoc__

ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory

[prjemian]

From the HISTORY.txt logs, openssl was upgraded as a side-effect while upgrading spec2nexus on 2019-09-23.

        The following packages will be UPDATED:

...
          openssl                                 1.0.2r-h7b6447c_0 --> 1.1.1d-h7b6447c_1
...
          spec2nexus         prjemian::spec2nexus-2020.0.0-pyh39e3~ --> aps-anl-tag::spec2nexus-2021.1.3-h39e3cac_0

Using conda as a package manager, this happens a lot lately (side-effect updates to other packages) and openssl is one of those frequently updated.

[prjemian]

This looks like the same error as ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory

[prjemian]

From that issue (now marked Closed), there are known problems with underlying libgcc-ng and these will not be resolved. Our installed versions are well past those mentioned in the remarks from 2019-03. Upgrade of openssl has not helped (got the same error). It may be possible (and best at this time) to revert openssl back to the version installed before the spec2nexus update. Jessica reports that netCDF4 was working for her in the summer operations cycle which support this idea to revert back to openssl 1.0.2.

[prjemian]

BTW, here is what we have right now (before the revert to previous openssl):

(base) bash-4.2$ conda list gcc
# packages in environment at /APSshare/anaconda/x86_64:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
gcc-5                     5.2.0                         1    psi4
gcc-5-mp                  5.2.0                         0    psi4
gcc_impl_linux-64         7.3.0                habb00fd_1  
gcc_linux-64              7.3.0                h553295d_7  
libgcc                    7.2.0                h69d50b8_2  
libgcc-ng                 9.1.0                hdf63c60_0  
(base) bash-4.2$ conda list cdf
# packages in environment at /APSshare/anaconda/x86_64:
#
# Name                    Version                   Build  Channel
libnetcdf                 4.3.3.1                       3  
netcdf4                   1.2.2               np110py27_0  
(base) bash-4.2$ conda list ssl
# packages in environment at /APSshare/anaconda/x86_64:
#
# Name                    Version                   Build  Channel
openssl                   1.1.1d               h7b6447c_2  
pyopenssl                 19.0.0                   py27_0  
ssl_match_hostname        3.7.0.1                  py27_0  

[prjemian]

After reverting to openssl-1.0.2, problem is moved along a bit:

In [1]: import netCDF4
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9588a3d4fb24> in <module>()
----> 1 import netCDF4

/APSshare/anaconda/x86_64/lib/python2.7/site-packages/netCDF4/__init__.py in <module>()
      1 # init for netCDF4. package
      2 # Docstring comes from extension module _netCDF4.
----> 3 from ._netCDF4 import *
      4 # Need explicit imports for names beginning with underscores
      5 from ._netCDF4 import __doc__, __pdoc__

ImportError: libhdf5_hl.so.10: cannot open shared object file: No such file or directory

[prjemian]

This looks related but a simple directory listing shows what may be the real problem here (100 exists while 10 is sought):

(base) bash-4.2$ ls -lF /APSshare/anaconda/x86_64/lib/libhdf5_hl*
-rw-rw-r-- 2 epicsadm aesbc 237570 May  9  2018 /APSshare/anaconda/x86_64/lib/libhdf5_hl.a
-rw-rw-r-- 2 epicsadm aesbc  13902 May  9  2018 /APSshare/anaconda/x86_64/lib/libhdf5_hl_cpp.a
-rwxrwxr-x 1 epicsadm aesbc   1174 Sep 23 13:33 /APSshare/anaconda/x86_64/lib/libhdf5_hl_cpp.la*
lrwxrwxrwx 1 epicsadm aesbc     25 Sep 23 13:33 /APSshare/anaconda/x86_64/lib/libhdf5_hl_cpp.so -> libhdf5_hl_cpp.so.100.1.1*
lrwxrwxrwx 1 epicsadm aesbc     25 Sep 23 13:33 /APSshare/anaconda/x86_64/lib/libhdf5_hl_cpp.so.100 -> libhdf5_hl_cpp.so.100.1.1*
-rwxrwxr-x 2 epicsadm aesbc  19416 May  9  2018 /APSshare/anaconda/x86_64/lib/libhdf5_hl_cpp.so.100.1.1*
-rwxrwxr-x 1 epicsadm aesbc   1061 Sep 23 13:33 /APSshare/anaconda/x86_64/lib/libhdf5_hl.la*
lrwxrwxrwx 1 epicsadm aesbc     21 Sep 23 13:33 /APSshare/anaconda/x86_64/lib/libhdf5_hl.so -> libhdf5_hl.so.100.1.0*
lrwxrwxrwx 1 epicsadm aesbc     21 Sep 23 13:33 /APSshare/anaconda/x86_64/lib/libhdf5_hl.so.100 -> libhdf5_hl.so.100.1.0*
-rwxrwxr-x 2 epicsadm aesbc 163840 May  9  2018 /APSshare/anaconda/x86_64/lib/libhdf5_hl.so.100.1.0*

Note installed hdf packages:

(base) bash-4.2$ conda list hdf
# packages in environment at /APSshare/anaconda/x86_64:
#
# Name                    Version                   Build  Channel
hdf4                      4.2.12                        0  
hdf5                      1.10.2               hba1933b_1  
python-hdf4               0.9                      pypi_0    pypi

[prjemian]

cd /APSshare/anaconda/x86_64/lib/
ln -s libhdf5_hl.so.100 libhdf5_hl.so.10

but now it wants

ImportError: libhdf5.so.10: cannot open shared object file: No such file or directory

We have:

(base) bash-4.2$ ls -l libhdf5*.so*
lrwxrwxrwx 1 epicsadm aesbc      22 Sep 23 13:33 libhdf5_cpp.so -> libhdf5_cpp.so.102.0.0
lrwxrwxrwx 1 epicsadm aesbc      22 Sep 23 13:33 libhdf5_cpp.so.102 -> libhdf5_cpp.so.102.0.0
-rwxrwxr-x 2 epicsadm aesbc  570392 May  9  2018 libhdf5_cpp.so.102.0.0
lrwxrwxrwx 1 epicsadm aesbc      26 Sep 23 13:33 libhdf5_fortran.so -> libhdf5_fortran.so.100.1.1
lrwxrwxrwx 1 epicsadm aesbc      26 Sep 23 13:33 libhdf5_fortran.so.100 -> libhdf5_fortran.so.100.1.1
-rwxrwxr-x 2 epicsadm aesbc  381336 May  9  2018 libhdf5_fortran.so.100.1.1
lrwxrwxrwx 1 epicsadm aesbc      25 Sep 23 13:33 libhdf5_hl_cpp.so -> libhdf5_hl_cpp.so.100.1.1
lrwxrwxrwx 1 epicsadm aesbc      25 Sep 23 13:33 libhdf5_hl_cpp.so.100 -> libhdf5_hl_cpp.so.100.1.1
-rwxrwxr-x 2 epicsadm aesbc   19416 May  9  2018 libhdf5_hl_cpp.so.100.1.1
lrwxrwxrwx 1 epicsadm aesbc      28 Sep 23 13:33 libhdf5hl_fortran.so -> libhdf5hl_fortran.so.100.0.2
lrwxrwxrwx 1 epicsadm aesbc      28 Sep 23 13:33 libhdf5hl_fortran.so.100 -> libhdf5hl_fortran.so.100.0.2
-rwxrwxr-x 2 epicsadm aesbc  159720 May  9  2018 libhdf5hl_fortran.so.100.0.2
lrwxrwxrwx 1 epicsadm aesbc      21 Sep 23 13:33 libhdf5_hl.so -> libhdf5_hl.so.100.1.0
lrwxrwxrwx 1 epicsadm aesbc      17 Oct  7 17:58 libhdf5_hl.so.10 -> libhdf5_hl.so.100
lrwxrwxrwx 1 epicsadm aesbc      21 Sep 23 13:33 libhdf5_hl.so.100 -> libhdf5_hl.so.100.1.0
-rwxrwxr-x 2 epicsadm aesbc  163840 May  9  2018 libhdf5_hl.so.100.1.0
lrwxrwxrwx 1 epicsadm aesbc      18 Sep 23 13:33 libhdf5.so -> libhdf5.so.101.1.0
lrwxrwxrwx 1 epicsadm aesbc      18 Sep 23 13:33 libhdf5.so.101 -> libhdf5.so.101.1.0
-rwxrwxr-x 1 epicsadm aesbc 4071936 Sep 23 13:33 libhdf5.so.101.1.0

More such soft links here are likely to start breaking other things.

[prjemian]

Trying to update hdf5 tries to update openssl as a consequence and no change to hdf5 package version. Not doing that.

(base) bash-4.2$ conda update hdf5
Collecting package metadata: done
Solving environment: | 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - dgursoy/linux-64::tomopy==1.0.1=py27_0
  - dgursoy/linux-64::dxchange==0.1.1=py27_0
  - defaults/linux-64::pytables==3.2.2=np110py27_1
  - defaults/linux-64::netcdf4==1.2.2=np110py27_0
  - defaults/linux-64::libnetcdf==4.3.3.1=3
  - defaults/linux-64::blaze==0.11.3=py27_0
  - defaults/linux-64::bcolz==1.2.1=py27h637b7d7_0
  - defaults/linux-64::anaconda==custom=py27h4a00acb_0
done

## Package Plan ##

  environment location: /APSshare/anaconda/x86_64

  added / updated specs:
    - hdf5

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.7.12               |           py27_0         3.0 MB
    conda-package-handling-1.6.0|   py27h7b6447c_0         865 KB
    cryptography-2.7           |   py27h1ba5d50_0         602 KB
    curl-7.65.3                |       hbc83047_0         141 KB
    hdf4-4.2.11                |                0         3.3 MB
    krb5-1.16.1                |       h173b8e3_7         1.4 MB
    libarchive-3.3.3           |       h5d8350f_2         1.5 MB
    libcurl-7.65.3             |       h20c2e04_0         588 KB
    libpq-11.2                 |       h20c2e04_0         2.7 MB
    libssh2-1.8.2              |       h1ba5d50_0         250 KB
    psycopg2-2.8.3             |   py27h1ba5d50_0         170 KB
    pycurl-7.43.0.3            |   py27h1ba5d50_0          69 KB
    python-2.7.16              |       h9bab390_7        12.4 MB
    ------------------------------------------------------------
                                           Total:        26.9 MB

The following NEW packages will be INSTALLED:

  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.0-py27h7b6447c_0

The following packages will be UPDATED:

  conda                                       4.6.14-py27_0 --> 4.7.12-py27_0
  cryptography                         2.3.1-py27hc365091_0 --> 2.7-py27h1ba5d50_0
  curl                             pkgs/free::curl-7.45.0-0 --> pkgs/main::curl-7.65.3-hbc83047_0
  krb5                                    1.16.1-hc83ff2d_6 --> 1.16.1-h173b8e3_7
  libarchive                               3.3.3-h7d0bbab_1 --> 3.3.3-h5d8350f_2
  libcurl                                 7.61.1-heec0ca6_0 --> 7.65.3-h20c2e04_0
  libpq                                     10.5-h1ad7b7a_0 --> 11.2-h20c2e04_0
  libssh2                                  1.8.0-h9cfc8f7_4 --> 1.8.2-h1ba5d50_0
  openssl                                 1.0.2t-h7b6447c_1 --> 1.1.1d-h7b6447c_2
  psycopg2                             2.7.5-py27hb7f436b_0 --> 2.8.3-py27h1ba5d50_0
  pycurl                            7.43.0.2-py27hb7f436b_0 --> 7.43.0.3-py27h1ba5d50_0
  python                                  2.7.15-h77bded6_2 --> 2.7.16-h9bab390_7

The following packages will be DOWNGRADED:

  hdf4                                             4.2.12-0 --> 4.2.11-0

[prjemian]

Web search for related issues: https://www.google.com/search?q=ImportError%3A+libhdf5.so.10%3A+cannot+open+shared+object+file%3A+No+such+file+or+directory&ie=utf-8&oe=utf-8&client=firefox-b-1-e

[prjemian]

Getting closer, but still an upgrade to openssl

(base) bash-4.2$ conda install h5py=2.6.0
Collecting package metadata: done
Solving environment: | 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - dgursoy/linux-64::tomopy==1.0.1=py27_0
  - dgursoy/linux-64::dxchange==0.1.1=py27_0
  - defaults/linux-64::pytables==3.2.2=np110py27_1
  - defaults/linux-64::netcdf4==1.2.2=np110py27_0
  - defaults/linux-64::libnetcdf==4.3.3.1=3
  - defaults/linux-64::blaze==0.11.3=py27_0
  - defaults/linux-64::bcolz==1.2.1=py27h637b7d7_0
  - defaults/linux-64::anaconda==custom=py27h4a00acb_0
done

## Package Plan ##

  environment location: /APSshare/anaconda/x86_64

  added / updated specs:
    - h5py=2.6.0

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.7.12               |           py27_0         3.0 MB
    conda-package-handling-1.6.0|   py27h7b6447c_0         865 KB
    cryptography-2.7           |   py27h1ba5d50_0         602 KB
    curl-7.65.3                |       hbc83047_0         141 KB
    hdf4-4.2.11                |                0         3.3 MB
    krb5-1.16.1                |       h173b8e3_7         1.4 MB
    libarchive-3.3.3           |       h5d8350f_2         1.5 MB
    libcurl-7.65.3             |       h20c2e04_0         588 KB
    libpq-11.2                 |       h20c2e04_0         2.7 MB
    libssh2-1.8.2              |       h1ba5d50_0         250 KB
    psycopg2-2.8.3             |   py27h1ba5d50_0         170 KB
    pycurl-7.43.0.3            |   py27h1ba5d50_0          69 KB
    python-2.7.16              |       h9bab390_7        12.4 MB
    ------------------------------------------------------------
                                           Total:        26.9 MB

The following NEW packages will be INSTALLED:

  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.0-py27h7b6447c_0

The following packages will be UPDATED:

  conda                                       4.6.14-py27_0 --> 4.7.12-py27_0
  cryptography                         2.3.1-py27hc365091_0 --> 2.7-py27h1ba5d50_0
  curl                             pkgs/free::curl-7.45.0-0 --> pkgs/main::curl-7.65.3-hbc83047_0
  krb5                                    1.16.1-hc83ff2d_6 --> 1.16.1-h173b8e3_7
  libarchive                               3.3.3-h7d0bbab_1 --> 3.3.3-h5d8350f_2
  libcurl                                 7.61.1-heec0ca6_0 --> 7.65.3-h20c2e04_0
  libpq                                     10.5-h1ad7b7a_0 --> 11.2-h20c2e04_0
  libssh2                                  1.8.0-h9cfc8f7_4 --> 1.8.2-h1ba5d50_0
  openssl                                 1.0.2t-h7b6447c_1 --> 1.1.1d-h7b6447c_2
  psycopg2                             2.7.5-py27hb7f436b_0 --> 2.8.3-py27h1ba5d50_0
  pycurl                            7.43.0.2-py27hb7f436b_0 --> 7.43.0.3-py27h1ba5d50_0
  python                                  2.7.15-h77bded6_2 --> 2.7.16-h9bab390_7

The following packages will be SUPERSEDED by a higher-priority channel:

  h5py                 pkgs/main::h5py-2.8.0-py27h8d01980_0 --> pkgs/free::h5py-2.6.0-np110py27_1
  hdf5                    pkgs/main::hdf5-1.10.2-hba1933b_1 --> pkgs/free::hdf5-1.8.16-0

The following packages will be DOWNGRADED:

  hdf4                                             4.2.12-0 --> 4.2.11-0

[prjemian]

This looks simpler (combine the two restrictions h5py=2.6.0 openssl=1.0.2):

(base) bash-4.2$ conda install h5py=2.6.0 openssl=1.0.2
Collecting package metadata: done
Solving environment: | 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - dgursoy/linux-64::tomopy==1.0.1=py27_0
  - dgursoy/linux-64::dxchange==0.1.1=py27_0
  - defaults/linux-64::pytables==3.2.2=np110py27_1
  - defaults/linux-64::netcdf4==1.2.2=np110py27_0
  - defaults/linux-64::libnetcdf==4.3.3.1=3
  - defaults/linux-64::blaze==0.11.3=py27_0
  - defaults/linux-64::bcolz==1.2.1=py27h637b7d7_0
  - defaults/linux-64::anaconda==custom=py27h4a00acb_0
done

## Package Plan ##

  environment location: /APSshare/anaconda/x86_64

  added / updated specs:
    - h5py=2.6.0
    - openssl=1.0.2

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.7.12               |           py27_0         3.0 MB
    conda-package-handling-1.6.0|   py27h7b6447c_0         865 KB
    ------------------------------------------------------------
                                           Total:         3.9 MB

The following NEW packages will be INSTALLED:

  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.0-py27h7b6447c_0

The following packages will be UPDATED:

  conda                                       4.6.14-py27_0 --> 4.7.12-py27_0

The following packages will be SUPERSEDED by a higher-priority channel:

  h5py                 pkgs/main::h5py-2.8.0-py27h8d01980_0 --> pkgs/free::h5py-2.6.0-np110py27_1
  hdf5                    pkgs/main::hdf5-1.10.2-hba1933b_1 --> pkgs/free::hdf5-1.8.16-0

[prjemian]

After installing that, looks good to me:

(base) bash-4.2$ ipython
Python 2.7.15 |Anaconda custom (64-bit)| (default, Oct 10 2018, 21:32:13) 
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import netCDF4

In [2]: 

Hope nothing else has been broken. Note that conda was updated (an earlier goal).

[prjemian]

I have more confidence that this will work when looking at the libhdf5*.so* files:

(base) bash-4.2$ ls -l libhdf5*.so*
lrwxrwxrwx 1 epicsadm aesbc      21 Oct  7 18:20 libhdf5_cpp.so -> libhdf5_cpp.so.11.0.0
lrwxrwxrwx 1 epicsadm aesbc      21 Oct  7 18:20 libhdf5_cpp.so.11 -> libhdf5_cpp.so.11.0.0
-rwxrwxr-x 2 epicsadm aesbc  593636 Apr  4  2016 libhdf5_cpp.so.11.0.0
lrwxrwxrwx 1 epicsadm aesbc      24 Oct  7 18:20 libhdf5_hl_cpp.so -> libhdf5_hl_cpp.so.11.0.0
lrwxrwxrwx 1 epicsadm aesbc      24 Oct  7 18:20 libhdf5_hl_cpp.so.11 -> libhdf5_hl_cpp.so.11.0.0
-rwxrwxr-x 2 epicsadm aesbc   13466 Apr  4  2016 libhdf5_hl_cpp.so.11.0.0
lrwxrwxrwx 1 epicsadm aesbc      20 Oct  7 18:20 libhdf5_hl.so -> libhdf5_hl.so.10.0.2
lrwxrwxrwx 1 epicsadm aesbc      20 Oct  7 18:20 libhdf5_hl.so.10 -> libhdf5_hl.so.10.0.2
-rwxrwxr-x 2 epicsadm aesbc  140931 Apr  4  2016 libhdf5_hl.so.10.0.2
lrwxrwxrwx 1 epicsadm aesbc      17 Oct  7 18:20 libhdf5.so -> libhdf5.so.10.1.0
lrwxrwxrwx 1 epicsadm aesbc      17 Oct  7 18:20 libhdf5.so.10 -> libhdf5.so.10.1.0
-rwxrwxr-x 2 epicsadm aesbc 3234498 Apr  4  2016 libhdf5.so.10.1.0

Links and versions all look reasonable. Something suspicious about version 100 or 101 with the HDF5 libraries.