installation/use on Hera

[AndrewEichmann-NOAA]

Hello,

Please let me know if this is not the correct forum for this issue.

I attempted to use emcpy on Hera according to what I believe was the latest presentation on the package: https://docs.google.com/presentation/d/1wgiUaqkpP1PiYD4ng3YIZVbyhWu7PEY_otPdHR9P2pA/edit#slide=id.gf9c66c7613_0_147

-bash-4.2$ module list

Currently Loaded Modules: 1) rocoto/1.3.3 2) intelpython/2021.3.0

-bash-4.2$ module use /scratch1/NCEPDEV/da/Cory.R.Martin/Modulefiles/python/ -bash-4.2$ module load anaconda/kevin -bash-4.2$ module load emcpy/0.1 /usr/bin/which: no fi_info in (/scratch1/NCEPDEV/da/Kevin.Dougherty/anaconda3/bin:/apps/oneapi/intelpython/latest/bin:/apps/oneapi/intelpython/latest/condabin:/scratch1/NCEPDEV/global/Fanglin.Yang/save/VRFY/vsdb/nwprod/util/exec:/apps/rocoto/1.3.3/bin:/home/glopara/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/apps/local/bin:/apps/local/sbin:/apps/slurm/default/tools:/apps/slurm/default/bin:/apps/slurm/default/sbin:/apps/slurm/tools/sbank/bin:/home/Andrew.Eichmann/bin) (UFO) -bash-4.2$ cd emcpy/examples/histograms/ (UFO) -bash-4.2$ python histogram.py Traceback (most recent call last): File "histogram.py", line 3, in from emcpy.plots.plots import Histogram File "/scratch1/NCEPDEV/da/Cory.R.Martin/Python/conda/local/lib/python3.7/site-packages/emcpy/init.py", line 9, in import emcpy.io.netCDF File "/scratch1/NCEPDEV/da/Cory.R.Martin/Python/conda/local/lib/python3.7/site-packages/emcpy/io/netCDF.py", line 10, in from netCDF4 import Dataset as _Dataset ModuleNotFoundError: No module named 'netCDF4' (UFO) -bash-4.2$

Previously I've used my own conda installation that has netCDF4 and which I've uninstalled here, but there seems to be assumptions about the environment that are not met.

An attempt to install with pip also fails:

(UFO) -bash-4.2$ cd ../../ (UFO) -bash-4.2$ ls examples LICENSE README.md setup.cfg setup.py src (UFO) -bash-4.2$ pip install . Processing /scratch1/NCEPDEV/da/Andrew.Eichmann/emcpy/emcpy Requirement already satisfied: numpy in /scratch1/NCEPDEV/da/Kevin.Dougherty/anaconda3/lib/python3.7/site-packages (from emcpy==0.0.1) (1.15.1) Requirement already satisfied: scipy in /scratch1/NCEPDEV/da/Kevin.Dougherty/anaconda3/lib/python3.7/site-packages (from emcpy==0.0.1) (1.1.0) Requirement already satisfied: pandas in /scratch1/NCEPDEV/da/Kevin.Dougherty/anaconda3/lib/python3.7/site-packages (from emcpy==0.0.1) (0.23.4) Collecting netcdf4 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b0d70ed0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/bd/5a/ac3508a0dbc737e6b3a45333fe5e3cabbcaed17147a58612b29ffe20016e/netCDF4-1.5.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b0d70c10>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/bd/5a/ac3508a0dbc737e6b3a45333fe5e3cabbcaed17147a58612b29ffe20016e/netCDF4-1.5.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b0d70790>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/bd/5a/ac3508a0dbc737e6b3a45333fe5e3cabbcaed17147a58612b29ffe20016e/netCDF4-1.5.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b0cc6150>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/bd/5a/ac3508a0dbc737e6b3a45333fe5e3cabbcaed17147a58612b29ffe20016e/netCDF4-1.5.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b0cc61d0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/bd/5a/ac3508a0dbc737e6b3a45333fe5e3cabbcaed17147a58612b29ffe20016e/netCDF4-1.5.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/bd/5a/ac3508a0dbc737e6b3a45333fe5e3cabbcaed17147a58612b29ffe20016e/netCDF4-1.5.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b0cc6f50>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

(UFO) -bash-4.2$

Thanks, Andy

[CoryMartin-NOAA]

@AndrewEichmann-NOAA does unloading intelpython beforehand make any difference? You're unable to install with Pip becasue it is attempting and failing to download dependencies from sites that are blocked by the firewall. You can install it with pip by skipping installing the dependencies but that might not work in your environment still.

[AndrewEichmann-NOAA]

@CoryMartin-NOAA Ok not having intelpython works. Should the pip install cover dependencies outside of Hera/Orion?

[CoryMartin-NOAA]

@AndrewEichmann-NOAA yes pip install will work on orion and many other computers, Hera just blocks many external websites Failed to establish a new connection: [Errno 101] Network is unreachable')':

[AndrewEichmann-NOAA]

@CoryMartin-NOAA I was successful on Orion with the instruction in the presentation but it seems that my home situation (from within conda) will require some more work. Thanks!