Install HSP2 Python on Linux

rburghol commented 2 years ago

Install system wide

Note: Current working branch is fixjup -- the commit from fixjup has been included in develop, so that may actually be ready to use, but as of 7/1/2023 only fixjup has been tested.
Need to run as root. This gets a lot of warnings, but hsp2 can then be run by any analyst (and modelers) -- since hsp2 is the only thing we use Python for, there is little risk of the central install.
Notes:
- If you originally installed as non-root user, the install will go into $HOME/.local/lib, so I had to do this: rm /home/rob/.local/lib/python3.8/site-packages/HSP* -R. Perhaps running the uninstall from pip from my users account would have achieved the same?
- setup.py appears to be the script that tells python/pip how to install hsp2, and the origin post that added linux install is here: https://github.com/respec/HSPsquared/issues/55
- Note: the -e switch on pip install breaks hsp2 for analysts (but not me), tho seems to create a more detailed hsp executable, and places it in /usr/bin, whereas without the -e, it goes in /usr/local/bin.
Most recent (had to upgrade pip Pillow first)

Helper utilities

wdmtoolbox: the from Tim Cera
- Main page: https://timcera.bitbucket.io/wdmtoolbox/docs/index.html
- Installation: https://timcera.bitbucket.io/wdmtoolbox/docs/readme.html

cd /opt/model/HSPsquared
sudo python3 -m pip install --upgrade pip
sudo python3 -m pip install --upgrade Pillow
# don't do this one as the analysts get errors, perhaps they cannot read certain libraries
#sudo pip install -e .
sudo pip install .

Subsequent Version Cloning

python3 -m pip install --upgrade pip
pip install -e .
Try to run:

May have to update dependencies, for example, hsp2 run OR1_7700_7980.h5 gave me:

raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (Pillow 7.0.0 (/usr/lib/python3/dist-packages), Requirement.parse('pillow>=7.1.0'), {'bokeh'})

Updated Pillow:


python3 -m pip install --upgrade pip
python3 -m pip install --upgrade Pillow

Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: Pillow in /usr/lib/python3/dist-packages (7.0.0) Collecting Pillow Downloading Pillow-9.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 19.9 MB/s eta 0:00:00 Installing collected packages: Pillow Successfully installed Pillow-9.1.1



### Original Install
- git clone the non-Conda repo: `git clone https://github.com/timcera/HSPsquared.git`
- install pip `sudo apt-get install pip`
- get pandas `pip install pandas` (had to do this  because `pip install .` failed with `ModuleNotFoundError: No module named 'pandas'`
- get numba `pip install numba` (same as above, `ModuleNotFoundError: No module named 'numba'`)
  - `ERROR: tables 3.7.0 has requirement numpy>=1.19.0, but you'll have numpy 1.17.4 which is incompatible.`
  - Install earlier tables `pip install tables==3.6.1`
  - Then re-do `pip install numba`
- Then Joy! 
  - `pip install -e .`
  - Yields: `Successfully installed HSPsquared`
- Install hdf5 db see HARPgroup/vahydro#481

rburghol commented 2 years ago

Try to run:


# convert the UCI fileto h5
hsp2 import_uci test10.uci test10.h5
# do the run
hsp2 run test10.h5

# run output
2022-01-06 18:48:56.74   Processing started for file test10.h5; saveall=False
2022-01-06 18:49:00.76   Simulation Start: 1976-01-01 00:00:00, Stop: 1977-01-01 00:00:00
2022-01-06 18:49:00.76      PERLND P001 DELT(minutes): 60
2022-01-06 18:49:02.77         SNOW
2022-01-06 18:49:08.95         PWATER
2022-01-06 18:49:14.81         PSTEMP
Save DataFrame Empty for RESULTS/PERLND_P001/PSTEMP
2022-01-06 18:49:15.91         PWTGAS
Save DataFrame Empty for RESULTS/PERLND_P001/PWTGAS
2022-01-06 18:49:18.55      RCHRES R001 DELT(minutes): 60
2022-01-06 18:49:18.66         HYDR
2022-01-06 18:49:25.71         ADCALC
2022-01-06 18:49:27.64         CONS
Save DataFrame Empty for RESULTS/RCHRES_R001/CONS
2022-01-06 18:49:29.48         HTRCH
Save DataFrame Empty for RESULTS/RCHRES_R001/HTRCH
2022-01-06 18:49:32.94         SEDTRN
Save DataFrame Empty for RESULTS/RCHRES_R001/SEDTRN
2022-01-06 18:49:44.82         GQUAL
2022-01-06 18:49:48.04            Error count 1: GQUAL: the value of phflag is 1, but timeseries PHVAL is not available as input
2022-01-06 18:49:48.04            Error count 1: GQUAL: the value of phytfg is 1, but timeseries PHYTO is not available as input
Save DataFrame Empty for RESULTS/RCHRES_R001/GQUAL
2022-01-06 18:49:48.05         RQUAL
2022-01-06 18:50:54.99      GENER G001 DELT(minutes): 60
GENER 'G001' encountered unsupported feature during initialization and may not function correctly. Unsupported feature: 'Invalid SVOL. GENER module does not currently support reading TimeSeries for 'RCHRES''
2022-01-06 18:50:54.99      RCHRES R002 DELT(minutes): 60
2022-01-06 18:50:55.47         HYDR
2022-01-06 18:51:01.86         ADCALC
2022-01-06 18:51:01.86         CONS
Save DataFrame Empty for RESULTS/RCHRES_R002/CONS
2022-01-06 18:51:01.87         HTRCH
Save DataFrame Empty for RESULTS/RCHRES_R002/HTRCH
2022-01-06 18:51:01.87         SEDTRN
2022-01-06 18:51:01.88            Error count 7543: SEDTRN: Warning -- bed storage of sediment size fraction sand is empty
Save DataFrame Empty for RESULTS/RCHRES_R002/SEDTRN
2022-01-06 18:51:01.88         GQUAL
2022-01-06 18:51:02.67            Error count 1: GQUAL: the value of phflag is 1, but timeseries PHVAL is not available as input
2022-01-06 18:51:02.67            Error count 1: GQUAL: the value of phytfg is 1, but timeseries PHYTO is not available as input
Save DataFrame Empty for RESULTS/RCHRES_R002/GQUAL
2022-01-06 18:51:02.67         RQUAL
Save DataFrame Empty for RESULTS/RCHRES_R002/NUTRX
2022-01-06 18:51:02.92      RCHRES R003 DELT(minutes): 60
2022-01-06 18:51:03.39         HYDR
2022-01-06 18:51:03.44         ADCALC
2022-01-06 18:51:03.44         CONS
Save DataFrame Empty for RESULTS/RCHRES_R003/CONS
2022-01-06 18:51:03.44         HTRCH
Save DataFrame Empty for RESULTS/RCHRES_R003/HTRCH
2022-01-06 18:51:03.45         SEDTRN
2022-01-06 18:51:03.46            Error count 6055: SEDTRN: Warning -- bed storage of sediment size fraction sand is empty
Save DataFrame Empty for RESULTS/RCHRES_R003/SEDTRN
2022-01-06 18:51:03.46         GQUAL
2022-01-06 18:51:04.08            Error count 1: GQUAL: the value of phflag is 1, but timeseries PHVAL is not available as input
2022-01-06 18:51:04.08            Error count 1: GQUAL: the value of phytfg is 1, but timeseries PHYTO is not available as input
Save DataFrame Empty for RESULTS/RCHRES_R003/GQUAL
2022-01-06 18:51:04.08         RQUAL
Save DataFrame Empty for RESULTS/RCHRES_R003/NUTRX
2022-01-06 18:51:04.31      RCHRES R004 DELT(minutes): 60
2022-01-06 18:51:04.66         HYDR
2022-01-06 18:51:04.71         ADCALC
2022-01-06 18:51:04.71         CONS
Save DataFrame Empty for RESULTS/RCHRES_R004/CONS
2022-01-06 18:51:04.72         HTRCH
2022-01-06 18:51:04.72            Error count 1: HTRCH: Water temperature is above 66 C (150 F) -- In most cases, this indicates an instability in advection
Save DataFrame Empty for RESULTS/RCHRES_R004/HTRCH
2022-01-06 18:51:04.72         SEDTRN
2022-01-06 18:51:04.74            Error count 6838: SEDTRN: Warning -- bed storage of sediment size fraction sand is empty
Save DataFrame Empty for RESULTS/RCHRES_R004/SEDTRN
2022-01-06 18:51:04.74         GQUAL
2022-01-06 18:51:05.60            Error count 1: GQUAL: the value of phflag is 1, but timeseries PHVAL is not available as input
2022-01-06 18:51:05.60            Error count 1: GQUAL: the value of phytfg is 1, but timeseries PHYTO is not available as input
Save DataFrame Empty for RESULTS/RCHRES_R004/GQUAL
2022-01-06 18:51:05.60         RQUAL
Save DataFrame Empty for RESULTS/RCHRES_R004/NUTRX
2022-01-06 18:51:05.86      IMPLND I001 DELT(minutes): 60
2022-01-06 18:51:05.92         SNOW
2022-01-06 18:51:06.03         IWATER
2022-01-06 18:51:07.88         SOLIDS
Save DataFrame Empty for RESULTS/IMPLND_I001/SOLIDS
2022-01-06 18:51:08.53         IWTGAS
Save DataFrame Empty for RESULTS/IMPLND_I001/IWTGAS
2022-01-06 18:51:09.43         IQUAL
Save DataFrame Empty for RESULTS/IMPLND_I001/IQUAL
2022-01-06 18:51:11.23      RCHRES R005 DELT(minutes): 60
2022-01-06 18:51:11.45         HYDR
2022-01-06 18:51:11.50         ADCALC
2022-01-06 18:51:11.51         CONS
Save DataFrame Empty for RESULTS/RCHRES_R005/CONS
2022-01-06 18:51:11.51         HTRCH
2022-01-06 18:51:11.52            Error count 1: HTRCH: Water temperature is above 66 C (150 F) -- In most cases, this indicates an instability in advection
Save DataFrame Empty for RESULTS/RCHRES_R005/HTRCH
2022-01-06 18:51:11.52         SEDTRN
Save DataFrame Empty for RESULTS/RCHRES_R005/SEDTRN
2022-01-06 18:51:11.53         GQUAL
2022-01-06 18:51:12.42            Error count 1: GQUAL: the value of phflag is 1, but timeseries PHVAL is not available as input
2022-01-06 18:51:12.42            Error count 1: GQUAL: the value of phytfg is 1, but timeseries PHYTO is not available as input
Save DataFrame Empty for RESULTS/RCHRES_R005/GQUAL
2022-01-06 18:51:12.42         RQUAL
Save DataFrame Empty for RESULTS/RCHRES_R005/NUTRX
2022-01-06 18:51:12.68      GENER G002 DELT(minutes): 60
GENER 'G002' encountered unsupported feature during initialization and may not function correctly. Unsupported feature: 'Invalid SVOL. GENER module does not currently support reading TimeSeries for 'RCHRES''
2022-01-06 18:51:12.68   Done; Run time is about 02:15.9 (mm:ss)

rburghol commented 2 years ago

Got error after update:

...
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (Jinja2 2.10.1 (/usr/lib/python3/dist-packages), Requirement.parse('jinja2>=3.0'), {'nbconvert'})

Did: python3 -m pip install --upgrade Jinja2 Ran ok after.

rburghol commented 2 years ago

See where HSP2 is installed: fgrep HSPsquared /files.dir |grep -v opt

See how it is linked to executable: cat /usr/local/lib/python3.8/dist-packages/HSPsquared.egg-link

rburghol commented 2 years ago

Need 3.9 python on dev environment on deq1

sudo apt-get install python3.9
which python3
ls -lhrt /usr/bin/python3
sudo rm /usr/bin/python3
sudo ln -s /usr/bin/python3.9 /usr/bin/python3

rburghol commented 1 year ago

Updated 1/25/2023 (after ubuntu updates):

sudo pip install .
# complained and had errors at the end
sudo python3 -m pip install --upgrade pip
# install was OK, but then model failed during run
sudo python3 -m pip install --upgrade urllib3
sudo python3 -m pip install --upgrade chardet
sudo pip install .

HARPgroup / HSPsquared