Install & Run HSP2 Python on Windows

[jdkleiner]

@rburghol Ignore if you've already cracked this yourself, but I've FINALLY figured out how to install and run HSP2 on my local machine. Just ran a test using the latest updated branch specom and verified all runs smoothly.

Installation process:

1. Open a Windows PowerShell (I used a PS command-line shell within VS Code)
2. Navigate to local cloned copy of HSPsquared repo
   • PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared
3. Check if HSP2 is already installed
   • PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared> hsp2 --help
   • ^ This will return an error if you dont have hsp2 installed yet
4. Try installing hsp2
   • PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared> pip install .
   • This will likely result in a host of errors

5. Solution

   • The ONLY file you have to edit is HSPsquared/HSP2/environment.yml
   • The key is checking the version of each python package you have installed on your windows machine
     • e.g: PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared> pip show pandas
       • resulting version on my machine is Version: 1.4.2 (see .yml below)
   • Then update the package version numbers in environment.yml to match your local versions
     • See below, for me there were only 4 packages that needed their version number tweaked in the .yml

       
       name: hsp2_py38
       channels:

     • conda-forge

   dependencies:

   Aligned with Anaconda 2021.05 (May 13, 2021)

   package management

   • conda
   • conda-build

   • pip

     Running HSP2

   • python =3.8*
   • scipy # Scipy also installs numpy

     - pandas ==1.2.* # Pandas installs most scientific Python modules, such as Numpy, etc.

   • pandas ==1.4.2

     - numba ==0.53.1

   • numba ==0.56.0

     - numpy ==1.20.* # for compatibility with numba version <0.53

   • numpy ==1.22.4
   • hdf5 ==1.10.6 # latest compatible with both versions of PyTables & h5py below - doesn't work with setup.py

     - pytables ==3.6.1

   • pytables ==3.7.0
   • h5py >=3.1
   • hdf5plugin # HDF5 compression filters for h5py & jupyterlab_hdf
   • mando # for Python CLI app in HSP2tools/HSP2_CLI.py; hasn't been updated since 2017 and doesn't support Python 3.9: https://github.com/rubik/mando

6. Verifying install
   • PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared> hsp2 --help <- this should now show help options

7. Run a test cd tests/testcbp/HSP2results

   PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared\tests\testcbp\HSP2results> hsp2 import_uci PL3_5250_0001.uci PL3_5250_0001.h5
   1003 reading from wdm
   1005 reading from wdm
   1000 reading from wdm
   1001 reading from wdm
   1002 reading from wdm
   ...
   
   PS C:\Users\nrf46657\Desktop\GitHub\HSPsquared\tests\testcbp\HSP2results> hsp2 run PL3_5250_0001.h5
   2022-12-20 16:25:07.40   Processing started for file ./; saveall=True
   Called read_uci()
   2022-12-20 16:25:08.29   Simulation Start: 2001-01-01 00:00:00, Stop: 2002-01-01 00:00:00
   2022-12-20 16:25:08.29      RCHRES R001 DELT(minutes): 60
   2022-12-20 16:25:13.88         HYDR
   {DEP: 1, IVOL: 2, O1: 3, O2: 4, O3: 5, OVOL1: 6, OVOL2: 7, OVOL3: 8, PRSUPY: 9, RO: 10, ROVOL: 11, SAREA: 12, TAU: 13, USTAR: 14, VOL: 15, VOLEV: 16}
   Calling new _hydr_ with specl
   2022-12-20 16:25:17.84         ADCALC
   2022-12-20 16:25:18.15   Done; Run time is about 00:10.7 (mm:ss)

Previous install steps

Setup Python on Windows

• Need to download 3.10 (not 3.11 which is latest as of 11/2022) because numba not available on 3.11
  • https://www.python.org/downloads/release/python-3108/

TRUST="--trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100"
python -m pip install --upgrade pip $TRUST
python -m pip install numba $TRUST

# to read hdf5 files
python -m pip install h5py $TRUST

# testing the numexpr library
python -m pip install numexpr $TRUST

# pandas
python -m pip install pandas $TRUST
# this is for pytables, but note, the package is called "tables" when you request with pip

python -m pip install tables $TRUST

# for testing I added "sage" to parse equations.  Needed to deal with self-signed cert error: 
#    certificate verify failed: self signed certificate in certificate chain
python -m pip install sage  $TRUST

# pyparsing is the jam for equations
python -m pip install pyparsing $TRUST
# to develop do this
# git -c http.sslVerify=false clone https://github.com/pyparsing/pyparsing.git

python -m pip install requests $TRUST

Set Up HSPsquared on Windows

• this still has not been completed fully
• from a gitbash terminal

  
  git clone git@github.com:HARPgroup/HSPsquared.git
  cd HSPsquared
  pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org .
  # got an error that it cou;dn't find tables==3.6.1,  but I was ableto install table 3.7.0 with the following:
  pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org tables

python -m pip install --upgrade pip pip install --upgrade wheel pip install --upgrade setuptools

close terminal, open again

pip install numba

[rburghol]

This is excellent! I only got to step 4 and kinda shrugged my shoulders after the host of errors!

[rburghol]

@jdkleiner Can you start an issue at HSPsquared for this? (or search for an existing one). I think opening the dialog about which required packages need to be fixed version, and which need to be >=. FWIW, this may be limited to our pip style installs, which is not the norm, as they recommend installing with conda. The pip installer that we use was made for linux, though clearly that is not required -- this is detailed here: https://github.com/respec/HSPsquared/issues/55 .

It is super cool to know that we can do it with pip and I think very helpful to us since we can have one path for both linux and windows... but also, the issues you discovered might have real impact for the conda installs as well.

Like, why does it have a fixed version of things like pandas ==1.2.*, rather than the at least version like with h5py >=3.1. In fact, I know they've had some issues in the queue that talk about wanting to advance certain packages (like numba). https://github.com/respec/HSPsquared/issues/100

All this tells me that they may have a very good reason to keep statically linked versions, though if python is suitably robust there seems no reason (to me) that things should change drastically between version necessitating so many packages being anchored to a single package version. It seems like a mixed bag of reasons, that is, some of them may be related to HSP2 development, whereas some, linked in the issue linked above, seem related to problems within pythons numba/numpy ecosystem.

I think 2 things would help:

• start an issue on exact (==) versus at least (>=) version in the environment.yml file.
• Run whatever automated tests they have so we can see if there are problems. Which reminds me that automated testing for our new components is a must as well and would be super helpful!

[rburghol]

Note: pip commands require a full path to python/pip because I did a non-admin install:

/c/usr/local/bin/python/python -m pip install --upgrade pip --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org

For more convenient use, I can do this at the beginning and then just use pip and python without the full path.

PATH="$PATH:/c/usr/local/bin/python"
export PATH

Note: had to totally override PATH because there was an old piece in there from before my profile got wiped when my board went bad:

PATH="/c/usr/local/home/robertwb/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/r/bin:/c/Program Files (x86)/RSA SecurID Token Common:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/WindowsPowerShell/v1.0:/c/windows/System32/OpenSSH:/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/usr/bin/vendor_perl:/usr/bin/core_perl:/c/usr/local/bin/python"
export PATH

Finally:

python -m pip install . --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org

[jdkleiner]

@rburghol FYI this issue is outdated, See this issue for optimal method for running HSP2 on windows: https://github.com/HARPgroup/HSPsquared/issues/55