Add script to help install executables

mwtoews commented 2 years ago

This handy script does a few things:

Download a release (using GitHub Release API); by default it will pick the latest
Extract the files to a directory, specified or suggestions via :select choice
For macOS and Linux, set the executable bit (e.g. chmod +x)
Show a listing of software and versions that were installed

The script uses only standard packages, and works with Python 3.6 or later. It's only been tested on Linux and Windows. I've tested each release so far, up to 8.0. The :select choice could be empty in some cases. There might be additional directories to suggest installing on different systems. Feedback and testing is welcome.

Examples

Show help:

$ python get_modflow_executables.py --help
usage: get_modflow_executables.py [-h] [--release-id RELEASE_ID] [--force] BINDIR

positional arguments:
  BINDIR                directory to extract executables; use ':select' to help choose

optional arguments:
  -h, --help            show this help message and exit
  --release-id RELEASE_ID
                        release_id (default: latest)
  --force               force re-download

Basic example:

$ mkdir bin
$ python get_modflow_executables.py bin
fetched release '8.0'
using previous download '/home/mtoews/Downloads/linux-8.0.zip' (use --force to re-download)
extracting files to '/data/mtoews/src/USGS-executables/bin'
installed 26 executables:
crt (1.3.1)         mf6 (6.3.0)         mp6 (6.0.1)         triangle (1.6)
gridgen (1.0.02)    mflgr (2.0.0)       mp7 (7.2.001)       vs2dt (3.3)
gsflow (2.2.0)      mflgrdbl (2.0.0)    mt3dms (5.3.0)      zbud6 (6.3.0)
libmf6.so (6.3.0)   mfnwt (1.2.0)       mt3dusgs (1.1.0)    zonbud3 (3.01)
mf2000 (1.19.01)    mfnwtdbl (1.2.0)    prms (5.2.1)        zonbudusg (1.5)
mf2005 (1.12.00)    mfusg (1.5)         sutra (3.0)
mf2005dbl (1.12.00) mfusgdbl (1.5)      swtv4 (4.00.05)

or some suggested locations:

$ python get_modflow_executables.py :select
select an extraction directory:
 1: /home/mtoews/miniforge3/envs/pyforge/bin
 2: /home/mtoews/.local/bin
> 2
fetched release '8.0'
using previous download '/home/mtoews/Downloads/linux-8.0.zip' (use --force to re-download)
extracting files to '/home/mtoews/.local/bin'
installed 26 executables:
crt (1.3.1)         mf6 (6.3.0)         mp6 (6.0.1)         triangle (1.6)
gridgen (1.0.02)    mflgr (2.0.0)       mp7 (7.2.001)       vs2dt (3.3)
gsflow (2.2.0)      mflgrdbl (2.0.0)    mt3dms (5.3.0)      zbud6 (6.3.0)
libmf6.so (6.3.0)   mfnwt (1.2.0)       mt3dusgs (1.1.0)    zonbud3 (3.01)
mf2000 (1.19.01)    mfnwtdbl (1.2.0)    prms (5.2.1)        zonbudusg (1.5)
mf2005 (1.12.00)    mfusg (1.5)         sutra (3.0)
mf2005dbl (1.12.00) mfusgdbl (1.5)      swtv4 (4.00.05)

jdhughes-usgs commented 2 years ago

@mwtoews This is similar (but probably better) to functionality that we have in pymake.

I am not sure this is the right repo for this script because it someone would have to clone the repo to get the script or download the script from the repo. I am thinking this should be a utility in flopy. If we did this the executables could be downloaded as part of a post-install step during flopy installation, installed in a specific flopy bin/lib (or maybe the standard python bin/lib) directory, and default executables with os dependent extensions could be set for a flopy installation.

I would also think it would be good if you could query the available versions of a specific executable (for example, mf2005) and install it.

There should also be a way to get mf6 executables from the nightly build.

Finally, I think the generic functionality in the script should be maintained to allow users to install specific releases, versions of executables, etc in a user defined directory not tied to flopy.

We currently use pymake to get the precompiled executables now in our repos. So if we proceed, we would need to refactor a number of our repos to switch to this functionality. We should also add some well thought out testing of this functionality to the CI pipeline. I have run into issues using the GitHub API in CI because of rate limits (there is probably a clever way around this).

@langevin-usgs @spaulins-usgs @Hofer-Julian @mjr-deltares do you agree/disagree or have any other thoughts?

mwtoews commented 2 years ago

I agree this would make more sense to exist in flopy, but it is intended to be a stand-alone script without any dependencies, like get-pip.py. The documentation, when written, would just say to download directly from raw.githubusercontent.com, rather than needing git to clone a whole repo. I would suggest it go to flopy/scripts. Flopy could still leverage it as a module too.

To query specific programs, I was considering adding (e.g.) --subset mf2005,mfnwt option to extract a subset of packages, indexed by the name in code.json and/or the executable name (e.g. --subset mf2005.exe,mfnwt.exe would work only on windows), so this could be added without too much difficulty. Adding the nightly mf6 would take a bit of extra work to determine how to integrate.

As for the generic functionally to install to specific releases in user defined directories, this is already done, e.g.

C:\> md C:\WRDAPP\bin
C:\> python get_modflow_executables.py C:\WRDAPP\bin --release-id 4.0
fetched release '4.0'
using previous download 'C:\Users\mtoews\Downloads\win64-4.0.zip' (use --force to re-download)
extracting files to 'C:\WRDAPP\bin'
installed 24 executables:
crt.exe (1.3.1)         mflgrdbl.exe (2.0.0)    mt3dusgs.exe (1.1.0)
gridgen.exe (1.0.02)    mfnwt.exe (1.2.0)       sutra.exe (3.0)
gsflow.exe (2.1.0)      mfnwtdbl.exe (1.2.0)    swtv4.exe (4.00.05)
mf2000.exe (1.19.01)    mfusg.exe (1.5)         triangle.exe (1.6)
mf2005.exe (1.12.00)    mfusgdbl.exe (1.5)      vs2dt.exe (3.3)
mf2005dbl.exe (1.12.00) mp6.exe (6.0.1)         zbud6.exe (6.1.1)
mf6.exe (6.1.1)         mp7.exe (7.2.001)       zonbud3.exe (3.01)
mflgr.exe (2.0.0)       mt3dms.exe (5.3.0)      zonbudusg.exe (1.5)

And yes, agree that CI testing for features should be present. I've done this for a few CLIs before, so it shouldn't be too much trouble with pytest.

Hofer-Julian commented 2 years ago

I agree this would make more sense to exist in flopy, but it is intended to be a stand-alone script without any dependencies, like get-pip.py. The documentation, when written, would just say to download directly from raw.githubusercontent.com, rather than needing git to clone a whole repo. I would suggest it go to flopy/scripts. Flopy could still leverage it as a module too.

Ah yes, I didn't get that this is what you were going for. It is quite untypical though for these kind of scripts to require Python. rustup for example provides a shell script for Linux and macOS and an .exe for Windows. ~If we go that route I think we could do something similar for MODFLOW 6.~ EDIT: One possibility to avoid requiring users to have Python installed would be to bundle this script with pyinstaller.

@mwtoews one application I could see for your script is within the conda-forge build. At the moment neither the macOS nor Windows build is working. We could use your script to download the executables on the conda-forge CI instead of building them on these two operating systems.

langevin-usgs commented 2 years ago

For what it's worth, I like the structure and functionality of this get_modflow_executables.py script. It seems like it would be a welcome addition by many of our flopy users. As developers, we could move to adopt it more broadly throughout our CI chain though we wouldn't have to do that right away. I also like the idea of adding flopy/scripts and putting it in there, perhaps with a markdown file with documentation. I know of at least one other miscellaneous script (flopy/mf6/utils/createpackages.py) that might also benefit from being moved to a more suitable flopy/scripts location.

mwtoews commented 2 years ago

one application I could see for your script is within the conda-forge build

Unfortunately the conda-build process requires packages to be built from source using compilers available in the conda ecosystem, which are limited and old for Fortran Windows. I'm not sure if there is a way to substitute pre-built binaries to the final package (but it could be worth asking). So as a user work-around, the the idea behind python get_modflow_executables.py :select is to detect a conda, then suggests to put the executables in that environment's PATH, e.g. /home/mtoews/miniforge3/envs/pyforge/bin

Thanks, great feedback, I'll continue with this script further to add --subset and investigate adding mf6 nightly releases too. However, changes and a PR will appear at the flopy repo.

Hofer-Julian commented 2 years ago

Unfortunately the conda-build process requires packages to be built from source using compilers available in the conda ecosystem, which are limited and old for Fortran Windows.

Too bad, thanks for checking.

So as a user work-around, the the idea behind python get_modflow_executables.py :select is to detect a conda, then suggests to put the executables in that environment's PATH, e.g. /home/mtoews/miniforge3/envs/pyforge/bin

Yes, that sounds like the best option then.

mwtoews commented 2 years ago

See https://github.com/modflowpy/flopy/pull/1465 for follow-up

Some changes since the version here:

renamed main file to shorter get_modflow.py
renamed bindir option :select to ?
enabled repo option to select mf6 nightly
subset option enabled
not moved to flopy/scripts, but moved within the flopy.utils module so it can be used internally
created console script get-modflow (or get-modflow.exe for Windows)

MODFLOW-USGS / executables

Add script to help install executables #6

Examples