Install these packages for GEL298

[moorepants]

They are trying to run this tutorial:

https://shtools.github.io/SHTOOLS/pages/mydoc/notebooks/Introduction-4.html

• [x] SHtools, this isn't packaged for conda, see https://github.com/conda-forge/staged-recipes/pull/9121, but might be pip installable, but we need to install the dependencies via conda
  • [x] astropy
  • [x] xarray
  • [x] requests
  • [x] numpy
  • [x] scipy
  • [x] matplotlib
• [x] cartopy, is available on conda forge
• pygmt's dependencies, note that pygmt is only installable from source via github, which we will not do
  • [x] xarray, on conda forge
  • [x] packaging, on conda forge
  • [x] gmt (seems pygmt wants this to be forced to be 6.0.0, but don't do that cause it could break other packages), on conda forge
  • [x] palettable, on conda forge
  • [x] netcdf4 on conda forge

[celine168]

I'm installing palettable, cartopy, and gmt right now. xarray and packaging seem to already be installed even though they're not explicitly stated in the Dockerfile, but I'll add them explicitly. Should we hold off from installing SHTools until further notice with the conda recipe?

[moorepants]

If you want to try a pip install pyshtools after all the dependencies are install via conda you can.

[moorepants]

but I'll add them explicitly

Good idea.

[moorepants]

FYI, I'm trying this: https://github.com/conda-forge/staged-recipes/pull/11298 but unlikely to be a solution that is timely.

[moorepants]

shtools seems like a non-trivial build, so I'm not sure this will work out.

[moorepants]

Probably not going to be worth trying to install shtools. I'm getting a host of fortran build errors here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=144282&view=logs&j=10bda42a-d420-5c28-a43f-bc4e66405e5c&t=bef775d0-a1a2-55a3-9721-17147029baf0

This isn't something we should be spending time on. Install everything from conda forge that you can in the list. If you want to try to install pyshtools via pip after that, maybe the binary installation will work. But don't spend any more time than that on pyshtools.

We'll have to leave them with just some of the depedencies and they have to figure out the rest.

[moorepants]

FYI, this is a work around for the class that works (and should work on the dockerfile too):

Open terminal:

conda install -y -c conda-forge gmt palettable cartopy xarray
git clone https://github.com/GenericMappingTools/pygmt.git
cd pygmt
python setup.py install
pip install pyshtools

Then open a notebook and this works:

import pyshtools as pysh
import matplotlib.pyplot as plt

from cartopy import crs as ccrs
import pygmt
from palettable import scientific as scm

So I think it is ok to pip install the pyshtools for now. Maybe use pip install --no-dep pyshtools so that it doesn't install over any conda packages.

[celine168]

The image is live on production. However, after users clone and install pygmt, they need to restart their servers in order for pyshtools to take effect properly (it seems to rely on pygmt somewhat). However, #148 doesn't seem to allow them to restart their servers themselves, so our options are:

• have them email us to manually stop their server until #148 is solved
• find commands to restart pyshtools themselves
  • reinstalling via pip / conda leads to a user/permissions error
  • restarting kernels doesn't seem to help either

[moorepants]

Ok, thanks for getting this setup.

We aren't going to restart servers for people, that's too much work on our end.

pygmt does not seem to be a dependency of shtools (inspected shtools setup.py). So I'm not sure what's up with that.

I'll have a look this afternoon to see if I can figure out the interaction that is causing this.

[moorepants]

This is a bit vague about pygmt: https://github.com/SHTOOLS/SHTOOLS/blob/8908e117c30665d070087f4e6810809e6e54a268/docs/pages/mydoc/installing-python.md#python-dependencies

Seems like it might be a dependency.

[moorepants]

I noticed that gmt 5.4.5 is installed instead of 6.0.0.

jovyan@jupyter-jasonmoore-40ucdavis-2eedu:~$ conda list | grep gmt
dcw-gmt                   1.1.4                      1001    conda-forge
gmt                       5.4.5                hbdec4b4_6    conda-forge
gshhg-gmt                 2.3.7                      1002    conda-forge

That could cause issues with pygmt I suppose. Wonder why it wouldn't select 6.0.

pyshtools imports fine without pygmt present:

jovyan@jupyter-jasonmoore-40ucdavis-2eedu:~$ conda list | grep pypi
biopandas                 0.2.5                    pypi_0    pypi
ipymol                    0.5                      pypi_0    pypi
pypdb                     1.300                    pypi_0    pypi
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtwebengine             5.12.1                   pypi_0    pypi
pyshtools                 4.6.2                    pypi_0    pypi
xmltodict                 0.12.0                   pypi_0    pypi
jovyan@jupyter-jasonmoore-40ucdavis-2eedu:~$ python -c "import pyshtools"

[moorepants]

Also looks like netcdf4 is a dependency of pygmt:

Installed /opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg
Processing dependencies for pygmt==0.0.1a0+80.g007a53a
Searching for netCDF4
Reading https://pypi.org/simple/netCDF4/
Downloading https://files.pythonhosted.org/packages/35/4f/d49fe0c65dea4d2ebfdc602d3e3d2a45a172255c151f4497c43f6d94a5f6/netCDF4-1.5.3-cp36-cp36m-manylinux1_x86_64.whl#sha256=c9ff82a07db6fb97800b329fa84f85031f403e98127c919eea95881a495b4281
Best match: netCDF4 1.5.3
Processing netCDF4-1.5.3-cp36-cp36m-manylinux1_x86_64.whl
Installing netCDF4-1.5.3-cp36-cp36m-manylinux1_x86_64.whl to /opt/conda/lib/python3.6/site-packages
Adding netCDF4 1.5.3 to easy-install.pth file
Installing nc3tonc4 script to /opt/conda/bin
Installing nc4tonc3 script to /opt/conda/bin
Installing ncinfo script to /opt/conda/bin

Installed /opt/conda/lib/python3.6/site-packages/netCDF4-1.5.3-py3.6-linux-x86_64.egg
Searching for cftime
Reading https://pypi.org/simple/cftime/
Downloading https://files.pythonhosted.org/packages/a0/ff/38302691a715e59084e71464ffc4288d0513f51b4baca4f6c1b5bd865ede/cftime-1.1.1.2-cp36-cp36m-manylinux1_x86_64.whl#sha256=9d49569ed1c2e8fc3a4c0f95bdbe14a2e146a794ed7a6970dcb54fea455acb88
Best match: cftime 1.1.1.2
Processing cftime-1.1.1.2-cp36-cp36m-manylinux1_x86_64.whl
Installing cftime-1.1.1.2-cp36-cp36m-manylinux1_x86_64.whl to /opt/conda/lib/python3.6/site-packages
Adding cftime 1.1.1.2 to easy-install.pth file

We should install that via conda: https://github.com/conda-forge/netcdf4-feedstock

[moorepants]

This is the error I get after installing pygmt master with python setup.py install and trying to import it:

jovyan@jupyter-jasonmoore-40ucdavis-2eedu:~$ python -c "import pygmt"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/__init__.py", line 29, in <module>
    _begin()
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/session_management.py", line 16, in begin
    with Session() as lib:
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/clib/session.py", line 177, in __enter__
    self.create("pygmt-session")
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/clib/session.py", line 328, in create
    restype=ctp.c_void_p,
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/clib/session.py", line 276, in get_libgmt_func
    self._libgmt = load_libgmt()
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/clib/loading.py", line 43, in load_libgmt
    check_libgmt(libgmt)
  File "/opt/conda/lib/python3.6/site-packages/pygmt-0.0.1a0+80.g007a53a-py3.6.egg/pygmt/clib/loading.py", line 153, in check_libgmt
    raise GMTCLibError(msg)
pygmt.exceptions.GMTCLibError: Error loading libgmt. Couldn't access function GMT_Get_Enum.

[moorepants]

This is because GMT is 5.4.5 instead of 6.0.0: https://github.com/GenericMappingTools/pygmt/issues/246

So, we have to try to forcw gmt with gmt >=6 would be the best option.

[moorepants]

I'm worried now that we are going to make it impossible for them to run the code due to the gmt 5.4.5 installed. I've been trying to manually update it and am running into issues. I'll keep trying, but have to go teach for a while.

[moorepants]

I managed to break my conda installation, trying to get this to work. What a pain. This might be a disaster.

@celine168 Any way you can update gmt to gmt >=6 and see if that builds for staging? If it does, I think that's going to be needed to have any hope of the students getting this installed.

[celine168]

Do you need to reset your environment? I can try deleting your pod. I'll try to update it this afternoon, may take some time since I have lectures. I apologize, I didn't notice the gmt version dependency before. After installing pygmt and trying to run the cartopy example, my error occurred when importing pyshtools, so I assumed that that was the problem.

[moorepants]

I told you not to make a strict pin to gmt. Its my fault. I figured it would install the latest.

I can get things figure out this afternoon. I probably can't touch it till after 4 or more.

[celine168]

It's quite strange because pygmt/the cartopy example seems to work fine on gmt=5.4.5 (as long as I delete my pod and respawn it afterwards). Conda is returning an error when I try to pin gmt to 6.0.0 in the Dockerfile...

[moorepants]

Are you doing gmt >= 6 or gmt ==6? Should be the former.

I was trying to figure out what else in our environment depends on gmt.

It's quite strange because pygmt/the cartopy example seems to work fine on gmt=5.4.5

Yes, strange, as the pygmt docs say that it strictly only works with 6.0+, but maybe there are some features that work with < 6.0.

[moorepants]

I guess since 6.0.0 is the latest conda release, gmt >=6 and gmt ==6 will give the same error. One of our other packages must force gmt <6.

[moorepants]

This is one of those cases that it would be nice if the users could make a custom conda environment that is persistent, but I haven't had luck with the instructions to do so yet.

[celine168]

The Default Environment on JupyterHub (in production) now has the updated image. I believe everything should work now! I tested out cloning the pygmt repo through a fresh account that never installed it before, and then ran the cells successfully in one go. The other kernels seem to work just fine for me. Files used here: https://github.com/LibreTexts/default-env/pull/35

[moorepants]

Did you add netcdf4?

[moorepants]

Its working on both my accounts but I'm not sure if I'm in the state I left them in or if the containers have been reset.

This sounds good though. If you tested and it works, then I think we are good to go.

[celine168]

Shoot, I did not add netcdf4. I'll try adding it in.

[celine168]

I tested the image out by going into the Admin Tab and spawning Luigi's server.

[celine168]

Installed netcdf4 based in this tag: https://github.com/LibreTexts/default-env/tree/1.13 default-env:1.13 is now live on JupyterHub. I haven't encountered any issues and since cartopy seems to work, I'll close.

[moorepants]

Thanks Celine. Great work on this!

[moorepants]

I've got a pygmt conda package recipe going here: https://github.com/conda-forge/staged-recipes/pull/11468 should be merged soon.