search

EarthCubeInGeo / resen-core

The core docker image used by resen.
GNU General Public License v3.0
2 stars 3 forks source link

[MNT]: Todo for next release #67

Closed asreimer closed 3 years ago

asreimer commented 3 years ago

TODOs for next release:

pmreyes2 commented 3 years ago

Plotly Installed following: [https://plotly.com/python/getting-started/#jupyterlab-support](https://plotly.com/python/getting-started/#jupyterlab-support adds like 146 MB to the py38 envs folder after pip install. but it only works on classic notebooks. For rendering on JupyterLab a labextension is needed which adds to a total of 354 MB Additional widgets extension brings the total to 383 MB

bucket Virtual Size Approx Diff
base bucket 3.61 GB -
xarray 3.62 GB 7 MB
OMMBV 3.61 GB 1 MB
plotly + labextension 3.97 GB 354 MB
ljlamarche commented 3 years ago

I've been looking into adding various empirical model python wrappers from https://github.com/space-physics. In general, they all seem to install and run fine, but some of them require cmake, which adds about 350 MB to our container size. Here's the results of a quick test I did, just by starting up a new bucket and running docker ps -s after each new installation. I'm not super familiar with cmake - maybe there's something we can do to install a more "bare bones" version?

Size Virtual Size Approx Diff
base bucket 5.38 kB 3.6 GB -
cmake 357 MB 3.96 GB 357 MB
IGRF 357 MB 3.96 GB 0 MB
MSIS-E 00 358 MB 3.96 GB 1 MB
IRI 2016 374 MB 3.98 GB 16 MB
WMM 2020 407 MB 4.01 GB 33 MB
WMM 2015 440 MB 4.04 GB 33 MB
HWM 93* 455 MB 4.06 GB 15 MB
IRI 90* 456 MB 4.06 GB 1 MB

HWM93 and IRI90 will install but not run due to what I believe is a numpy version issue:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

If we think these are important models to include, I can look into solving that. Also, although these models install with cmake, they don't need it to run. If I uninstalled cmake and completely removed it after all these installations, the size of the container goes back down to 161 MB (3.76 GB) and I think a good chunk of that is the two WMM models (I'm not sure how common these are, maybe it's most useful to just include IGRF, MSIS-E, and IRI).

asreimer commented 3 years ago

I've upgraded resen-core to jupyterlab 3. Reading the release notes, this shouldn't break compatibility with the jupyterhub stuff on the demo server. The benefit of going to 3 is that we don't need npm and nodejs to install common extensions so the resulting image is smaller. This was done in commit https://github.com/EarthCubeInGeo/resen-core/commit/d7d1743ba88acd13a864e623342b2888120cd8c1

Edit: One consequence of moving to jupyterlab 3 is that they code to shutdown the jupyterlab server has moved. As a result, we need to change (these lines in Resen.py)[https://github.com/EarthCubeInGeo/resen/blob/master/resen/Resen.py#L574-L576] to be

    python_cmd = 'exec(\\\"try:  from jupyter_server.serverapp import shutdown_server, list_running_servers\\n'
    python_cmd += 'except:  from notebook.notebookapp import shutdown_server, list_running_servers\\n'
    python_cmd += 'svrs = [x for x in list_running_servers() if x[\\\\\\"port\\\\\\"] == %s]; ' % (port)
    python_cmd += 'sts = True if len(svrs) == 0 else shutdown_server(svrs[0]); print(sts)\\\")'

Edit2: This PR fixes this problem: https://github.com/EarthCubeInGeo/resen/pull/88

asreimer commented 3 years ago

I was able to get the empirical models igrf, iri2016, and nrlmsise-00 installed from https://github.com/space-physics. This is done in the enh_emp_models branch and is ready to be tested in this PR: https://github.com/EarthCubeInGeo/resen-core/pull/72

asreimer commented 3 years ago

All tasks completed and release PR is ready for review.