Package review - Githubissues

edschofield commented 2 years ago

Here are the new Python packages and updates we need as of ~2022-10-27~ 2022-11-03:

preferably, more recent versions of numpy and scipy
nbconvert: this has been broken since the last update (~6 months?). Before each course the trainers run pip install -U nbconvert, which fixes the issue.
torch: we have 1.11.0. An upgrade to the most recent stable version (1.13.0, and compatible versions of its friends torchvision and torchaudio) would be good for the upcoming deep learning course.
kaleido: it would be great if this were installed by default so we don't need to restart the kernel after importing plotly in order to demonstrate static image export.
streamlit: this has been moving fast. The latest stable version is 1.13.0 but 1.14.0 might be out soon.
python itself: our particular patch version (3.9.7) has a bug that is incompatible with recent Streamlit versions. For the last course I asked participants to run a script to patch a file in Python (typing.py in /opt/conda/lib/python3.9/). (I used the most recent stable Python 3.9.x version for that and it seemed to work.) Python 3.10.x has some useful features and would be a worthy upgrade. But if there are compatibility issues, we could stay on Python 3.9.x for now.
update to plotly: 5.6.0 to 5.10.0
updates to bokeh, holoviews, panel, ipywidgets, scikit-learn
loguru
hvplot
geoplot
mypy
rich
update eli5 to latest version (current version is incompatible with installed Jinja2)
updates to numba, dask, distributed

These additional OS packages would be great too:

fd-find: https://packages.ubuntu.com/focal/fd-find
htop

On the GPU cluster, also these:

skorch
pytorch-lightning
autopytorch (if it installs cleanly)
tensorboard

robertlayton commented 2 years ago

Tensorflow would be worth getting as well - we still have it as a module, and questions about it come up from time to time. If it conflicts with anything, just drop it, but otherwise i think its worth adding

I think the tensorflow package by itself will be sufficient at this stage.

edschofield commented 1 year ago

Thanks for the updates, @maegul ! The new GPU image is working beautifully

Here's a record of what appears to be installed & working (according to my testing) and what's not:

Installed and working:

[x] Dash and jupyter-dash: this is epic! More details below.
[x] Streamlit
[x] numpy and scipy
[x] python 3.9.13
[x] vim, htop, fd-find
[x] rich
[x] loguru
[x] torch, including CUDA device use
[x] pytorch-lightning
[x] skorch
[x] nbconvert
[x] mypy, black
[x] tensorboard
[x] autopytorch
[x] voila: working beautifully

Missing or broken:

bokeh

This isn't installed by default. After a user runs pip install bokeh, it doesn't display plots in-line in JupyterLab notebooks by default. This might be fixed by including the command conda install bokeh jupyter_bokeh in the Docker build script.

However, as one workaround, plots show inline after a user pip install bokeh if you run

from bokeh.io import output_notebook
output_notebook()

to initialize Bokeh before plotting.

(As another, lamer workaround, it is possible to export Bokeh plots as HTML and view them in a separate browser tab.)

geoplot

This seems to work fine after a user runs pip install geoplot -- but the docs recommend conda install geoplot as being more likely to work.

bqplot

bqplot doesn't work after a local pip install bqplot. JS errors:

Clicking reveals:

Notes on Dash

On gpuhub.pythoncharmers.com with the latest image, Dash works within JupyterLab if you replace these lines with a simple Dash app:

app = dash.Dash(__name__)
...
app.run_server(debug=True)

with this instead:

from jupyter_dash import JupyterDash

JupyterDash.infer_jupyter_proxy_config()
app = JupyterDash(__name__)
...
app.run_server(mode='jupyterlab', port=8050)