New conda library requests

[Zeitsperre]

We recently received an email from one of our collaborators wondering if we could add the following to our PAVCIS user images:

nested_dict (https://anaconda.org/conda-forge/nested_dict) paramiko (https://anaconda.org/conda-forge/paramiko) python-scp (https://anaconda.org/conda-forge/scp)

I feel like Paramiko and SCP would be very useful additions (streamlines file transfer via Python API - great for moving processed data off of images). Nested_dict is a nice implementation of default_dict with nesting (not specifically useful for PAVICS).

Thoughts @tlvu ?

[tlvu]

@Zeitsperre no objections. Assigned this to myself so I remember to add those next time I rebuild the Jupyter env.

[tlvu]

Note to self: try also to add the following packages that allow png export from a bokeh plot. Note these might bloat up a lot the image so it's a go only if not too big.

  - selenium
  - firefox
  - geckodriver

Demo code for testing export to png (export to html already works without these 3 extra packages)

import panel as pn
import numpy as np
import xarray as xr
import hvplot
import hvplot.xarray
import panel as pn
import holoviews as hv
pn.extension()
xy = xr.merge((xr.DataArray(np.random.rand(100,1), name='x'),xr.DataArray(np.random.rand(100,1), name='y')))
test = pn.Column(pn.pane.Markdown("## CUSTOM TITLE"),xy.hvplot.scatter(x='x',y='y'))
display(test)
test.save('test.html')
test.save('test.png')

FYI @tlogan2000

[tlogan2000]

semi related question : I just built a new local conda env using https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/master/docker/environment.yml and clisops is not available? @Zeitsperre Is this because it has been removed from RavenPY? We would need to include explicitly in the environment.yml in this case I guess

[Zeitsperre]

Yes, that's correct. Clisops and RavenPy don't mix all that well at the moment. If you absolutely need it, I would pin it at the version before xesmf was integrated for the time being. The ROOCS team is looking into the segfault issues now.

[tlvu]

Note to self: try also to add the following packages that allow png export from a bokeh plot. Note these might bloat up a lot the image so it's a go only if not too big.

  - selenium
  - firefox
  - geckodriver

Demo code for testing export to png (export to html already works without these 3 extra packages)

import panel as pn
import numpy as np
import xarray as xr
import hvplot
import hvplot.xarray
import panel as pn
import holoviews as hv
pn.extension()
xy = xr.merge((xr.DataArray(np.random.rand(100,1), name='x'),xr.DataArray(np.random.rand(100,1), name='y')))
test = pn.Column(pn.pane.Markdown("## CUSTOM TITLE"),xy.hvplot.scatter(x='x',y='y'))
display(test)
test.save('test.html')
test.save('test.png')

@tlogan2000 Just installing firefox from conda do not work I think. I find the installation surprising small since it does not pull any X libraries. But firefox is basically a desktop apps, most likely it will need X libraries, hence it did not work even if installed in the container since there is no X libraries in the container.

test.save('test.png')

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_7610/981553047.py in <module>
----> 1 test.save('test.png')

/opt/conda/envs/birdy/lib/python3.7/site-packages/panel/viewable.py in save(self, filename, title, resources, template, template_variables, embed, max_states, max_opts, embed_json, json_prefix, save_path, load_path, progress, embed_states, **kwargs)
    767                     template_variables, embed, max_states, max_opts,
    768                     embed_json, json_prefix, save_path, load_path,
--> 769                     progress, embed_states, **kwargs)
    770 
    771     def server_doc(self, doc=None, title=None, location=True):

/opt/conda/envs/birdy/lib/python3.7/site-packages/panel/io/save.py in save(panel, filename, title, resources, template, template_variables, embed, max_states, max_opts, embed_json, json_prefix, save_path, load_path, progress, embed_states, **kwargs)
    236         return save_png(
    237             model, resources=resources, filename=filename, template=template,
--> 238             template_variables=template_variables, **kwargs
    239         )
    240     elif isinstance(filename, string_types) and not filename.endswith('.html'):

/opt/conda/envs/birdy/lib/python3.7/site-packages/panel/io/save.py in save_png(model, filename, resources, template, template_variables, timeout)
     63     from bokeh.io.webdriver import webdriver_control
     64     if not state.webdriver:
---> 65         state.webdriver = webdriver_control.create()
     66 
     67     webdriver = state.webdriver

/opt/conda/envs/birdy/lib/python3.7/site-packages/bokeh/io/webdriver.py in create(self, kind)
    151 
    152     def create(self, kind: Optional[DriverKind] = None) -> WebDriver:
--> 153         driver = self._create(kind)
    154         self._drivers.add(driver)
    155         return driver

/opt/conda/envs/birdy/lib/python3.7/site-packages/bokeh/io/webdriver.py in _create(self, kind)
    169                 return driver
    170 
--> 171             raise RuntimeError("Neither firefox and geckodriver nor a variant of chromium browser and " \
    172                                "chromedriver are available on system PATH. You can install the former " \
    173                                "with 'conda install -c conda-forge firefox geckodriver'.")

RuntimeError: Neither firefox and geckodriver nor a variant of chromium browser and chromedriver are available on system PATH. You can install the former with 'conda install -c conda-forge firefox geckodriver'.

from shutil import which
which("firefox")

'/opt/conda/envs/birdy/bin/firefox'

[tlvu]

@tlogan2000 as suspected, firefox installed by conda was missing all the X-libraries and that's why it did not work.

! firefox --version

XPCOMGlueLoad error for file /opt/conda/envs/birdy/bin/FirefoxApp/libmozgtk.so:
libgtk-3.so.0: cannot open shared object file: No such file or directory
Couldn't load XPCOM.

Installing firefox using apt-get will pull all the missing X-libraries but will bloat the docker image by roughly 750MB !

So the question is ... will our users make a lot of use of the png export feature to make it worth it?

[tlvu]

@tlogan2000 finally got firefox to work properly inside the docker container. Medus has it, check it out.