Headless container with mne pyvista backend

[christianhacker]

Describe the bug

Attempt to follow instructions for headless install of MNE-Tools. See this gist for config.

Steps to reproduce

docker build ./Dockerfile -t mne
docker run -it -e GRANT_SUDO=yes --user root --name mne -p 888:8888 -p 6006:6006 mne

Now open http://localhost:8888/tree? in a browser. Download and try running the Plotting EEG sensors on the scalp example in this jupyter server.

Expected results

Jupyter notebook properly renders the scalp electrode example by returning the widget.

Actual results

Code cell 2 of the above example does not render scalp plot. Kernel crashes, get error message: qt.qpa.xcb: could not connect to display

If I launch an Xvfb virtual display via: xvfb-run -s '-screen 0 1920x1080x24' as part of the docker entrypoint script (uncomment line 9 in docker-entrypoint.sh) and install libraries required by pyvista, the plotting cell executes without error, but nothing is returned. If I explicitly return fig in a separate cell, I get: <mne.viz.backends._pyvista._Figure at 0x7fde97989c70>

Additional information

Platform: Linux-5.4.39-linuxkit-x86_64-with-glibc2.10 Python: 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 19:08:05) [GCC 7.5.0] Executable: /opt/conda/bin/python CPU: x86_64: 24 cores Memory: Unavailable (requires "psutil" package) mne: 0.21.2 numpy: 1.19.4 {blas=NO_ATLAS_INFO, lapack=lapack} scipy: 1.5.3 matplotlib: 3.3.3 {backend=module://ipykernel.pylab.backend_inline}

sklearn: 0.23.2 numba: Not found nibabel: 3.2.1 cupy: Not found pandas: 1.1.5 dipy: Not found mayavi: Not found pyvista: 0.27.4 {pyvistaqt=0.2.0, OpenGL 3.3 (Core Profile) Mesa 18.3.1 via llvmpipe (LLVM 7.0, 256 bits)} vtk: 9.0.0 PyQt5: 5.12.3

[larsoner]

@christianhacker your server_environment.yml has pyqt in the list, can you try this newer version that does not:

https://github.com/mne-tools/mne-python/blob/master/server_environment.yml

You see PyQt5: not found in sys_info like https://dev.azure.com/mne-tools/mne-python/_build/results?buildId=10691&view=logs&j=f8b184f0-8536-5d07-cb2e-df31a09e5d0f&t=1e39b4f7-1217-5e61-db4f-987e414d3830

For headless you shouldn't need it, and if you have it but not all the libs to run it you will have problems...

[christianhacker]

I tried it both ways. I get a module import error for PyQt5 for the pyvista backend if I don't install the pyqt package.

[larsoner]

I get a module import error for PyQt5 for the pyvista backend if I don't install the pyqt package.

Can you give the error there? That way we're sticking to the stock MNE instructions as much as possible.

[christianhacker]

Here you are:

ModuleNotFoundErrorTraceback (most recent call last)
<ipython-input-3-df3b90b4b7be> in <module>
     13 raw = mne.io.read_raw_fif(data_path + '/MEG/sample/sample_audvis_raw.fif')
     14 # Plot electrode locations on scalp
---> 15 fig = plot_alignment(raw.info, trans, subject='sample', dig=False,
     16                      eeg=['original', 'projected'], meg=[],
     17                      coord_frame='head', subjects_dir=subjects_dir)

<decorator-gen-138> in plot_alignment(info, trans, subject, subjects_dir, surfaces, coord_frame, meg, eeg, fwd, dig, ecog, src, mri_fiducials, bem, seeg, fnirs, show_axes, fig, interaction, verbose)

/opt/conda/lib/python3.8/site-packages/mne/viz/_3d.py in plot_alignment(***failed resolving arguments***)
    976 
    977     # initialize figure
--> 978     renderer = _get_renderer(fig, bgcolor=(0.5, 0.5, 0.5), size=(800, 800))
    979     if interaction == 'terrain':
    980         renderer.set_interaction('terrain')

/opt/conda/lib/python3.8/site-packages/mne/viz/backends/renderer.py in _get_renderer(*args, **kwargs)
     36 def _get_renderer(*args, **kwargs):
     37     set_3d_backend(_get_3d_backend(), verbose=False)
---> 38     return backend._Renderer(*args, **kwargs)
     39 
     40 

/opt/conda/lib/python3.8/site-packages/mne/viz/backends/_pyvista.py in __init__(self, fig, size, bgcolor, name, show, shape, notebook, smooth_shading)
    196                 self.figure.smooth_shading = False
    197             with _disabled_depth_peeling():
--> 198                 self.plotter = self.figure.build()
    199             self.plotter.hide_axes()
    200             if hasattr(self.plotter, "default_camera_tool_bar"):

/opt/conda/lib/python3.8/site-packages/mne/viz/backends/_pyvista.py in build(self)
     87         if self.plotter is None:
     88             if self.plotter_class is BackgroundPlotter:
---> 89                 from PyQt5.QtWidgets import QApplication
     90                 app = QApplication.instance()
     91                 if app is None:

ModuleNotFoundError: No module named 'PyQt5'

[larsoner]

@GuillaumeFavelier can you figure out how we end up in this PyQt5-dependent code path even when on a headless server? It seems like this should end up using the notebook backend somehow but it does not

[GuillaumeFavelier]

I don't think the notebook 3d backend is set automatically. The following is required:

import mne
mne.viz.set_3d_backend('notebook')

[christianhacker]

That works, per our conversation on gitter. I suppose my question is whether the pyvista backend, not the notebook backend, works for a notebook in a headless server. I know that pyvista works just fine (confirmed with a simple sphere example), but mne with the pyvista backend does not appear to work in this specific situation.

(Edit) I realize this probably isn't a priority right now; mayavi covers most of my use cases anyway. It is interesting however that pyvista appears to function correctly in jupyter notebook, and jupyterlab to boot, but mne with the pyvista backend does not.

[larsoner]

I suppose my question is whether the pyvista backend, not the notebook backend, works for a notebook in a headless server.

I think this is where the disconnect is -- the set_3d_backend('notebook') in MNE is using PyVista (plus ipywidgets, etc.) under the hood, just with a different interface. It's analogous I think to using mayavi + mlab.init_notebook(). @GuillaumeFavelier feel free to correct me if I'm wrong...

When you don't do set_3d_backend('notebook'), it uses PyVista + PyVistaQt rather than PyVista + ipywidgets (etc.) to display the PyVista renderings and interface. Does that make sense?

[christianhacker]

Yes, I understood that the notebook backend does use pyvista under the hood. The mne documentation suggests as much. However, I would like to be able to have an interface controlled solely by mouse clicks, not ipywidget sliders, as is the case with mayavi (virtual display) or simple pyvista examples unrelated to mne. This is sounding more and more like a feature request than a bug. I made an assumption about mne.viz functionality that appears to not exist, and it would take extra work to implement that functionality. Something compatible with itkwidgets, perhaps.

I may have misunderstood the meaning of the mne headless install instructions. It appears that the instructions for mayavi and pyvista do not result in a functionally analogous environment; the former does not require sliders for interactive control, while the latter does. Is that correct?

[GuillaumeFavelier]

It appears that the instructions for mayavi and pyvista do not result in a functionally analogous environment; the former does not require sliders for interactive control, while the latter does. Is that correct?

Correct and although it might be possible to disable those sliders for the notebook 3d backend, interactions would have to be done programmatically.

I would like to be able to have an interface controlled solely by mouse clicks, not ipywidget sliders

We tried multiple backends (associated with pyvista) to support this feature (panel, itkwidgets and ipywidgets). The latest and most interesting one is probably ipyvtk. I'm currently working on a prototype, feel free to try it out or comment there if this is something you could be interested in. Please, do keep in mind this is a still a work in progress and while the preliminary results are promising, I have to ensure that the entire feature set can be migrated. A full feasibility study is necessary.

[christianhacker]

Understood, closing. Thank you for taking time to address this; my lab is tied to brainstorm currently and I'm looking to redo as much of our pipeline as possible in MNE, eventually. Being able to share a completely self-contained development environment with colleagues, without having to deal with Matlab licensing or requiring a physical monitor, would be great for reproducibility and consistency. I'll give the prototype a shot - thanks @GuillaumeFavelier, @larsoner.