Widgets not displaying when using spark magic between external cluster and jupyterhub running in a docker container.

[tegid77]

Describe the bug In AWS EMR cluster I am using sparkmagic to run a spark job on the cluster that I want to display as a bqplot widget in jupyterhub that is running in Docker container. Instead of displaying as a widget it shows the definition of the widget in code. I have verified that node.js is installed on both the container and the cluster and that the environment variables are set correctly, But I haven't been able to get the results to display properly

To Reproduce Create an EMR-5.28.0 cluster with Hadoop, Spark, JupyterHub, Hive, Hue, Zookeeper. In your bootstrap install bqplot==0.11.6, numpy==1.16.4, ipywidgets==7.5.0, jupyter-contrib-nbextensions==0.5.1, jupyter-nbextensions-configurator==0.4.1, widgetsnbextension==3.5.0, nodejs-v12.14.0

Then run jupyter nbextension enable --system --py widgetsnbextension jupyter nbextension enable --system --py bqplot

In a step or in the docker image on the cluster install the same packages as above and run the enables.

Create a pyspark3 notebook in jupyterhub and run this code:

from bqplot import pyplot as plt
import numpy as np

plt.figure(1, title='line chart')
np.random.seed(120)
n = 200
x = np.linspace(0.0, 10.0, n)
y = np.cumsum(np.random.randn(n))
plt.plot(x, y)
plt.show()

Instead of a graph like the one in the Screenshots section you will get a definition of your spark job and output like:

VBox(children=(Figure(fig_margin={'top': 60, 'bottom': 60, 'left': 60, 'right': 60}, 
layout=Layout(min_width='125px'), scale_x=LinearScale(allow_padding=False, max=1.0, min=0.0), 
scale_y=LinearScale(allow_padding=False, max=1.0, min=0.0), title='line chart'), 
Toolbar(figure=Figure(fig_margin={'top': 60, 'bottom': 60, 'left': 60, 'right': 60}, 
layout=Layout(min_width='125px'), scale_x=LinearScale(allow_padding=False, max=1.0, min=0.0), 
scale_y=LinearScale(allow_padding=False, max=1.0, min=0.0), title='line chart'))))

Expected behavior A graph looking like the one in the attached Screenshot

Screenshots

Versions:

• SparkMagic 0.12.7
• Livy (if you know it) 0.6.0
• Spark 2.4.4
  • Python 3.6.8

Additional context This is an extremely dumbed down version of what I am trying to do but this demonstrates my issue we are also wanting to use qgrid widgets as well.

[pancodia]

Try add %matplot plt at the end of the code block. I use it to show the plot in a notebook with pyspark kernel.