search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.33k stars 447 forks source link

sparkmagic output html table #512

Closed Tagar closed 1 year ago

Tagar commented 5 years ago

image

Is there is a way to actually display an html output, like it works without %%spark ?

Tagar commented 4 years ago

@itamarst please have a look if there is a way to fix this ?

thanks!

winston-zillow commented 1 year ago

Same in pyspark kernel, HTML outputs are not displayed but the HTML codes printed. Also the pandas DataFrames (note: not the pyspark DataFrame) are not being rendered prettified, even using display() or display_html().

Whereas running regular kernel (even in the same instance/python env) will render correctly.

Screenshot 2023-03-28 at 5 33 54 PM Screenshot 2023-03-28 at 5 33 38 PM
winston-zillow commented 1 year ago

@devstein please take a look. This seems an annoying problem.

devstein commented 1 year ago

@winston-zillow Thanks for the example. As demonstrated in first example. Sparkmagic naively renders the response from Livy, which is the { 'text/html': '<h1>H1</h1>' } JSON. This should be fixable. Are you open to contributing?

The second example is much tricky. Your local notebook has access to the raw pandas dataframe while Sparkmagic has to parse the dataframe values from the JSON string returned from Livy. This is what the %%pretty magic attempts to do.