search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.33k stars 446 forks source link

sparkmagic displays pandas DataFrames as texts #489

Open sergun opened 5 years ago

sergun commented 5 years ago

Hi!

A have an issue with displaying pandas DataFrame in sparkmagic. They are always displayed as text.

Any ideas?

sparkmagic 0.12.6 livy 0.5

bakousylla commented 5 years ago 
      Hi!

A have an issue with displaying pandas DataFrame in sparkmagic. They are always displayed as text. Any ideas? sparkmagic 0.12.6 livy 0.5

Hi! I have exactly the same issue with plotly, did you fix that or have any ideas ? sparkmagic 0.12.6

Bootcampanalytics commented 5 years ago

Hi, same here, and I'd love a fix!!

itamarst commented 5 years ago

Have you tried the latest version, 0.12.9?

Bootcampanalytics commented 5 years ago

Have you tried the latest version, 0.12.9?

Yes, this is the version that AWS uses today.

I posted this here: https://stackoverflow.com/questions/57794930/pandas-formatting-in-emr-notebooks-in-cluster-mode-or-transferring-to-local-with

devender-yadav commented 5 years ago

@analytics-bootcamp I doubt it if it's EMR. EMR notebook is using 0.12.7 as of EMR 5.26 version.

Bootcampanalytics commented 5 years ago

@analytics-bootcamp I doubt it if it's EMR. EMR notebook is using 0.12.7 as of EMR 5.26 version.

This is what I did:

In AWS, I start a cluster and open an EMR notebook.

import sparkmagic as sm. -> error.

next try: %local import sparkmagic as sm print (sm.__version__)

-> 0.12.9

devender-yadav commented 5 years ago

Looks like they updated it in last 1-2 days. I noticed progress bar in EMR notebook today.

itamarst commented 5 years ago

Can you give me an easy way to reproduce this? Thanks!

jwillis0720 commented 3 years ago

@itamarst 

spark_df = spark.read.parquet("s3://path_to_parquet")
pandas_df = spark_df.limit(100).to_pandas()
pandas_df
> shows text instead of html markup in emr jupyter lab or jupyter notebook