search

acroz / pylivy

A Python client for Apache Livy, enabling use of remote Apache Spark clusters.
MIT License
70 stars 32 forks source link

Livy Example for Python #59

Closed danielhanbitlee closed 5 years ago

danielhanbitlee commented 5 years ago

Hi,

I am having a hard time getting livy running on my local machine. I have Windows 10, Python 3.6, and pyspark 2.4.1, and livy 0.5.0. I have a csv file that I want to read in with pyspark. Can someone show me simple steps of using livy to do this? I'm trying to see if livy is a good option to be able to run spark code from web applications.

acroz commented 5 years ago

Hi Daniel,

I've unfortunately not got any experience with setting up Spark or Livy on Windows - I've mainly used with it through vendor-configured distributions like Amazon EMR or Cloudera. For support with setup, I suggest asking on the Livy users mailing list (see https://livy.apache.org/community/).

If you're looking for an example on building a web application interfacing with Spark through Livy, I've given a talk on this that you can watch here (the app stuff is at the end):

https://www.youtube.com/watch?v=iHg0NwEjbxk

However, if you're writing a web application in Javascript (and not a Python tool like Plotly Dash in my talk), you might want to cut out the middle man and call the Livy HTTP API directly from Javascript.

danielhanbitlee commented 5 years ago

Sounds good. Thanks for the helpful response.