search

acroz / pylivy

A Python client for Apache Livy, enabling use of remote Apache Spark clusters.
MIT License
70 stars 32 forks source link

Connecting to a Livy Cluster #105

Closed dibsonmuad closed 2 years ago

dibsonmuad commented 2 years ago

Hi, I was trying to connect to an API endpoint with an X-AUTH-TOKEN. I received a requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://wellness.qubole.com/livy-spark-<cluster_id>/version. I configured my LIVY_URL and AUTH_TOKEN as the doc suggested and did this:

with LivySession.create(LIVY_URL, requests_session=requests_session) as session:
  session.run("1+1")

A configuration works for local Jupyter notebooks with another package called sparkmagic. I was wondering if you had any thoughts on what could be different causing the above exception to occur. The jupyter notebook/livy integration is documented here: https://github.com/jupyter-incubator/sparkmagic. Thanks for your time!

dibsonmuad commented 2 years ago

User error. I had not updated the requests_session with the correct auth token. Apologies.

acroz commented 2 years ago

Glad you got sorted @dibsonmuad !