Encoding error using spark magic

[devender-yadav]

Using:

• Livy 0.5.0
• Spark 2.4.0
• EMR 5.21.0
• Python 3.6.8

Sparkmagic karnel - pyspark

export PYSPARK_DRIVER_PYTHON=python3
export PYSPARK_PYTHON=python3

I want to create both spark and pandas dataframe. Sample code using spark magic:

%%spark -o df
df = spark.range(1 << 2).toDF("id")

Not able to create pandas dataframe.

Error: Cannot parse object as JSON: '['b\'{"id":0}\'', 'b\'{"id":1}\'', 'b\'{"id":2}\'', 'b\'{"id":3}\'', '----------------------------------------', "Exception happened during processing of request from ('127.0.0.1', 43754)", 'Traceback (most recent call last):', ' File "/usr/lib64/python3.6/socketserver.py", line 320, in _handle_request_noblock', ' self.process_request(request, client_address)', ' File "/usr/lib64/python3.6/socketserver.py", line 351, in process_request', ' self.finish_request(request, client_address)', ' File "/usr/lib64/python3.6/socketserver.py", line 364, in finish_request', ' self.RequestHandlerClass(request, client_address, self)', ' File "/usr/lib64/python3.6/socketserver.py", line 724, in init', ' self.handle()', ' File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 266, in handle', ' poll(authenticate_and_accum_updates)', ' File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 241, in poll', ' if func():', ' File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 254, in authenticate_and_accum_updates', ' received_token = self.rfile.read(len(auth_token))', "TypeError: object of type 'NoneType' has no len()", '----------------------------------------']'

What am I missing?

[devender-yadav]

After looking at https://livy.incubator.apache.org/docs/latest/rest-api.html

I tried to configure a few params

%%configure -f
{"executorMemory": "1000M", "executorCores": 4, "conf": 
{"spark.yarn.appMasterEnv.PYSPARK_PYTHON":"/usr/bin/python3.6", 
"PYSPARK_PYTHON":"/usr/bin/python3.6"}}

Still same error.

[itamarst]

This is now fixed in the master branch. I will try to do a release soon.

[itamarst]

I have released 0.12.8 which will hopefully fix this.