Open ying1 opened 6 years ago
@aggFTW (Not sure who's maintaining this repo)
I think the way to do this would be similar to Zeppelin.
E.g. %%spark.python for a python statement, and %%spark.scala for a scala statement. (Allowing users to set a default language which can be called as %%spark)
This would also require us change the %%sql implementation, but would stop us having to spin up a new session for sql queries.
Livy 0.5 has a new feature that enables multiple interpreter support in one session: https://github.com/apache/incubator-livy/commit/c1aafeb6cb87f2bd7f4cb7cf538822b59fb34a9c
Documentation: https://livy.incubator.apache.org/docs/latest/rest-api.html#session-kind
This seems to cause an issue with spark magic in which if the livy server has a shared session running, the "Add Endpoint" would fail with :
Which traces down to: https://github.com/jupyter-incubator/sparkmagic/blob/master/sparkmagic/sparkmagic/utils/constants.py