search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.33k stars 447 forks source link

add support for cell-level sparkmagic language type #509

Open Tagar opened 5 years ago

Tagar commented 5 years ago

FYI - Livy 0.5+ has session kind which is actually useless

See this comment in LIVY-469

Starting from 0.5 session kind is actually useless, we should encourage user to not specify session kind, instead we should set code kind when submitting statement.

So this could be both a notebook-level property (just as a default for newly created cells), and a cell-level property. For example, this opens up a way to for something like this in sparkmagic:

%%spark -l python
# .. some pyspark code.. 
df = spark.sql("select * from universe")
%%spark -l scala 
// .. some Scala code.. 
var df = spark.sql("select * from universe")

on the same (shared) backend Spark session.

Tagar commented 5 years ago

Spin-off from https://github.com/jupyter-incubator/sparkmagic/pull/507