jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.31k stars 441 forks source link

Run cells with specific language #892

Open shay1bz opened 4 months ago

shay1bz commented 4 months ago

Description

Addresses the same functionality as #598 but with some differences, this PR is for allowing users to execute cells with specific languages against the same Spark Session. Given that #598 is nearly 5 years old and seems abandoned, this implementation provides a minimal, backward-compatible solution.

Due to a lack of familiarity with the sparkmagic code base, no tests have been added. I kindly request guidance on how to implement unit tests.

The changes have been manually tested on a notebook, ensuring compatibility with shared DataFrames and UDFs between Scala and Python.

Checklist

shay1bz commented 4 months ago

@devstein would you mind taking a look at this?