Addresses the same functionality as #598 but with some differences, this PR is for allowing users to execute cells with specific languages against the same Spark Session. Given that #598 is nearly 5 years old and seems abandoned, this implementation provides a minimal, backward-compatible solution.
Due to a lack of familiarity with the sparkmagic code base, no tests have been added. I kindly request guidance on how to implement unit tests.
The changes have been manually tested on a notebook, ensuring compatibility with shared DataFrames and UDFs between Scala and Python.
Description
Addresses the same functionality as #598 but with some differences, this PR is for allowing users to execute cells with specific languages against the same Spark Session. Given that #598 is nearly 5 years old and seems abandoned, this implementation provides a minimal, backward-compatible solution.
Due to a lack of familiarity with the sparkmagic code base, no tests have been added. I kindly request guidance on how to implement unit tests.
The changes have been manually tested on a notebook, ensuring compatibility with shared DataFrames and UDFs between Scala and Python.
Checklist
black
CHANGELOG.md
fileREADME.md
file