search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.33k stars 447 forks source link

Document extending SparkMagic #806

Closed bradleyhurley closed 1 year ago

bradleyhurley commented 1 year ago

Is your feature request related to a problem? Please describe. Im looking for documentation or guidance on how to extend the provided SparkMagic magics.

Describe the solution you'd like It would be helpful to provide some documentation or examples of how to extend the list of provided magics in a way that they can interact with an existing livy session or create one.

Additional context The sql magic collides with other magics in our ecosystem and we would like to create a custom version that behave like sql but modified to meet our needs.

We have proved out this change can be done by modifying the kernelmagic.py once deployed to JupyterHub, but would ideally like a solution that would enable us to extend the KernelMagics class.

devstein commented 1 year ago

Hey @bradleyhurley thanks for making an issue! Unfortunately, I don't have guidance on how to natively extend Sparkmagic's magics. The most common approach is to fork Sparkmagic and change %%sql to meet your needs.

This is definitely a pain point and will keep in mind for future refactors. If you have any ideas, please let me know!

bradleyhurley commented 1 year ago

Hi @devstein - Thanks for the response. I don't think I have any good ideas, but Im not super familiar with the codebase.

We explored a few approaches with extending the underlying classes and creating our own kernel, but never got something that completely worked. It felt like I was on a track that could work, but I lacked the expertise around how jupyter loaded extensions and didn't really have to time to dig as deep as probably required.

We ended creating a patch file as a short term effort to avoid fork.

Please feel free to close this issue.