spark magic command to shut down current session

[cometta]

any spark magic command that i can run on last notebook cell to call lighter to shut down current running spark session and shut down current session k8s pods?

[Minutis]

Hello,

try %%cleanup -f. We do not use this kind of functionality so it's hard to say for sure.

[cometta]

cleanup -f will delete all sessions including other people session. reference: https://github.com/jupyter-incubator/sparkmagic/blob/master/examples/Pyspark%20Kernel.ipynb . I look for way to delete my current session

[Minutis]

Not the prettiest solution, but you could simply try this:

spark.stop()
exit()

This will shutdown the current application.

[cometta]

it works but the k8s session pod in kubectl describe

State:          Terminated
│       Reason:       Error
│       Exit Code:    1
│       Started:      Fri, 14 Jul 2023 17:24:19 +0800
│       Finished:     Fri, 14 Jul 2023 17:27:40 +0800
│     Ready:          False

[Minutis]

I don't think there is a way to exit gracefully. Also this feature should probably be developed on Sparkmagic side.