Open edwardps opened 1 year ago
Add more details. According to SparkMagic owner, seems this is caused by the asynchronous nature of the execute method in the SQLQuery class.
There is some troubleshooting ongoing. FYI https://github.com/jupyter-incubator/sparkmagic/issues/833#issuecomment-1710559834
🐛 Bug
Hi,
Issue
Our use case is to use SparkMagic wrapper kernels with PaperMill job. Most of the functions are working as expected except the %%sql magic which will get stuck during execution. The SparkMagic works properly when executed in interactive mode in JupyterLab and issue only happens for %%sql magic when running with PaperMill.
From the debugging log(attached), I can see the %%sql logic had been executed and response was retrieved back. The execution state was back to idle at the end. But the output of %%sql cell was not updated properly and the following cells were not executed.
Following content was printed by PaperMill, which shows the %%sql has been executed properly. This content was not rendered into cell output.
Output notebook of papermill:
Expected output(from JupyterLab)
Reproducing steps
Following is package list which might be highly related. I also attached one text contains all the packages.
log and other files.zip contains:
I am not very sure if this is a issue of papermill or sparkmagic. I will also copy this issue to SparkMagic to see if there happend to be any expert who can provide advice. If it's confirmed it's caused by SparkMagic, please feel free to close this issue. Thanks in advance.