search

jupyter-lsp / jupyterlab-lsp

Coding assistance for JupyterLab (code navigation + hover suggestions + linters + autocompletion + rename) using Language Server Protocol
https://jupyterlab-lsp.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.79k stars 147 forks source link

LSP for slow-starting (remote) kernels #1030

Open Zsailer opened 9 months ago

Zsailer commented 9 months ago

Description

I've been testing jupyterlab-lsp (4.3.x) with JupyterLab 3.6.x running on a system that

  1. has slow (~1 min) kernel start up times.
  2. runs kernels remotely from the server.

Due to (1), jupyterlab-lsp's Notebook adapter never connects on the first instance of the Notebook document. Instead, I wait for the kernel to fully start and connect, then close the document and re-open it for LSP to properly connect.

Due to (2), the LSP server is running in a different e.g. Python environment (i.e. on the server) than the (remote) kernel. This isn't a big deal, we can give some generic Python LSP power from language server running on Jupyter Server, and let the kernel give more useful help once it's warmed up.

Ideally, I would love to use jupyterlab-lsp to provide some basic (coming from the Jupyter server's python env) LSP hinting while the Notebook is starting (or even if no kernel is provided). Then, once the kernel is ready and connects, bring in the additional context from the kernel.

I don't think this is possible today, correct?

If so, I looked into writing a custom NotebookAdapter (from 4.x branch—I'm still stuck in JupyterLab 3.6.x land for awhile), but the interfaces and classes I need (e.g. NotebookAdapter and WidgetAdapter) are not exported.

So... (sorry for the long intro 😅) a few follow-up questions:

  1. Would you be willing to export NotebookAdapter and WidgetAdapter in a 4.x release soon? I can submit the necessary PR and help with shipping, etc.
  2. Does it make sense to fix for the slow, remote kernel case more generally?

(JupyterLab itself is pretty brittle when it comes to slow, remote kernels—so I think there is a broader discussion about how to improve the whole application for this usecase.)

Reproduce

Reproducing this is a bit of work. You need to mock a slow starting kernel (~1 min). I run this on a cluster, with kernels managed by a separate server from the Jupyter Server, then connect to the kernel remotely.

Expected behavior

Ideally, I would love to use jupyterlab-lsp to provide some basic (coming from the Jupyter server's python env) LSP hinting while the Notebook is starting (or even if no kernel is provided). Then, once the kernel is ready and connects, bring in the additional context from the kernel.

Context

Required: installed server extensions 
Paste the output from running `jupyter server extension list` (JupyterLab >= 3)
or `jupyter serverextension list` (JupyterLab < 3) from the command line here.
You may want to sanitize the paths in the output.
Required: installed lab extensions 
Paste the output from running `jupyter labextension list` from the command line here.
You may want to sanitize the paths in the output.
Troubleshoot Output 
Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.
Command Line Output 
Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.
Browser Output (recommended for all interface issues) 
Paste the output from your browser JavaScript console replacing the text in here.

To learn how to open the developer tools in your browser:
https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools#How_to_open_the_devtools_in_your_browser
If too many messages accumulated after many hours of working in JupyterLab,
consider refreshing the window and then reproducing the bug to reduce the noise in the logs.
Zsailer commented 9 months ago

If so, I looked into writing a custom NotebookAdapter (from 4.x branch—I'm still stuck in JupyterLab 3.6.x land for awhile), but the interfaces and classes I need (e.g. NotebookAdapter and WidgetAdapter) are not exported.

I see this was already requested: https://github.com/jupyter-lsp/jupyterlab-lsp/issues/774#issuecomment-1062463905

krassowski commented 9 months ago

Would you be willing to export NotebookAdapter and WidgetAdapter in a 4.x release soon

If needs be, sure! But it would be perfect to have a more general solution so maybe it would not be needed?

Does it make sense to fix for the slow, remote kernel case more generally?

Certainly! Is there a way to get kernel_info earlier? If not we could use the one from notebook metadata, right?

Zsailer commented 9 months ago

If not we could use the one from notebook metadata, right?

Yeah, that's right.

In fact, this is what I was attempting to do when writing with my own NotebookAdapter. Determine the language from the Notebook's metadata and begin streaming LSP info right away... then try to ask the kernel for more context once its ready.

I'll work on a PR. 🙂

krassowski commented 7 months ago

Due to (2), the LSP server is running in a different e.g. Python environment (i.e. on the server) than the (remote) kernel.

Cross-linking with https://github.com/jupyter-lsp/jupyterlab-lsp/issues/1041 which discusses how some language servers can be configured to include packages from separate kernel environments in the inference process.