Autocomplete with remote kernels

TheMightyOnyx commented 8 months ago

What I am trying to do...

Hey, I'm trying to figure out if there's a way to get autocomplete suggestions for packages that are not installed on the jupyterlab container. I have packages installed on the kernel only which is started via jupyter enterprise gateway, for which I would like to get autcomplete functionality. If I have pylsp installed locally (on the jupyterlab container) I can get autocomplete for standard python packages etc just fine, but for example pandas, which is installed on the kernel only, has no autocomplete.

How I would like to learn how to do it...

I have searched the documentation and I haven't been able to locate any info regarding remote kernels. I was wondering if you guys could point me in the right direction. Thanks!

How the project might keep the docs accurate...

N/A

krassowski commented 8 months ago

Kernel should provide their own completions and LSP their own. These are merged and reconciled so you should be getting both. I would suggest checking in settings whether kernel completions do not happen to be disabled.

TheMightyOnyx commented 8 months ago

Thank you for the reply! Unfortunately it doesn't seem to be working that way for me, unless I missed some configuration:

Lab pod (with jupyterlab-lsp, jupyter-lsp and python-lsp-server installed):

root@jupyter:/srv/jupyterhub# pip list | grep lsp
jupyter-lsp                2.2.2
jupyterlab-lsp             5.0.2
python-lsp-jsonrpc         1.1.2
python-lsp-server          1.10.0

Kernel (with jupyter-lsp (wasn't sure if necessary but did it just in case) and python-lsp-server installed):

edw@hadoop-yarn-nodemanager-0:/$ pip list | grep lsp
jupyter-lsp                  2.2.2
jupyterlab-lsp               5.0.2
python-lsp-jsonrpc           1.1.2
python-lsp-server            1.10.0

Path has autocomplete:

Pandas doesn't do anything:

Relevant settings (I tried setting the timeout to 10000):

krassowski commented 8 months ago

Currently the completions from IPython kernel only work after you actually imported the Path from pathlib/DataFrame from pandas and defined the variable you are trying to complete on (a). It is not clear from the screenshot if that was the case. For local installations pylsp can be configured to include search paths from other locations, I am not sure if this would work for remote kernels though.

TheMightyOnyx commented 8 months ago

The a variable was already defined at the time of me trying to get completions for it (the a.values is printed below in the screenshot). So just for me to see if I understand correctly so I can experiment with this further, the auto discover functionality of jupyterlab-lsp uses the lsp installed on the remote kernel? Or how do the kernel side completions work?

What packages need to be installed where? Jupyterlab: jupyter-lsp, jupyterlab-lsp and pylsp Kernel: pylsp? jupyter-lsp?

By pylsp can be configured to include search paths from other locations do you mean via the hidden symlink trick I've seen somewhere around the docs, or is there another configuration I can try?

krassowski commented 7 months ago

By pylsp can be configured to include search paths from other locations do you mean via the hidden symlink trick I've seen somewhere around the docs, or is there another configuration I can try?

No, I mean that pylsp has pylsp.plugins.jedi.extra_paths option (https://github.com/python-lsp/python-lsp-server/blob/develop/CONFIGURATION.md) which tells Jedi (a library used by pylsp for completion and some other features) to search in specific paths for installed libraries.

What packages need to be installed where?

Depends on your configuration; one option is:

JupyterLab environment: jupyter-lsp, jupyterlab-lsp, pylsp + install all libraries you import on runtime
Kernel environment: all libraries you import on runtime

Another option is:

JupyterLab environment: jupyter-lsp, jupyterlab-lsp, pylsp + configure pylsp.plugins.jedi.extra_paths to point to kernel environment
Kernel environment: all libraries you import on runtime

But the second option only works if the lab environment can see the kernel environment (e.g. they are on the same machine).

MordorianGuy commented 6 months ago

By pylsp can be configured to include search paths from other locations do you mean via the hidden symlink trick I've seen somewhere around the docs, or is there another configuration I can try?

No, I mean that pylsp has pylsp.plugins.jedi.extra_paths option (https://github.com/python-lsp/python-lsp-server/blob/develop/CONFIGURATION.md) which tells Jedi (a library used by pylsp for completion and some other features) to search in specific paths for installed libraries.

What packages need to be installed where?

Depends on your configuration; one option is:
* JupyterLab environment: jupyter-lsp, jupyterlab-lsp, pylsp + install all libraries you import on runtime

* Kernel environment: all libraries you import on runtime
Another option is:
* JupyterLab environment: jupyter-lsp, jupyterlab-lsp, pylsp + configure `pylsp.plugins.jedi.extra_paths` to point to kernel environment

* Kernel environment: all libraries you import on runtime
But the second option only works if the lab environment can see the kernel environment (e.g. they are on the same machine).

Do I understand you right that these both options bind the jupyterlab environment to only one working kernel environment? Is there a general way to use autocompletion & jumping to sources while I am working with multiple environments at a time?

P.S. My case include only local environments on the same machine/OS.

jupyter-lsp / jupyterlab-lsp