Programatically Select a Kernel from a Remote Kernel Provider

[rjra2611]

Applies To

• [X] Notebooks (.ipynb files)
• [ ] Interactive Window and\/or Cell Scripts (.py files with #%% markers)

What happened?

I'm trying to upgrade from the deprecated registerRemoteServerProvider / addRemoteJupyterServer to createJupyterServerCollection

My requirement is to register the kernels of my JupyterServer object programmatically and not when the user clicks on Select Kernel -> Server Collection -> Server Provider -> Server Kernels

I'm eagerly providing the connectionInformation of my JupyterServer but I don't see the required kernel available in the "all" list in the below log snippet.

WARN wanted kernel DOES NOT EXIST, wanted: ms-toolsai.jupyter/3d0788656d32d0ae35e60dca2d6fb768ff8834a09cf723b8f2b2eb946b4b0608..python3.python.\.-m#ipykernel_launcher, all: ms-toolsai.jupyter/..net-csharp.dotnet.\.dotnet#interactive#jupyter#--default-kernel#csharp#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/..net-fsharp.dotnet.\.dotnet#interactive#jupyter#--default-kernel#fsharp#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/..net-powershell.dotnet.\.dotnet#interactive#jupyter#--default-kernel#powershell#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/.jvsc74a57bd06158585552ee969444871af435853f627afd4d272d93bfe7d8398764498bf7fa.d:\projects\AI\text-generation-webui\installer_files\conda\python.exe.d:\projects\AI\text-generation-webui\installer_files\conda\python.exe.-m#ipykernel_launcher,ms-toolsai.jupyter/.jvsc74a57bd06bf80c1c90e9cf006aca841dd66d43ffb3ee8f693be718480ef962d54e296f78.d:\projects\AI\text-generation-webui\installer_files\env\python.exe.d:\projects\AI\text-generation-webui\installer_files\env\python.exe.-m#ipykernel_launcher,ms-toolsai.jupyter/.jvsc74a57bd01443aad6ee6ac63f067ffd57b952034da9e51c0d9bfbb593699952188d41867c.d:\python-3.10\python.exe.d:\python-3.10\python.exe.-m#ipykernel_launcher

Once I go through the manual flow of selecting the kernel from the UI the required kernel starts appearing as seen in the log snipped below

WARN wanted kernel DOES NOT EXIST, wanted: ms-toolsai.jupyter/3d0788656d32d0ae35e60dca2d6fb768ff8834a09cf723b8f2b2eb946b4b0608..python3.python.\.-m#ipykernel_launcher, all: ms-toolsai.jupyter/4de27e131975a17b732cb295a087c5981bc41a3fb74774540f27103205bb3b9b..python3.python.\.-m#ipykernel_launcher,ms-toolsai.jupyter/..net-csharp.dotnet.\.dotnet#interactive#jupyter#--default-kernel#csharp#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/..net-fsharp.dotnet.\.dotnet#interactive#jupyter#--default-kernel#fsharp#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/..net-powershell.dotnet.\.dotnet#interactive#jupyter#--default-kernel#powershell#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/.jvsc74a57bd06158585552ee969444871af435853f627afd4d272d93bfe7d8398764498bf7fa.d:\projects\AI\text-generation-webui\installer_files\conda\python.exe.d:\projects\AI\text-generation-webui\installer_files\conda\python.exe.-m#ipykernel_launcher,ms-toolsai.jupyter/.jvsc74a57bd06bf80c1c90e9cf006aca841dd66d43ffb3ee8f693be718480ef962d54e296f78.d:\projects\AI\text-generation-webui\installer_files\env\python.exe.d:\projects\AI\text-generation-webui\installer_files\env\python.exe.-m#ipykernel_launcher,ms-toolsai.jupyter/4de27e131975a17b732cb295a087c5981bc41a3fb74774540f27103205bb3b9b..csharp.dotnet.\.dotnet#interactive#jupyter#--default-kernel#csharp#{connection_file}#--http-port-range#1000-3000,ms-toolsai.jupyter/4de27e131975a17b732cb295a087c5981bc41a3fb74774540f27103205bb3b9b..foundation-pomegranate.Foundation-Pomegranate\bin\python.\.-m#ipykernel_launcher,ms-toolsai.jupyter/4de27e131975a17b732cb295a087c5981bc41a3fb74774540f27103205bb3b9b..hudson-thames.Hudson-Thames\bin\python.\.-m#ipykernel_launcher,ms-toolsai.jupyter/.jvsc74a57bd01443aad6ee6ac63f067ffd57b952034da9e51c0d9bfbb593699952188d41867c.d:\python-3.10\python.exe.d:\python-3.10\python.exe.-m#ipykernel_launcher

• Also, now that addRemoteJupyterServer(providerId: string, handle: string): Promise<void>; is deprecated. How can I determine the server Id which is a SHA256 of https://remote/?id=${providerId}&uriHandle=${handle}&extensionId=${extensionId}. given the handle is no longer valid? Is it replaced with other attribute of JupyterServer like id / label or the pattern itself doesn't include ${handle} anymore?

Jupyter Output Logs

extension debugging logs.txt

VS Code Version

1.86.1

Jupyter Extension Version

2024.1.1

Jupyter logs

No response

Coding Language and Runtime Version

No response

Language Extension Version (if applicable)

No response

Anaconda Version (if applicable)

No response

Running Jupyter locally or remotely?

None

[rjra2611]

I did find out how the handle and Id are now assigned but still not able to register the kernels without the manual flow. I also tried firing the onDidChangeServers event which fetches the available servers by calling provideJupyterServers but that doesn't result in the kernels of the provided server being registered

const serverProviderHandle = {
      extensionId: collection.extensionId,
      handle: server.id,
      id: collection.id
  };

[DonJayamanne]

I'm eagerly providing the connectionInformation of my JupyterServer but I don't see the required kernel available in the "all" list in the below log snippet.

How are you accesing the all list? I.e. can you share the code you use to access this all list.

My requirement is to register the kernels of my JupyterServer object programmatically and not when the user clicks on Select Kernel -> Server Collection -> Server Provider -> Server Kernels

Please can you share some screenshots or similar so I can understand the desired workflow.

The idea behind the API, is for 3rd party to provide servers, and users will then select kernels from those servers and run code against it. However it seems you would like to do something else.

[DonJayamanne]

Also, now that addRemoteJupyterServer(providerId: string, handle: string): Promise; is deprecated. How can I determine the server Id which is a SHA256 of https://remote/?id=${providerId}&uriHandle=${handle}&extensionId=${extensionId}.

Not sure what you mean. Not sure whether you've seen this sample code, https://github.com/microsoft/vscode-extension-samples/blob/main/jupyter-server-provider-sample/src/extension.ts

Basically extension authors no longer need to know anything about SHA values, handles, provider ids, etc Extension authors can use any mechanism they chose. In the sample, I embed the port and the like into the id of the server. This isn't great, but shows simple approaches. If you have URLs, tokens or the like, you can store this in secret storage or the like and them map it to the id provided to the Jupyter Extension.

[rjra2611]

However it seems you would like to do something else.

Right, my workflow is that on notebook open I automatically select kernel for my users. For that I use

vscode.commands.executeCommand('notebook.selectKernel');

How are you accesing the all list? I.e. can you share the code you use to access this all list.

When triggering the above command I get the error with all list which I believe is because the createRemoteKernelFinder hasn't been called yet and the list kernels getListOfKernelsWithCachedConnection from the server I provided haven't been registered.

I can debug the issue to some extent and see that when jupyter runs the method lookForServersInCollectionAndRemoveOldServers it returns before creating the kernel finder (createRemoteKernelFinder)

// If this sever was never used in the past, then no need to create a finder for this.
  if (this.mappedServers.has(serverId) || !usedServers.has(serverId)) {
      return;
  }

snippet reference

If the expected workflow requires the user to use the kernel selector QuickPick and only then from next time onwards the kernels finder is created. Is there a way I can trigger the kernel finder process automatically when I submit a new server / first time? ( or skip the return in the above snippet to proceed to (createRemoteKernelFinder) )

[DonJayamanne]

w requires the user to use the kernel selector QuickPick and only then from next time onwards the

Yes, this is the expectation.

I can trigger the kernel finder process automatically when I submit a new server / first time? ( or skip the return in the above snippet to proceed to (createRemoteKernelFinder) )

Thanks for the feature suggestion, I'll discuss this with the team.

[DonJayamanne]

Right, my workflow is that on notebook open I automatically select kernel for my users. For that I use vscode.commands.executeCommand('notebook.selectKernel');

Curious, how does calling force a particular kernel to be selected? Please can you share some of the code used.

[DonJayamanne]

Marking as partner as Synapse & Azure ML also require a similar feature (ability to bypass Kernel picker for some cases). @realDuang @sevillal /cc

[rjra2611]

Right, my workflow is that on notebook open I automatically select kernel for my users. For that I use vscode.commands.executeCommand('notebook.selectKernel');

Curious, how does calling force a particular kernel to be selected? Please can you share some of the code used.

Once the kernel is registered (createRemoteKernelFinder) and available. I run the below command to force select a kernel

// let fullId = SHA256(`https://remote/?id=${providerId}&uriHandle=${handle}&extensionId=${extensionId}`) + "." + ".python3./python./.-m#ipykernel_launcher"
vs.commands.executeCommand('notebook.selectKernel', {
                    id: fullId,
                    extension: "ms-toolsai.jupyter"
                });

[DonJayamanne]

fullId

Please can you share the code used to generate/get this