Kernel protocol roadmap

[JohanMabille]

Below is a list of the JEPs related to the Kernel protocol, with the actions required for both the documentation and the implementation. The idea is to have an easier process to update the protocol and trigger the required actions in downstream projects. The first step would be to have a dedicated repo for the kernel protocol specification instead of having it in Jupyter Client and probably move this issue there.

Current version of the protocol: 5.4

Title	Links	Status	Notes
Debugger	PR JEP	- JEP Accepted - In the protocol 5.5 - Implemented in xeus-python and ipykernel	Should be marked as optional when the Optional Features JEP is accepted
Deprecate shutdown request on shell		- No JEP - In the protocol 5.4
XPub	PR JEP	- JEP Accepted - Not in the protocol
Handshake mechanism	PR	JEP opened
Optional Features	PR	JEP opened
Parametrized kernels	PR	JEP opened
Subshells	PR	JEP opened	Depends on Optional Features JEP
Kernel info request on control	PR	JEP opened
Status messages on control		No PR yet
Unkillable kernels		No PR yet

Note: in the format below, when we require kernels to warn, we mean:

• Warn on whatever stderr output stream makes sense for the kernel process.
• Possible issue a warning over the kernel protocol (to be defined, with an IOPub or a reply).

XPub

Having kernels implement the XPub pattern for IOpub will not break the clients to the protocol that may not use XPub yet.

If a client requires kernels to implement XPub, they can remove the "nudging" mechanism which was a workaround to the IOpub taking too long to connect.

Proposal:

• Minor version of the protocol that requires XPub.
• Update jupyter-kernel-test to error when Xpub is not implemented in this new protocol version.
• Make jupyter_client, jupyter_server, jupyverse emit a warning when encountering a kernel that does not support this version of the protocol.
• Eventually (in a few years), a major version of the clients will be needed to require XPub (require this version of the protocol).

Handshake

The new handshake mechanism is incompatible with the old one.

Kernels will need to support both for some time, and clients will warn when encountering kernels that implement only the old protocol.

Proposal:

• Minor version of the protocol that requires the handshake mechanism to be supported next to the old connection mechanism.
• Update jupyter-kernel-test to error when handshake is not implemented in this new protocol version.
• A major version of the protocol will be required to remove the old connection mechanism.
• Make jupyter_client, jupyter_server, jupyverse emit a warning when encountering a kernel that does not support this version of the protocol (and therefore does not support the new handshake).
• Eventually (in a few years), require kernels to implement the new version of the protocol and remove the old connection mechanism in a major version of clients.

Optional features

Today we have a singular way to tell whether kernels support debugging.

Proposal:

• Minor version of the protocol that allows listing the optional features in the kernel_info_reply message, and requires debugging kernels to signal support for debugging that way.
• Update jupyter-kernel-test to error when a debugging kernel does not signal debugging support via optional features.
• Update clients that use debugging to check whether this optional feature is listed when the boolean for debugging support is true. If not, warn.
• Major version of the protocol that removes the boolean attribute indicating debugger support.

Parameterized kernels

Proposal:

• Minor version of the protocol allowing parameterized kernels (Are kernelspecs considered part of the protocol?)

Subshells

Proposal:

TODO

Kernel info request on control

Proposal:

• Minor version of protocol requiring kernels to warn when receiving kernel info requests on shell.
• Update jupyter-kernel-test to require kernels to warn when receiving kernel info requests on shell.
• Update clients to send kernel info requests on control.
• Major version of protocol requiring kernels to error when receiving kernel info request on shell (better than failing silently in case of an old client that would never see the connection to the kernel completing).

Deprecate shutdown request on shell

Proposal:

• Minor version of protocol deprecating shutdown request on shell (Done in 5.4)
• Update jupyter-kernel-test to require kernels to warn when receiving shutdown requests on shell for protocol 5.4.
• Major version of protocol requiring kernels to error when receiving shutdown request on shell (better than failing silently in case of an old client that would never see the kernel being shut down).

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[bollwyvl]

Best prior art for this kind of spec and documentation:

https://xmpp.org/extensions/

[bollwyvl]

Another idea: encourage treating docker-stacks as the upstream of testable kernels.