Code/syntax highlighting broken on Jupyter Lab

[jaanli]

Description

In this example notebook I made yesterday:

https://colab.research.google.com/github/jaanli/language-model-notebooks/blob/main/notebooks/getting-started.ipynb

The prompts render correctly:

But in Jupyter lab if I run jupyter lab notebooks/getting-started.ipynb from the root of the repo (https://github.com/jaanli/language-model-notebooks/) then the prompt cells don't render correctly:

(Note the red highlighting after the single quote, not sure what is happening)

Reproduce

Create a code cell and use the %%ai cell magic, then add a prompt using and or ' or other such characters/operators.

Expected behavior

Syntax highlighting works.

This is really cool! Thank you for an awesome jupyter feature.

[JasonWeill]

See https://github.com/jupyterlab/jupyterlab/issues/3869 and https://github.com/jupyterlab/jupyterlab/issues/15324. In JupyterLab 4, magic commands are treated as code with the current kernel's language; for example, if you're using a Python kernel, all magic commands are highlighted as if they were Python code, so any apostrophe is treated as the start or end of a string.

Fixing this bug will require upstream changes in JupyterLab; it is not necessarily a bug in Jupyter AI, but it is evident to people using Jupyter AI magic commands with JupyterLab.

[krassowski]

:100: if someone is interested in contributing to JupyterLab a PR which modifies the IPython language mode so that cell magics get an overlay with proper syntax highlighting (or plain formatting in case of %%ai) I am happy to review the PR (or help in other ways).

[dlqqq]

Yes, I think plain formatting would work best here; otherwise we would somehow need to declare the syntax of the magics back up to the highlighter, which seems too complex at this time.

[jaanli]

Sounds good, thank you @dlqqq - would it be easy to remove all formatting and switch it to plain text? I'm happy to give this a shot just not sure where to start :S