Notebook implementation in core (rendering / editor features)

[rebornix]

Follow up of https://github.com/microsoft/vscode/issues/88243, and the work is now happening in https://github.com/microsoft/vscode/pull/86632 . We will build a solid implementation of Notebook Editor in the core this iteration based on the feature list we curated.

We will focus on the rendering of inputs/outputs and basic cell manipulation commands first and then explore how to support editor features in native notebook editor. At the same time we should move off workarounds in the prototype to ensure the PR in a mergeable state.

• Rendering
  • Outputs Mimetypes
  • [ ] 🏃 LaTeX (see details https://github.com/microsoft/vscode/issues/90382#issuecomment-584479435).
  • [x] application/json.
  • [ ] image/*. We render PNG and JEPG in core but svg and gifs should be rendered in webview/iframe. (GIF support is a must as we don't support videos).
  • [x] application/scripts.
  • Metadata
  • [ ] Cell metadata which controls the renderings of cells, like editable, execution_count
  • [ ] Output metadata, like whether the output should be rendered in an isolated context.
• Cell Manipulation
  • [ ] run cell, select below
  • [ ] run cell and insert below
  • [ ] insert cell above
  • [ ] insert cell below
  • [ ] cut selected cells
  • [ ] copy selected cells
  • [ ] paste
• Editor features
  • [ ] Save/SaveAs/AutoSave
  • [ ] Hot Exit
  • [ ] View States preserve and restore
  • [ ] Cursor movement across cells
• Engineering
  • List View
  • [ ] Dynamic height update
  • [ ] isRendering state
  • [ ] Optional List Item Height style
  • [ ] Willscroll event
  • [ ] Backing layer
  • Webview
  • [ ] body height/width 100%
  • [ ] wheel event handler
  • [x] ~Event for initial layout (for dimensions info)~
  • CodeEditorWidget
  • [x] Remove DOM safely

[rebornix]

LaTeX

The markdown engine in classic Jupyter Notebook supports MathJax subset of TeX and LaTeX, this is a challenge to our current design as we render Markdown inputs/outputs in the core by using marked.js which ships with VS Code, but marked.js is not MathJax-aware. If we don't want to ship VS Code with MathJax, we need to provide a hook for notebook extensions to process the MathJax syntaxes for us.

Grammar

Math expressions are wrapped inside two double dollar signs in markdown cells, like below

# header

$$c = \sqrt{a^2 + b^2}$$

Syntax highlighting for Math expressions can be contributed by a markdown/notebook extension as the markdown editor is a regular Monaco Editor.

Rendering

Above math expression will be rendered like below in notebooks

Currently MathJax node library can convert math expressions to three different outputs

1. MML, Math Markup Language (a w3c standard for for mathematical and scientific content). It's supported in Safari and Firefox but not yet in Chromium. The advantage of MML is it doesn't require any additional resources like fonts or CSS to render properly. Igalia is actively working on MML support in Chromium (tracking issue/pr) so it might come in the near future.
2. HTML. MathJax can be converted to HTML elements like span but it requires additional web font and styles to render scientific symbols.
3. SVG

_1 is promising but it's not there yet, #_2 and #_3 look similar but #_3 is easier to handle as it doesn't require additional resources to be loaded.

With approach #_3, the workflow of rendering a markdown cell with math expression will be

• Render markdown as normal
• Find math expression tag ($$__$$)
• Request Notebook Extension to convert the expression to svg (or maybe MarkdownString)
• Replace the placeholder for math expression to an iframe which embeds the svg

Update

We considered converting math expression to png and then we can directly render it in the core but the svg-png conversion is slow (4 seconds to convert above expression from svg to png).

[davidanthoff]

I started playing with a Julia notebook implementation as part of the Julia extension over at https://github.com/julia-vscode/julia-vscode/pull/980. Barely functional, but I thought it might be helpful to try this out with something other than Jupyter notebooks. So far so good, it looks all pretty simple!

One piece of feedback on the markdown stuff: we would want to use a different markdown dialect than the Jupyter one (Julia had its own dialect for ages), and I'm sure other projects would want to use something different yet again. Could that be supported somehow? I guess there are two options: 1) you provide a hook for us to convert the markdown that is in the cell to the markdown that your renderer understands, and then you put that hook into the display pipeline (i.e. whenever markdown is displayed, it first flows through our md->md converter, then to the converstion to HTML), or 2) you provide extensions a hook to convert to HTML themselves?

On the math stuff: we would also need inline math, i.e. math that doesn't show up as its own paragraph, but is in the middle of a sentence, say.

Oh, and finally on the math stuff: equation numbering! Really crucial to have a way to number equations in markdown, and have the equation number show up nicely.

[rebornix]

@davidanthoff thanks for your early feedback on this ❤️ . Supporting classic Jupyter notebook is one of our goals but not the only one, our design will not introduce Jupyter notebook specifics into the core or API. The reason that we validate against Jupyter is that we want to make sure that extensions can actually build a Jupyter complaint notebook. Please note that the API is far from finalization so please be prepared that it can change often and significantly.

@mjbvz and me were talking about the markdown engine to use under the hood, one idea is moving to markdown-it which is used by the builtin markdown extension. IMHO we will only choose one dialect in the core, say CommonMark (probably with GFM).

Speaking of allowing extensions to control the grammar/syntax, currently I'm in favor of the first option you mentioned above: extensions are responsible for converting Markdown content to Markdown and the core is responsible for converting it to HTML. We'll end up rendering everything in webview/iframe if the extensions handle the md-html conversion, which will change how we render the editor completely. I'll investigate a bit more on this part and look forward to feedbacks.

BTW thanks for pointing out inline math, I didn't realize that $__$ syntax is for inline expression.

[davidanthoff]

Please note that the API is far from finalization so please be prepared that it can change often and significantly.

Yep, fully aware of that, no worries :) So far this did not take much time at all, and if we discover any design issues that way, I'm sure it is more useful than discovering them once things have shipped, so I'm happy to play the guinea pig here.

One more feature that is present in some non Jupyter notebooks and that I'd love to support for Julia: an ability to add a code block inline in the markdown, for example with

Normal markdown here, but then we have an inline `j call_func_foo()` code call here.

R Markdown has something like that as well. If one then knits (their terminology) such a R Markdown file into say a PDF, then that code part is replaced with the return value of that function call. I'm a bit unclear right now how that would all look in an interactive notebook editing experience, but I think for scientific computing that is a super useful idea.

Is this the best place to communicate with you guys about this?

[rebornix]

Currently I put feature list and analysis in https://github.com/rebornix/notebook-test so we can have discussions there too (I saw you created issues there already https://github.com/rebornix/notebook-test/issues/4).

[rebornix]

Now we are using #91987 to track the notebook features.