microsoft / vscode

Visual Studio Code
https://code.visualstudio.com
MIT License
164.11k stars 29.27k forks source link

Native notebooks LaTeX support #119536

Closed joyceerhl closed 3 years ago

joyceerhl commented 3 years ago

Steps to Reproduce:

  1. Download https://github.com/joyceerhl/notebooks/blob/main/markdown.ipynb. This notebook contains a roundup of LaTeX examples that users have reported did not work in either VS Code notebooks or the Jupyter extension's webview-based notebooks implementation. Each cell contains the LaTeX code and the GitHub issue it was reported in.
  2. Open the notebook in the new native notebooks interface in the latest VS Code Insiders with notebook.experimental.useMarkdownRenderer set to true.
  3. Note that some but not all of the examples are correctly rendered.

Does this issue occur when all extensions are disabled?: Yes

vscodebot[bot] commented 3 years ago

(Experimental duplicate detection) Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

mjbvz commented 3 years ago

I don't think we've committed to full latex support. We ship support for katex math formulas to validate the new, extensible markdown renderers. It's still TBD if we would bundle this limited support or require notebook providers to bring their own markdown extensions

joyceerhl commented 3 years ago

Following up on Friday's meeting, I did some investigation to root cause the math examples we talked about that aren't rendering with the current native notebooks experimental markdown renderer. Please let me know if I can provide other context that might be more helpful.

Math in markdown in other notebooks

Notebook offering Markdown parser Math library Additional notes
JupyterLab marked MathJax 2 JupyterLab preprocesses markdown to hide math expressions from their markdown parser and defer math handling to MathJax. The @jupyterlab/rendermime package provides default renderers for Markdown in Jupyter. Logic for preprocessing markdown: remove math, convert markdown to HTML, replace math, then render HTML. Logic for preprocessing math. JupyterLab also provides extensions for rendering math using KaTeX or MathJax 3: https://github.com/jupyterlab/jupyter-renderers
nteract @nteract/outputs package vendors @nteract/markdown which uses react-markdown MathJax 2 via @nteract/mathjax @nteract/mathjax is a React wrapper that pulls MathJax from https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.9/MathJax.js?config=TeX-MML-AM_CHTML

(Google Colaboratory notebooks aren't open source but this comment indicates they try to be consistent with JupyterLab.)

(jupyter/nbformat has an issue tracking standardizing its markdown + LaTeX support as part of the nbformat spec, although that issue hasn't seen activity in some time.)

Math in markdown in VS Code native notebooks

Code that doesn't work Cause What to do about it
Jupyter notebooks documentation on examples of MathJax equations https://katex.org/'s playground correctly renders the source for almost all of these examples, so this is not a syntax parity issue. Current (unvalidated) theory is this works in JupyterLab due to the math preprocessing that they perform. Implement same preprocessing hacks in math renderer extension and see if that suffices
Euler's identity: $$ e^{i \pi} + 1 = 0 $$ Likely caused by markdown parser. Inline $$-delimited math equations are rendered in block style in JupyterLab. Users can work around this by placing the $$ bound section on a newline. Implement same preprocessing hacks in math renderer extension and see if that suffices
$ e^{i \pi} + 1 = 0 $ Known limitation of markdown-it-katex library: doesn't support inline math equations where the equation delimiter $ is surrounded by spaces. See https://github.com/waylonflinn/markdown-it-katex#syntax Upstream fix to markdown-it-katex
$$ \begin{eqnarray} \frac{dx}{dt} &=& v, \text{ and} \\ \frac{dv}{dt} &=& a, \end{eqnarray} $$ Issue provides the same equation in single- and multi-line format. Both work in JupyterLab. Neither works with experimental markdown renderer support. Single-line variant works in MathJax demo and not in https://katex.org playground. Root cause seems to be a KaTeX-specific limitation: https://github.com/KaTeX/KaTeX/issues/1931. Add support for MathJax math flavor e.g. using markdown-it-mathjax or @nteract/mathjax-electron
$$ \text{For } x \in \mathbb{R}^n \text{, } sigmoid(x) = sigmoid\begin{pmatrix} x_1 \ x_2 \ ... \ x_n \end{pmatrix} = \begin{pmatrix} \frac{1}{1+e^{-x_1}} \ \frac{1}{1+e^{-x_2}} \ ... \ \frac{1}{1+e^{-x_n}} \end{pmatrix}\tag{1} $$ LaTeX equation provided is correctly rendered in JupyterLab but doesn't work in MathJax demo or KaTeX playground, so it doesn't seem to be a syntax parity issue. Replacing \\end with \end seems to get this particular example working in the MathJax demo and with the experimental markdown renderer in native notebooks. Possibly invalid LaTeX example
https://github.com/microsoft/vscode-jupyter/issues/4938 Code sample provided doesn't render in JupyterLab, MathJax playground, or KaTeX playground. Including for completeness. Possibly invalid LaTeX example
LaTeX in HTML in Markdown Likely requires a fix at the markdown parser level, although I don't know enough about markdown to say for sure. TBD

Other background

Leaving this background info here for anyone else who is starting out with understanding LaTeX in Markdown.

mjbvz commented 3 years ago

@joyceerhl I've forked markdown-it-katex (https://github.com/mjbvz/markdown-it-katex) and applied a few minor fixes

I think the Python team should look into these since Python is the primary consumer of math support. On the VS Code side, we're focusing on extensible markdown renderers right now. Feel free to submit PRs to the upstream repo or let me know if you find a better markdown-math integration library

joyceerhl commented 3 years ago

Thanks @mjbvz we agree and we plan to investigate leveraging the notebookMarkdownRenderer contribution point you added, tracked in https://github.com/microsoft/vscode-jupyter/issues/5365

mjbvz commented 3 years ago

@joyceerhl Just to make sure you we are on the same page: you shouldn't need to worry about notebookMarkdownRenderer all right now. We bundle https://github.com/mjbvz/markdown-it-katex into VS Code, so if your team can submit fixes for the markdown markdown parsing issues you identified, we'll pull them into VS Code and python won't have to have any custom Latex logic

mjbvz commented 3 years ago

A few more findings on KaTeX vs MathJax

Bundle Size

Tested using our current webpack script (production but no advanced optimization):

KaTeX

260kB of JS + 23kB of CSS + lazily loaded fonts. For a simple equation, I see this loading a 30kB font file

MathJax3

1.7MB all bundled together (using https://github.com/tani/markdown-it-mathjax3)

Code Loading

Using the above bundles, render times for equations are roughly similar. However MathJax takes significantly longer to initially evaluate (which makes sense since the script source is much larger).

The profiles below were collected inside a VS Code notebook

Katex

Screen Shot 2021-04-07 at 4 30 15 PM

MathJax

Screen Shot 2021-04-07 at 4 29 09 PM

Relayout

We've put in a lot of work to avoid having notebook cells shift around while loading or scrolling.

KaTeX

With KaTeX, here's a profile showing how equations are rendered:

https://user-images.githubusercontent.com/12821956/113949119-5d05a500-97c3-11eb-9635-4cfd8f0728be.mov

There are essentially three stages:

  1. Unrendered formula
  2. Mostly correct but not fully styled (unloaded font?)
  3. Final render

The only layout shift happens going between step 1 and 2

MathJax

Now here's using MathJax3:

https://user-images.githubusercontent.com/12821956/113949232-9fc77d00-97c3-11eb-8b42-13e93dcceca9.mov

Again, there are three stages:

  1. Unrendered formula
  2. Partially rendered
  3. Final render

However in this case there are two layout shifts: one during the first load and another between steps 2 and 3.

The KaTeX rendering is preferable here since it avoids an extra resize

Comparing LaTeX Support

Here's a list of LaTeX features that KaTeX supports or doesn't support.

Here's a list of supported LaTeX command for MathJax (I can't find a list of unsupported features and some elements on this list require using built-in MathJax extensions)

Using the developer console, I put together this really poor scraper to generate a list of LaTeX features supported in MathJax but not in Katex:

// Run these two snippets on each page

// https://katex.org/docs/support_table.html
const katexNotSupported = Array.from(document.querySelectorAll('td'))
    .filter(x => { const s = x.querySelector('span'); return s?.textContent === 'Not supported' })
    .map(x => x.previousSibling.textContent)

// https://docs.mathjax.org/en/latest/input/tex/macros/index.html
const mathJaxSupported = new Set(Array.from(document.querySelectorAll('tr td:first-child')).map(x => x.textContent))

// Then on some page with both sets of results
katexNotSupported.filter(x => mathJaxSupported.has(x))

This produces the following list:

- \abovewithdelims
- \array
- \Arrowvert
- \arrowvert
- \atopwithdelims
- \bbox
- \bracevert
- \buildrel
- \cancelto
- \cases
- \class
- \cssId
- \ddddot
- \dddot
- \DeclareMathOperator
- \definecolor
- \displaylines
- \enclose
- \eqalign
- \eqalignno
- \eqref
- \hfil
- \hfill
- \idotsint
- \iiiint
- \label
- \leftarrowtail
- \leftroot
- \leqalignno
- \lower
- \mathtip
- \matrix
- \mbox
- \mit
- \mmlToken
- \moveleft
- \moveright
- \mspace
- \newenvironment
- \Newextarrow
- \notag
- \oldstyle
- \overparen
- \overwithdelims
- \pmatrix
- \raise
- \ref
- \renewenvironment
- \require
- \root
- \Rule
- \scr
- \shoveleft
- \shoveright
- \sideset
- \skew
- \Space
- \strut
- \style
- \texttip
- \Tiny
- \toggle
- \underparen
- \unicode
- \uproot
mjbvz commented 3 years ago

I pushed a change that handles a few of the cases listed. Source text:

Screen Shot 2021-04-08 at 9 30 29 PM

Rendered result:

Screen Shot 2021-04-08 at 9 30 34 PM

Specifically this addresses:

The next easy one I'm looking into is the single $ math block :

$\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}$

Please test in the next insiders a let me know if you notice any regressions

/cc @kieferrm @rebornix

mjbvz commented 3 years ago

Ok, the majority of examples from the Jupyter Math example notebook now work too:

Screen Shot 2021-04-08 at 10 07 08 PM

Not sure if this change made the insiders for tomorrow or not.

The two remaining cases from the Jupyter notebook that don't work due to KaTeX:

delucca commented 3 years ago

Is the experimental native LaTeX support works in the VSCode version of Github Codespaces?

I'm trying it with the following version:

Version: 1.56.0-insider
Commit: 58186680eb97fe979e1d986bcfa983ab0a6228e4
Date: 2021-04-06T10:32:31.104Z
Browser: Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0

Even after enabling notebook.experimental.useMarkdownRenderer flag, any LaTeX formula simply won't work (even those reported here that are working).

Any idea why?

mjbvz commented 3 years ago

@delucca Please provide steps to reproduce the issue. Are you sure you are using native notebooks for example?

delucca commented 3 years ago

@mjbvz yes I am.

Do you have access to Github Codespaces? If so, you can do the following:

  1. Launch a new Codespaces on any repository
  2. Switch to insiders version
  3. Activate the notebook.experimental.useMarkdownRenderer flag
  4. Create a new blank Jupyter Notebook with Ctrl+P
  5. Try to type any formula on a markdown cell
mjbvz commented 3 years ago

Oh actually we have disabled the new renderers on web

https://github.com/microsoft/vscode/blob/9afc734800062717d835b9ac63235c0ec8a7bf50/src/vs/workbench/contrib/notebook/browser/notebookEditorWidget.ts#L353

Will revisit once we feel things are looking good on desktop

delucca commented 3 years ago

Oh actually we have disabled the new renderers on web

https://github.com/microsoft/vscode/blob/9afc734800062717d835b9ac63235c0ec8a7bf50/src/vs/workbench/contrib/notebook/browser/notebookEditorWidget.ts#L353

Will revisit once we feel things are looking good on desktop

Oh, I see.

There is no way to bypass that setting? I use the Github Codespaces as my primary IDE, and I would like to use this feature

mjbvz commented 3 years ago

@joyceerhl https://github.com/microsoft/vscode/commit/cd9a6a48201099f9441d5cd76581197c88ec34f3 Added telemetry to track how often something that looks like unsupported KaTeX directive appears in a native notebook

We'd still like your help analyzing a broader notebook population (i.e. not just those being opened in VS Code) to see which directives are used most commonly so we know what to prioritize

bellerofonte commented 3 years ago

Ok, the majority of examples from the Jupyter Math example notebook now work too:

Hi! I have the VSCode Version: 1.55.2 on MacOS Mojave, Jupyter extension v2021.5.745244803. Just tried this notebook - no equation was rendered properly. Tried to run all cells and then every cell manually.

VSCode output:

2021-05-03_1621

Browser-based notebook worked fine:

2021-05-03_1622

What is wrong in my VSCode?

Fate6174 commented 3 years ago

@bellerofonte I observe the same behavior. Try putting your equation- and align-environments within $$ ... $$, that works for me. Albeit the align environment giving each equation the same number...

Also, putting those environments within display style $$ ... $$'s is wrong syntax, I don't know why it works. Should be fixed, imho.

grafik

mjbvz commented 3 years ago

Closing as we've shipped support for math equations

Please open new issues for any further bugs / feature requests