Multiline HTML comments throw a MDX compilation error in canary (3.0.0-alpha.0)

andrewnicols commented 1 year ago

Have you read the Contributing Guidelines on issues?

[X] I have read the Contributing Guidelines on issues.

Prerequisites

[X] I'm using the latest version of Docusaurus.
[ ] I have tried the npm run clear or yarn clear command.
[ ] I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
[ ] I have tried creating a repro with https://new.docusaurus.io.
[X] I have read the console error message carefully (if applicable).

Description

The following HTML comments are fine:

<!-- Single line comment -->
<!-- Multi
line comment
-->

However, a multi-line comment which does not start on the opening line leads to an error:

<!--
Github Flavoured Markdown does not support tables without headers.
We must use an HTML table here.
Please note that Spacing in this table is important.
Markdown must have empty newlines between it and HTML markup.
-->

And the error:

Error: MDX compilation failed for file "/Users/nicols/git/moodlehq/devdocs/general/releases/2.3.md"
Cause: chunks[startIndex].slice is not a function
Details:
{}

Reproducible demo

No response

Steps to reproduce

Create an mdx file with:

<!--
Github Flavoured Markdown does not support tables without headers.
We must use an HTML table here.
Please note that Spacing in this table is important.
Markdown must have empty newlines between it and HTML markup.
-->

Expected behavior

HTML comment not shown

Actual behavior

An error in the yarn console:

Error: MDX compilation failed for file "/Users/nicols/git/moodlehq/devdocs/general/releases/2.3.md"
Cause: chunks[startIndex].slice is not a function
Details:
{}

Your environment

Public source code: https://github.com/moodle/devdocs
Public site URL: https://moodledev.io
Docusaurus version used: 3.0.0-alpha.0
Environment name and version Netlify
Operating system and version Netlify

3.0 branch is at https://github.com/andrewnicols/devdocs/commit/docusaurus3

Self-service

[ ] I'd be willing to fix this bug myself.

slorber commented 1 year ago

Thanks for reporting.

Unfortunately there's no stacktrace logged in this case in the console output, but only on the webpack dev server:

TypeError: chunks[startIndex].slice is not a function
    at sliceChunks (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/micromark/lib/create-tokenizer.js:514:32)
    at sliceStream (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/micromark/lib/create-tokenizer.js:154:12)
    at Object.sliceSerialize (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/micromark/lib/create-tokenizer.js:149:28)
    at Object.onexitdata (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/mdast-util-from-markdown/lib/index.js:866:24)
    at compile (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/mdast-util-from-markdown/lib/index.js:353:40)
    at fromMarkdown (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/mdast-util-from-markdown/lib/index.js:187:29)
    at parser (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/@mdx-js/mdx/node_modules/remark-parse/lib/index.js:15:12)
    at Function.parse (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/unified/lib/index.js:273:12)
    at executor (file:///Users/sebastienlorber/Desktop/projects/docusaurus/node_modules/unified/lib/index.js:393:31)
    at new Promise (<anonymous>)

I can reproduce on our own website, but can't reproduce on the MDX playground surprisingly 🤷‍♂️ Will need to investigate all this a bit more, but HTML comments like this are definitively supposed to work with the CommonMark format.

First of all, for this file, do you want to use keep using the mdx format (as in v2?) or do you want to switch to commonmark? From now on MDX does not support HTML comments anymore (although we provide a compat mode), so if you want to keep using the same parsing logic you eventually have to use MDX comments instead. I just want to make sure that you understand that using the .md file extension currently switches you to a commonmark mode, as explained in the PR: https://github.com/facebook/docusaurus/pull/8288. I'm going to change that soon and use mdx as a default even for .md files to make it easier to upgrade without renaming file extensions, but for the first alpha the switch is based on the extension, see also https://github.com/facebook/docusaurus/issues/3018#issuecomment-1601260242.

andrewnicols commented 1 year ago

Thanks @slorber,

I suspect that we'll initially stick with .md and commonmark, but I suspect we'll migrate to mdx as default for new content and then I'll have the unenviable task of updating existing content too (currently 860 pages).

I'm still trying to get my head around the various changes we need to make to get this to work. So far this is the only breaking change apart from the expected rewrite of our mdx remark plugins.

slorber commented 1 year ago

Hmmm apparently this is related to a bug in remark-comment, that I already reported 🤪

https://github.com/leebyron/remark-comment/issues/2

slorber commented 1 year ago

Already sent a PR with a fix: https://github.com/leebyron/remark-comment/pull/3 We'll see when it will be merged/published.

In the meantime you can use this escape hatch:

const siteConfig = {
  markdown: {
    mdx1Compat: {
      comments: true
    },
    preprocessor: ({ filePath, fileContent }) => {
      // TODO temporary fix for https://github.com/facebook/docusaurus/issues/9084
      fileContent = fileContent.replaceAll("<!--\n", "<!-- \n");
      return fileContent;
    }
  }
};

To make sure we are able to support multi-line comments, I added some to our own site as part of https://github.com/facebook/docusaurus/pull/9093

facebook / docusaurus