This LUA filter stopped working with the format `commonmark_x`, and is not displayed correctly with the format `markdown_github` on Codebraid

gusbemacbe commented 11 months ago

Hello!

I use the latest version of @jgm's Pandoc;
I use the latest version of @gpoore's Codebraid and Codebraid Preview;
I use mostly the format markdown_github because of supported features.

Here is a small YAML default file code:

filters:
  - assets/filters/lua/list-table.lua
template: "assets/templates/confluence"
variables:
  confluence_html: true
  languages:
    - english
    - español
    - français
    - português

And VSCode's Codebraid settings:

{
  "codebraid.preview.css.overrideDefault": true,
  "codebraid.preview.css.useDefault": true,
  "codebraid.preview.css.useMarkdownPreviewFontSettings": true,
  "codebraid.preview.pandoc.defaultsFile": "confluence.yaml",
  "codebraid.preview.pandoc.executable": "pandoc",
  "codebraid.preview.pandoc.showRaw": true,
  "codebraid.preview.pandoc.showStderr": "never",
  "codebraid.preview.security.allowEmbeddedScripts": true,
  "codebraid.preview.security.allowInlineScripts": true,
  "codebraid.preview.security.allowLocalScripts": true,
  "codebraid.preview.security.allowRemoteFonts": true,
  "codebraid.preview.security.allowRemoteImages": true,
  "codebraid.preview.security.allowRemoteMedia": true,
  "codebraid.preview.security.allowRemoteScripts": true,
  "codebraid.preview.security.allowRemoteStyles": true,
  "codebraid.preview.scrollEditorWithPreview": true,
  "codebraid.preview.scrollPreviewWithEditor": true,
  "codebraid.preview.pandoc.build": 
  {
    "*.md": 
    {
      "reader": "markdown_github",
      "preview": 
      {
        "html": 
        {
          "defaults": {},
          "options": 
          [
            "--preserve-tabs",
            "--standalone",
            "--wrap=auto",
          ]
        }
      },
      "export": {}
    },
  },
}

Here is a small Pandoc template code:

$if(confluence_html)$
<html $if(lang)$lang="$lang$" $endif$>

<head>
    <meta charset="UTF-8" />
    <meta name="generator" content="pandoc" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>$if(title-prefix)$$title-prefix$ - $endif$$pagetitle$</title>
</head>

<body>
    $endif$
    <div class="ids-markdown">
        Testing...
        $body$
    </div>
    $if(confluence_html)$
</body>

</html>
$endif$

I tested the simple list-table code in a Markdown file:

# Lorem ipsum dolor sit amet consectetur adipisicing elit. Fugit, minus?

Lorem ipsum dolor, sit amet consectetur adipisicing elit. Vero, adipisci sequi? Maiores autem quaerat praesentium?

:::list-table
   * - row 1, column 1
     - row 1, column 2
     - row 1, column 3

   * - row 2, column 1
     -
     - row 2, column 3

   * - row 3, column 1
     - row 3, column 2
:::

I tested with the markdown_github format and here is the result. It failed to render the code:

It failed to display the `list-table` code with the `markdown_github` format

I switched to the format commonmark_x and tested it. Here is the error:

Screenshot – Codebraid Preview error with the commonmark_x format

gpoore commented 11 months ago

In the commonmark_x case, the issue is that the Lua filter isn't compatible with the divs that Codebraid Preview inserts into the abstract syntax tree to provide scroll sync. Scroll sync for commonmark_x is different than for all other formats, because Pandoc provides source location information instead of Codebraid Preview having to attempt to reconstruct it. There's a note in the README about adapting filters. Basically, they need to skip elements with a data-pos attribute.

In the markdown_github case, it appears that the issue is that the format doesn't automatically support fenced divs. So you need to specify the input format as markdown_github+fenced_divs.

gusbemacbe commented 11 months ago

In the markdown_github case, it appears that the issue is that the format doesn't automatically support fenced divs. So you need to specify the input format as markdown_github+fenced_divs.

I adjusted the confluence.yaml, updating from to markdown_github+fenced_divs, but Codebraid Preview informed the failure. The problem is probably in the Lua list-table code.

gpoore commented 11 months ago

It seems to be working fine for me. I'd suggest checking the Markdown document between the sh code block and whatever follows. For some reason, this is being treated as code instead of regular Markdown. There might be some sort of syntax error in the document. Maybe the * in the list are indented too far.

gusbemacbe commented 11 months ago

It seems to be working fine for me. I'd suggest checking the Markdown document between the sh code block and whatever follows. For some reason, this is being treated as code instead of regular Markdown. There might be some sort of syntax error in the document. Maybe the * in the list are indented too far.

I removed the sh part;
I fixed the * indentation;

But still the same error. I investigated the table indentation. I fixed, using VSCode's Ctrl + [, and it worked now:

:::list-table
Markup languages

* - Name
  - Initial release

* - Markdown
  - 2004

* - reStructuredText
  - 2002
:::

in comparison to the buggy code copied from this repository's README.md:

:::list-table
   Markup languages

   * - Name
     - Initial release

   * - Markdown
     - 2004

   * - reStructuredText
     - 2002
:::

gusbemacbe commented 11 months ago

Only the complicated tables do not work.

gusbemacbe commented 11 months ago

Ah, I reviewed my old Codebraid projects, I copied all the missing features for markdown_github variant format and all the complicated tables worked now.

The @pandoc-ext personnel to document the required features for any list-table table to work. They need to fix the table indentation bug.

gusbemacbe commented 11 months ago

Here are the missing features:

from: markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum

wlupton commented 11 months ago

As far as I can see, the example shown below is valid and produces the same AST, and therefore the same output, as the non-indented version (note that the indent is three spaces, which is explicitly permitted by commonmark but perhaps not explicitly called out for pandoc markdown).

Please could you supply the full command line that shows the problem with the example below? Thanks.

:::list-table Markup languages

- Name
  - Initial release
- Markdown
  - 2004
- reStructuredText
  - 2002 :::

gusbemacbe commented 11 months ago

@wlupton

Please could you supply the full command line that shows the problem with the example below? Thanks.

With 3 indentation spaces:

Using the markdown_github with enabled on Codebraid Preview, it failed.
Using the full command pandoc --preserve-tabs --standalone --wrap=auto --defaults "confluence.yaml" --from markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum --to html "README.md", it also failed.

Switching to commonmark_x.

Using the Codebraid Preview, and disabling the scroll settings, it failed.
Using the full command pandoc --preserve-tabs --standalone --wrap=auto --defaults "confluence.yaml" --from commonmark_x+bracketed_spans+fenced_divs+raw_attribute+raw_html --to html "README.md" -o "result.html", it worked.

wlupton commented 11 months ago

Thanks for that @gusbemacbe (I realize that I should have been able to extract this info from the existing discussion; sorry about that).

I think @gpoore's analysis is spot on, and that the main thing is that you need to enable the fenced_divs extension. With this input:

% cat spiz.md
:::list-table
- - zero-indented
:::

:::list-table
   - - three-indented
:::

...this works for me:

% pandoc spiz.md -f markdown_github+fenced_divs -L list-table.lua 
[WARNING] Deprecated: markdown_github. Use gfm instead.
<table>
<thead>
<tr class="header">
<th>zero-indented</th>
</tr>
</thead>

</table>
<table>
<thead>
<tr class="header">
<th>three-indented</th>
</tr>
</thead>

</table>

So why do you get the original codebraid error? I suspect (from the output shown below) that the indentation has somehow been changed. Is it possible to show precisely the markdown that's being passed to pandoc?

As for possible updates to the filter, I think that we could/should emphasise (in the documentation) that the fenced_div extension has to be enabled. Also, I suppose that if any of the supported pandoc markdown variants don't support 3-space indentation (are there any that don't?) then we could remove it from the examples. I also think that the filter should be adapted to work with codebraid (I can create a separate issue for that).

gusbemacbe commented 11 months ago

Here is the full command given by Codebraid:

pandoc "README.md"
    --css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/%40vscode/codicons/dist/codicon.css" \ 
    --css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/vscode-markdown.css" \
    --css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/codebraid-preview.css" \
    --preserve-tabs --standalone --wrap=auto \ 
    --defaults "$HOME/Tests/confluence/_codebraid/temp_defaults/_codebraid_preview.yaml" \
    --lua-filter="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/sourcepos_sync.lua" \
    --lua-filter="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/show_raw.lua" \
    --katex="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/katex/dist/" \ 
    --from markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum \
    --to html

In the case of defaults, replace the path with where the defaults file is located. Copy my defaults code:

from: markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum
default-image-extension: '.png'
extract-media: dir
metadata:
  author: "Gustavo Costa"
  date: '2023-12-29'
  keywords:
    - codebraid
    - css
    - guide
    - html
    - markdown
    - markdown_mmd
    - multimarkdown
    - pandoc
  lang: "pt-PT"
  title: "Codebraid Pandoc Markdowm Guide"
  toc-title: "Table of Guides"
  version: "1.0.0"
preserve-tabs: true
shift-heading-level-by: -1
tab-stop: 2
trace: true
track-changes: accept

# General writer options
filters:
  - "list-table.lua"
highlight-style: kate
no-check-certificate: true
request-headers:
  - ["User-Agent", "Mozilla/5.0"]
strip-comments: false
toc: true
toc-depth: 7
variables:
  confluence_html: true
  languages:
    - english
    - español
    - français
    - português

# Options affecting specific writers
ascii: false
email-obfuscation: references
html-q-tags: true
incremental: true
listings: true
markdown-headings: atx
number-sections: true
reference-links: true
reference-location: block
section-divs: true
self-contained: false
slide-level: 2
top-level-division: chapter

# Citation rendering
cite-method: biblatex
citeproc: true

# Math rendering in HTML
html-math-method:
  method: katex

wlupton commented 11 months ago

After some investigation, I think that the problem is that the tab stop has been set to 2. Consider this example:

% pandoc <<<'  two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>

Compare with this:

% pandoc <<<'  two-indent' --tab-stop=4
<p>two-indent</p>

The CommonMark parser doesn't behave this way:

% pandoc <<<'  two-indent' --tab-stop=2 --from commonmark
<p>two-indent</p>

I don't see any suggestion in the manual that --tab-stop should affect indented code block indentation, so is this a bug in the markdown reader?

gusbemacbe commented 11 months ago

I made another test, but different, with both --tab-stop=2 and --tab-size=4, on Codebraid Preview and using markdown_github and other variants:

* testing... 

  :::list-table
  * - Name
    - Description
    - Origin

  * - Apple
    - Fruit
    - China

  * - Banana
    - Fruit
    - India
  :::

It worked with all the variants, except CommonMark, CommonMark X and gfm, on Codebraid Preview.

If I tab-indented the table part, it stopped working with all the variants, including CommonMark and CommonMark X, on Codebraid Preview.

wlupton commented 10 months ago

@jgm, I'm not sure whether you saw this. Would you expect this behavior (see the last-but-one comment for more context)? Thanks.

% pandoc <<<'  two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>

jgm commented 10 months ago

Yes, that is expected. John Gruber's original Markdown documentation says:

To produce a code block in Markdown, simply indent every line of the block by at least 4 spaces or 1 tab.

IF we change a tab from 4 to 2 spaces, then 2 spaces will do it.

wlupton commented 10 months ago

Thanks. I'd expect this if there was a tab in the input, but there are two spaces, not a tab. FWIW the CommonMark parser doesn't treat this as a code block.

jgm commented 10 months ago

commonmark just hard-codes 4 spaces tab stop, which I think is the best plan. --tab-stop is sort of a vestigial feature -- I doubt many people use it. What are you using it for?

gusbemacbe commented 10 months ago

I do not know who you are addressing to, @jgm.

As for me, to work on Codebraid Preview, I would prefer using markdown_github and any variant, except commonmark_x and gfm.

Without depending on the Pandoc Lua filter list-table, any Markdown variant works on both Codebraid Preview and the terminal.

With the dependence of the Pandoc Lua filter list-table, and with 2 or 4 tab stop spaces, commonmark_x and gfm variants do not work on Codebraid Preview, but work on the terminal. The markdown_github, markdown_mmd and markdown_phpextra variants work on Codebraid Preview and on the terminal, but only if with the 2 tab stop spaces, and with the fenced_divs and simple_tables are enabled.

However, markdown_github has more features than commonmark_x and gfm, but I reconsider switching to markdown_mmd and markdown_phpextra, but I want to stop Pandoc from wrapping a p element in every line. @mb21 recommends me to use gfm, that stops Pandoc from wrapping p element in every line, but gfm will not work on Codebraid Preview with the dependence of the Pandoc Lua filter list-table.

gusbemacbe commented 10 months ago

Well. I have tabe-stop in the defaults file:

preserve-tabs: true
shift-heading-level-by: -1
tab-stop: 2

Do you think that defaults file's tab-stop can be the cause of these bugs?

wlupton commented 10 months ago

I definitely think you should try removing it, or changing it to 4!

@jgm, sorry for dragging you into this discussion. I'll create a pandoc issue to discuss the point that I raised with you.

gusbemacbe commented 10 months ago

Good evening, @wlupton

I have just made this test:

With the removal of tab-stop from the defaults file or with the modification of tab-stop's value to 4, it worked on Codebraid Preview with the Markdown variant markdown_github, but not with the other variants commommark_x and gfm. I think it turns to be the bug of tab-stop.

However, I want to keep 2 tab stop spaces for the HTML elements when converting from Markdown to HTML. But it will not work with this Pandoc Lua filter on Codebraid Preview.

wlupton commented 10 months ago

I hoped that this example from a couple of days ago would make it clear what's going on here. With a tab-stop of 2, indentation of 2 or more will be formatted as code.

% pandoc <<<'  two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>

If you want to use tab-stop=2 then you'll need to change your input file, because 3 > 2! BUT, are you sure that the tab-stop setting affects the generated HTML? I don't see why it would do!

I suggest using very simple input and defaults files (in fact, avoid using a defaults file if you can), and also making sure that it works as expected from the command-line before trying it with Codebraid.

Perhaps this example will make it clearer.

% cat spiz.md
:::list-table
   * - 3
:::

:::list-table
 * - 1
:::

% pandoc spiz.md --tab-stop=2
<div class="list-table">
<pre><code> * - 3</code></pre>
</div>
<div class="list-table">
<ul>
<li><ul>
<li>1</li>
</ul></li>
</ul>
</div>

I'm not using the list-table.lua filter here because I don't need to! It will fail, because it'll see a code block where it expects a list.

Let's assume that you have resolved the tab-stop problem, either by using tab-stop=4 (which is what I would recommend) or by editing the markdown.

Now you can try it with codebraid. You'll have (for now) to live with the restrictions explained here by @gpoore, which mean that it won't work with commonmark variants (commonmark, gfm and commonmark_x).

As I mentioned, I'd be happy to make list-table.lua work with codebraid's line numbering, but that should happen on a separate issue. I'll create this other issue tomorrow.

Does this all make sense?

gusbemacbe commented 10 months ago

Does this all make sense?

Yes, this does, @wlupton.

wlupton commented 10 months ago

@gusbemacbe please could you try the feature/support-sourcepos branch with commonmark_x. Thanks.

gusbemacbe commented 10 months ago

Good morning, @wlupton!

I have just tested with both

from: commonmark_x;
from: commonmark_x-ascii_identifiers-east_asian_line_breaks-implicit_figures-pipe_tables-smart-sourcepos-wikilinks_title_after_pipe-wikilinks_title_before_pipe+attributes+autolink_bare_uris+bracketed_spans+definition_lists+emoji+fancy_lists+fenced_divs+footnotes+gfm_auto_identifiers+hard_line_breaks+implicit_header_references+raw_attribute+raw_html+rebase_relative_paths+strikeout+subscript+superscript+task_lists+tex_math_dollars+yaml_metadata_block (@gpoore's Codebraid Preview's Lua filter says not to support alerts and tex_math_gfm);

It worked perfectly and fully with Codebraid Preview with enabled and disabled scrolling, except the third table:

Only - []{rowspan=2} `italic` doesn't work because it is not spanned with two rows.

And the last table failed to work with Codebraid Preview:

:::{.list-table}
- []{style="background: lightblue"}
  - Ocean
  - Area (sq. miles)
- - Atlantic
  - 41.1 million
- - Pacific
  - 63.8 million
:::

:::{.list-table style="background: lightsalmon"}
* - Fish
  - Ocean

* :::{.list-table-body style="background: lightgreen"}
  - - Arctic char
    - Arctic
  - - Humuhumunukunukuāpuaʻa
    - Pacific
  :::

* ------------- ---------
  Cod           Atlantic
  Notothenioids Antarctic
  ------------- ---------
:::

Here is the error:

Codebraid Preview Error
pandoc failed:
Error: Command failed: pandoc --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/%40vscode/codicons/dist/codicon.css --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/vscode-markdown.css --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/codebraid-preview.css --defaults "/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/defaults/JCaZoD3OG8INzm4MPjk-TFkkqtepMQqgUZeBa0McErk.yaml" --standalone --wrap=auto --defaults "/home/gus/Tests/confluence/_codebraid/temp_defaults/_codebraid_preview.yaml" --standalone --lua-filter="/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/sourcepos_sync.lua" --katex=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/katex/dist/ --lua-filter="/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/show_raw.lua" --from "/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/readers/commonmark_x.lua-ascii_identifiers-east_asian_line_breaks-implicit_figures-pipe_tables-smart-sourcepos-wikilinks_title_after_pipe-wikilinks_title_before_pipe+attributes+autolink_bare_uris+bracketed_spans+definition_lists+emoji+fancy_lists+fenced_divs+footnotes+gfm_auto_identifiers+hard_line_breaks+implicit_header_references+raw_attribute+raw_html+rebase_relative_paths+strikeout+subscript+superscript+task_lists+tex_math_dollars+yaml_metadata_block+sourcepos" --to html
Error running filter assets/filters/lua/list-table-sourcepos.lua:
assets/filters/lua/list-table-sourcepos.lua:169: expected list item to have row attrs  at
------------------------------------------------------------------------

stack traceback:
    assets/filters/lua/list-table-sourcepos.lua:51: in upvalue 'assert_'
    assets/filters/lua/list-table-sourcepos.lua:169: in function <assets/filters/lua/list-table-sourcepos.lua:134>

gusbemacbe commented 10 months ago

With markdown_github and with the updated list-table filter, the spanned rows worked:

I suspend CommonMark X does not support it.

wlupton commented 10 months ago

Thanks. The rowspan problem was my fault (I hadn't skipped the data-pos Divs in this case). It should now be fixed.

The other problem was that the table shown below is a "simple table", but commonmark_x only supports pipe tables, so I've changed this README.md example to use a pipe table.

* ------------- ---------
  Cod           Atlantic
  Notothenioids Antarctic
  ------------- ---------

I believe that all the examples are now working in Codebraid with commonmark_x. Do you agree?

gusbemacbe commented 10 months ago

I believe that all the examples are now working in Codebraid with commonmark_x. Do you agree?

It almost worked, but it does not solve the colspan or rowspan problem.
I enabled pipe_tables and it still failed:

wlupton commented 10 months ago

Please double-check that you have the latest filter code. I believe that I have commented out the test that would lead to that error.

gusbemacbe commented 10 months ago

I have just tested the latest list-table codes with the latest Lua filter code on Codebraid Preview with the commonmark_x+pipe_tables and it worked fully and perfectly!

wlupton commented 10 months ago

Great! Note that you shouldn't need the +pipe_tables (commonmark_x already enables pipe tables).

pandoc-ext / list-table

This LUA filter stopped working with the format `commonmark_x`, and is not displayed correctly with the format `markdown_github` on Codebraid #4