search

accordproject / markdown-transform

Parse and transform markdown text, including TemplateMark markdown templates
Apache License 2.0
71 stars 49 forks source link

HTML Transform: Erronous HTML Table Parsing #582

Open jetlime opened 8 months ago

jetlime commented 8 months ago

Bug Report 🐛

Whenever a html table is defined with a caption, the transformation to Markdown yields to an invalid md table.

Expected Behavior

The following html table,

<table>
<caption>Average monthly active recipients of the service, in the EU region over prior 6 months (est.)
</caption>
<tbody><tr>
<th>
</th>
<th>Aug. 2022 - Jan. 2023
</th>
<th>Feb. 2023 - July 2023
</th></tr>
<tr>
<td>Wikibooks
</td>
<td>6,919,000
</td>
<td>1,611,000
</td></tr>
<tr>
<td>Wikidata
</td>
<td>1,056,000
</td>
<td>1,051,000
</td></tr>
<tr>
<td>Wikimedia Commons
</td>
<td>2,845,000
</td>
<td>3,272,000
</td></tr>
<tr>
<td>Wikinews
</td>
<td>6,283,000
</td>
<td>1,035,000
</td></tr>
<tr>
<td>Wikipedia
</td>
<td><b>151,556,000</b>
</td>
<td><b>151,088,000</b>
</td></tr>
<tr>
<td>Wikiquote
</td>
<td>6,811,000
</td>
<td>1,548,000
</td></tr>
<tr>
<td>Wikisource
</td>
<td>7,106,000
</td>
<td>1,845,000
</td></tr>
<tr>
<td>Wikispecies
</td>
<td>29,000
</td>
<td>37,000
</td></tr>
<tr>
<td>Wikiversity
</td>
<td>6,360,000
</td>
<td>1,082,000
</td></tr>
<tr>
<td>Wikivoyage
</td>
<td>616,000
</td>
<td>632,000
</td></tr>
<tr>
<td>Wiktionary
</td>
<td>8,955,000
</td>
<td>8,425,000
</td></tr>
<tr>
<td><i><span style="color: gray; white-space: pre-wrap">Est. devices per person</span></i>
</td>
<td>2.4<sup id="cite_ref-Cisco_1-0" class="reference"><a href="#cite_note-Cisco-1">&#91;1&#93;</a></sup>
</td>
<td>2.4<sup id="cite_ref-Cisco_1-1" class="reference"><a href="#cite_note-Cisco-1">&#91;1&#93;</a></sup>
</td></tr></tbody></table>

Shall be parsed in the following valid markdown, 

|     |     |     |
| --- | --- | --- |  
Average monthly active recipients of the service, in the EU region over prior 6 months (est.)
|     | Aug. 2022 - Jan. 2023 | Feb. 2023 - July 2023 |
| Wikibooks | 6,919,000 | 1,611,000 |
| Wikidata | 1,056,000 | 1,051,000 |
| Wikimedia Commons | 2,845,000 | 3,272,000 |
| Wikinews | 6,283,000 | 1,035,000 |
| Wikipedia | **151,556,000** | **151,088,000** |
| Wikiquote | 6,811,000 | 1,548,000 |
| Wikisource | 7,106,000 | 1,845,000 |
| Wikispecies | 29,000 | 37,000 |
| Wikiversity | 6,360,000 | 1,082,000 |
| Wikivoyage | 616,000 | 632,000 |
| Wiktionary | 8,955,000 | 8,425,000 |
|     | 2.4[\[1\]](#cite_note-Cisco-1) | 2.4[\[1\]](#cite_note-Cisco-1) |

Which parses into a valid Markdown table:

Average monthly active recipients of the service, in the EU region over prior 6 months (est.) | | Aug. 2022 - Jan. 2023 | Feb. 2023 - July 2023 | | Wikibooks | 6,919,000 | 1,611,000 | | Wikidata | 1,056,000 | 1,051,000 | | Wikimedia Commons | 2,845,000 | 3,272,000 | | Wikinews | 6,283,000 | 1,035,000 | | Wikipedia | 151,556,000 | 151,088,000 | | Wikiquote | 6,811,000 | 1,548,000 | | Wikisource | 7,106,000 | 1,845,000 | | Wikispecies | 29,000 | 37,000 | | Wikiversity | 6,360,000 | 1,082,000 | | Wikivoyage | 616,000 | 632,000 | | Wiktionary | 8,955,000 | 8,425,000 | | | 2.4[1] | 2.4[1] |

Current Behavior

Given the previous html table, including a caption, the tool transform the html into the following markdown content,

Average monthly active recipients of the service, in the EU region over prior 6 months (est.)
|  | Aug. 2022 - Jan. 2023
 | Feb. 2023 - July 2023
 |
| Wikibooks
 | 6,919,000
 | 1,611,000
 |
| Wikidata
 | 1,056,000
 | 1,051,000
 |
| Wikimedia Commons
 | 2,845,000
 | 3,272,000
 |
| Wikinews
 | 6,283,000
 | 1,035,000
 |
| Wikipedia
 | 151,556,000 | 151,088,000 |
| Wikiquote
 | 6,811,000
 | 1,548,000
 |
| Wikisource
 | 7,106,000
 | 1,845,000
 |
| Wikispecies
 | 29,000
 | 37,000
 |
| Wikiversity
 | 6,360,000
 | 1,082,000
 |
| Wikivoyage
 | 616,000
 | 632,000
 |
| Wiktionary
 | 8,955,000
 | 8,425,000
 |
| Est. devices per person | 2.4[[1]](#cite_note-Cisco-1 "") | 2.4[[1]](#cite_note-Cisco-1 "") |

Which is an invalid md table:

Average monthly active recipients of the service, in the EU region over prior 6 months (est.) Aug. 2022 - Jan. 2023 Feb. 2023 - July 2023
Wikibooks
6,919,000
1,611,000
Wikidata
1,056,000
1,051,000
Wikimedia Commons
2,845,000
3,272,000
Wikinews
6,283,000
1,035,000
Wikipedia
151,556,000 151,088,000
Wikiquote
6,811,000
1,548,000
Wikisource
7,106,000
1,845,000
Wikispecies
29,000
37,000
Wikiversity
6,360,000
1,082,000
Wikivoyage
616,000
632,000
Wiktionary
8,955,000
8,425,000
Est. devices per person 2.4[1] 2.4[1]

Steps to Reproduce

  1. npm install -g @accordproject/markdown-cli
  2. wget https://foundation.wikimedia.org/wiki/Legal:EU_DSA_Userbase_Statistics --output-file test.html
  3. markus transform --from html --to markdown --input test.html --output test.md
  4. Open test.md using a md parser to visiualise the invalid table parsing.

Context (Environment)

Parsing HTML to Markdown for web archiving.

Desktop

dselman commented 8 months ago

Thank you got the clear issue report. Very helpful.

It appears that the is no agreed way to represent captions for markdown tables. Given that we use markdown-it for parsing markdown, this would appear to be the best option:

https://github.com/martinring/markdown-it-table-captions