markedjs / marked

A markdown parser and compiler. Built for speed.
https://marked.js.org
Other
33.27k stars 3.39k forks source link

[Bug] `Heading` token with mutiple end-of-line characters does not tokenized into `Space` token #3513

Open Bistard opened 3 weeks ago

Bistard commented 3 weeks ago

Marked version: 14.1.2

Describe the bug

Heading Token (buggy)

Heading token with mutiple end-of-line characters does not tokenized into Space token:

// '# Heading\n\n'
{type:"heading", raw:"# heading\n\n", depth:1, text:"heading", tokens:[
  {type:"text", raw:"heading", text:"heading"}
]}

But for other block-level tokens, two \n will be tokenized into a Space token:

BlockQuote Token

// '> hello\n\n'
{type:"blockquote", raw:"> hello", tokens:[
  {type:"paragraph", raw:"hello", text:"hello", tokens:[
    {type:"text", raw:"hello", text:"hello"}
]}
], text:"hello"}
{type:"space", raw:"\n\n"}

CodeBlock Token

// '```\nhello\n```\n\n'
{type:"code", raw:"```\nhello\n```", lang:"", text:"hello"}
{type:"space", raw:"\n\n"}

Paragraph Token

// 'paragraph here\n\n'
{type:"paragraph", raw:"paragraph here", text:"paragraph here", tokens:[
  {type:"text", raw:"paragraph here", text:"paragraph here"}
]}
{type:"space", raw:"\n\n"}

List Token

// '* list item\n\n'
{type:"list", raw:"* list item", ordered:false, start:"", loose:false, items:[
  {type:"list_item", raw:"* list item", task:false, checked:undefined, loose:false, text:"list item", tokens:[
    {type:"text", raw:"list item", text:"list item", tokens:[
      {type:"text", raw:"list item", text:"list item"}
]}
]}
]}
{type:"space", raw:"\n\n"}

Expected behavior

Why is the Heading token ending with two (or more) \n not tokenized into a space token? Is this expected?

Thank you for your response.

UziTech commented 3 weeks ago

I don't think there is a reason for it. If you would like to create a PR to make it consistent we could get it into the next major version. 😁👍

Bistard commented 3 weeks ago

OK. In the next few days or weeks, I will look up the source code and try to make it consistent through a PR.