search

mmarkdown / mmark

Mmark: a powerful markdown processor in Go geared towards the IETF
https://mmark.miek.nl
Other
480 stars 45 forks source link

Request: Make note of newline character sensitivity? #176

Closed paragonie-security closed 2 years ago

paragonie-security commented 2 years ago

To help us debug your issue please add these details.

While trying to update the PASETO RFC draft, I kept getting this error:

$ make
mmark paseto.md > draft-paragon-paseto-rfc-02.xml
xml2rfc --v2 --html draft-paragon-paseto-rfc-02.xml
Error: Unable to parse the XML document: draft-paragon-paseto-rfc-02.xml
 /home/robyn/projects/paseto-rfc/draft-paragon-paseto-rfc-02.xml: Line 45: Extra content at the end of the document

After a lot of wrong guesses, I found out the issue: I'm using a VirtualBox shared folder. My host OS is Windows, which checked out newlines as CRLF rather than just LF. This caused the title block to not be parsed, which caused invalid XML to be generated. This caused a downstream failure with xml2rfc.

What went wrong?

What version of mmark are you using?

Version 2,2,25.

What is the most minimal markdown snippets that shows the problem?

Take a valid .md file from your test files, run unix2dos on that file, then pass it to the mmark program.

Recommendation

One of the following:

miekg commented 2 years ago

Odd, https://github.com/mmarkdown/mmark/blob/master/mmark.go#L76 should normalize those newlines

On Mon, 23 May 2022, 22:43 P.I.E. Security Team, @.***> wrote:

To help us debug your issue please add these details.

While trying to update the PASETO RFC draft, I kept getting this error:

$ make mmark paseto.md > draft-paragon-paseto-rfc-02.xml xml2rfc --v2 --html draft-paragon-paseto-rfc-02.xml Error: Unable to parse the XML document: draft-paragon-paseto-rfc-02.xml /home/robyn/projects/paseto-rfc/draft-paragon-paseto-rfc-02.xml: Line 45: Extra content at the end of the document

After a lot of wrong guesses, I found out the issue: I'm using a VirtualBox shared folder. My host OS is Windows, which checked out newlines as CRLF rather than just LF. This caused the title block to not be parsed, which caused invalid XML to be generated. This caused a downstream failure with xml2rfc. What went wrong? What version of mmark are you using?

Version 2,2,25. What is the most minimal markdown snippets that shows the problem?

Take a valid .md file from your test files, run unix2dos on that file, then pass it to the mmark program. Recommendation

One of the following:

  • Support CRLF instead of just LF, or
  • Clearly document this behavior so other people will know to normalize their line endings

— Reply to this email directly, view it on GitHub https://github.com/mmarkdown/mmark/issues/176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACWIW2CORMCQJWVVVSJPYTVLPUXDANCNFSM5WXAO5CA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

miekg commented 2 years ago 
% unix2dos 1035.md
unix2dos: converting file 1035.md to DOS format...
% make 1035.xml
../mmark 1035.md > 1035.xml
% make 1035.txt
xml2rfc --text --v3 1035.xml
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2397): Warning: Unused reference: There seems to be no reference to [IEN116] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2407): Warning: Unused reference: There seems to be no reference to [Quarterman86] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0742] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0799] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0805] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0810] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0811] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0812] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0819] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0821] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0830] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0883] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0920] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0952] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0953] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC0973] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC1001] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC1002] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC1031] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC1032] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2396): Warning: Unused reference: There seems to be no reference to [RFC1033] in the document
/home/miek/src/github.com/mmarkdown/mmark/rfc/1035.xml(2441): Warning: Unused reference: There seems to be no reference to [Solomon82] in the document
 Created file 1035.txt
rm -f 1035.xml

And

% head -20 1035.txt  

Internet Engineering Task Force (IETF)                    P. Mockapetris
Request for Comments: 1035                                           ISI
Obsoletes: 882, 883, 973                                     24 May 2022
Category: Standards Track                                               
ISSN: 2070-1721

            Domain Names - Implementation and Specification

Abstract

   This RFC describes the details of the domain system and protocol, and
   assumes that the reader is familiar with the concepts discussed in a
   companion RFC, "Domain Names - Concepts and Facilities" [RFC1034].

   The domain system is a mixture of functions and data types which are
paragonie-security commented 2 years ago

Interesting. I'm not sure what caused this specific failure, but we know on our end how to fix it.

Virtualbox shared folders and inconsistent behavior with Git + Jetbrains IDEs over the years might've created a weird state, which was fixed when I ran dos2unix.