Compatibility with standard Markdown: a more serious look

[tlienart]

Ultimately, JuDoc uses Julia's Markdown to HTML conversion; this is mostly fine for "standard" things like basic text formatting, adding paragraphs, etc. Now after #201 it seems a more serious look would be required so that people who are used to "standard Markdown" as used with Jekyll or other popular engines don't get too surprised.

This issue is to list things that should be checked/tested/accomodated and will link to sub-issues for specific problems. It might take some time to fix/accomodate/document everything so probably a good idea to keep track of things in one place.

Markdown specs refs

• Gruber
• Pritchard

High-priority

• [x] code: empty line followed by lines that start with four spaces or a tab followed by empty line = block of code see #207
• [x] indirect links: where you use text [link name] text and later [link name]: url, these are currently not processed this is also mentioned in Base: https://github.com/JuliaLang/julia/issues/19844 see PR #214
• [x] multiple backticks: allow the use of multiple backticks rather than just one and three, in particular should probably allow two as similar to one to allow things like ` and five as similar to triple (?)

Low(er)-priority

• [x] HTML entities, some people seem used to writing things like π at the moment these will show exactly like that with the ampersand and all instead of π; note however that if one writes AT&T then this will be (properly) converted to AT&T; it should be clarified that \$ is acceptable to introduce a dollar, & will be translated to & etc; one way could be to just look for the regex &([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6}); and fence it with ~~~. Should probably keep a list of things that can be escaped with backslash (e.g. dollar sign, dot, * there's a list at the bottom of gruber's specs, not all work (e.g. backtick)). ==> for backslash and escaping see #205 ; for html entities see #206 Now done: see PR #209
• [x] headers, in some standards, headers are underlined with === or --- instead of preceded with a number of #; probably can automatically change those. Exec decision: will not support for now bc problems with TOC generation, ## seems reasonable enough
• [x] lists, it seems that in some standards something like what follows, will be put in a list even though the blocks are on multiple lines: I think the convention to keep everything on the same line is sensible so probably won't change (note that it can be achieved by indenting things) Executive decision: will not support for now; one way or the other it can be confusing, probably best to just document see #213
• [x] link naming, in Gruber's specs something like this [Yahoo](http://search.yahoo.com/ "Yahoo Search") is allowed I'm not convinced we should strive to support this. Executive decision: will not support for now until someone explicitly asks for it.

Quirks / brainstorm

• quoting a code block will not work well; some of the > will be capture and shown in the code block
• inserting a horizontal line; while ***, ***** and -------------- will work, * * * or - - - will not

[cormullion]

What do you make of this type of link? (from Jeff & Co's blog post)

The [free lunch][] from Moore's Law is still over.

[free lunch]: http://www.gotw.ca/publications/concurrency-ddj.htm

Not sure whether Julia Markdown does these reference links...

[tlienart]

Yes I guess these would be part of what I call "indirect links", it seems that either

The [free lunch] from ...
...
[free lunch]: http://...

or

The [free lunch][] from ...
...
[free lunch]: http://...

are used; the former is used in Stefan's piping one and the latter in the post you quote, so I guess I'll have too find both ;-)

(that's not too hard though, but yeah I'm slowly using less and less of Julia's markdown parser and more and more of a handcrafted one 🙈 )

[cormullion]

Good stuff!. Could any of these changes be introduced into the Julia Markdown parser - making it more standard-compliant (if that's the right word to be used in a sentence about Markdown)?

[tlienart]

yeah I guess ultimately that makes sense though it's a bit tricky since in base there are multiple "flavours" (GitHub, Julia, ...); I guess the "Julia" one could be extended to match a bit more of Gruber's specs. But when I feel like I've satisfied the Markdown gods and have a good understanding of what's missing etc, then I may make suggestions to the mother ship

[tlienart]

@cormullion the different ways for "indirect" links definitions are now supported in PR #214 (I need to think a bit and test it a bit more before merging but it should be close to good to go). It's a bit trickier than what I initially thought actually for a simple reason: Judoc effectively slices the document into smaller blocks it can process very rapidly and then pieces everything back together.

But here, the link definition could be anywhere in the document; so this needs to be done after the piecing-back-together; as a result my solution is not very elegant (it's done after the HTML is generated); by the way I also had to resort to similar trickery to handle "latex-like" bibliographies; it works well but it's a bit ugly.

All this being said, issues about markdown syntax have been opened in Julia base (see links in the post at the top); I'll be glad to try to help there once I've better understood how Julia's Markdown parser works but solving it there would not solve it here for the reason mentioned above 🙂

[tlienart]

with #217, it's the (temporary?) end of this; one day I'll muster the courage to read through the commonmark specs and see what fun edge cases are still lurking.

I'll close this now and will report once I've tried to use JuDoc with JuliaBlog posts as per #201 in that issue.