Feature request: CommonMark Renderer

[pmario]

This issue started at: https://github.com/TiddlyWiki/tiddlywiki/issues/193

@n8sabes posted.

Is there an accurate CommonMark plugin for TiddlyWiki that exactly matches GitHub rendered pages, inclusive of link syntax, inline code and code block formatting?

[pmario]

reference implementation: https://github.com/jgm/commonmark.js

[pmario]

some more reasoning form @ n8sabes

.. Markdown had anomalies making it render differently than GitHub and common plugins for WebStorm and VS Code. As I recall, inline code, code blocks, and a few other things were different enough that it was abandoned.

We install instances of TwiddlyWiki for each project as it's more rich than .md files alone, although VS Code and WebStorm .md plugins are rapidly improving over the last 12 months.

At minimum, it would be nice to have a CommonMark plugin that renders exactly the same as GitHub and IDEs so we can move content between platforms without having to refactor or tweak syntax.

It would be ideal to have an easy way to migrate (or share) content between TiddlyWiki and .md file evironments -- but, that might be too far reaching with the current TiddlyWiki architecture.

[pmario]

We install instances of TwiddlyWiki for each project as it's more rich than .md files alone, although VS Code and WebStorm .md plugins are rapidly improving over the last 12 months.

That's interesting. ... I think the .md community will gravitate towards CommonMark in the future. ... I'm interested.

[n8sabes]

@pmario, thanks for migrating our conversation over.

[sukima]

We install instances of TwiddlyWiki for each project as it's more rich than .md files alone

And this is the very reason I write my personal website in TiddlyWiki. It is much more expressive.

[n8sabes]

@sukima indeed. TiddlyWiki has potential for far greater impact across disciplines. Two areas of low hanging fruit as I see it:

1. Adopt CommonMark to remain relevant, making it easier to migrate knowledge in (or out).
2. Upgrade watcher logic to refresh pages if files update in the background. This simple (or not so simple) change would enable TiddlyWiki to use Dropbox, or even Git as hosting platform, bringing TiddlyWiki on par with commercially hosted wikis.

[danielo515]

Upgrade watcher logic to refresh pages if files update in the background. This simple (or not so simple) change would enable TiddlyWiki to use Dropbox,

Could you explain how such change will help hosting tiddlywiki on Dropbox?

[n8sabes]

@danielo515, TW5 is an amazing platform, but lacks ease of adoption for "groups" of users.

Imagine if TW5 was group accessible without central servers, simply pointing to dropbox folders or git working directories. If TW5 actively watched for underlying tiddler store changes, TW5 becomes a group collaboration solution without the need for central servers. It could be canned as a command line tool, or node app.

Currently, the most common way for groups to use a fully-functional TW5 is to set up a central node server on EC2 for all to access. This greatly restricts adoption to the technically competent.

As most users are non-technical, they choose Dropbox, SkyDrive, or Google Docs to move information around their groups. These solutions have a very low barrier to entry because they are simple to onboard and get value. However, none of these solutions offer the dynamic linking of knowledge like TW5 offers.

TW5 could reach many more users via local instances of TW5 (like I use) that depend upon “Free Replication” and “Share Security” of Dropbox, GoogleDrive, or even Git.

The impediment is TW5’s node server will not refresh pages if the underlying tiddler store is updated behind the scenes, say when the tiddler store is on a Dropbox share. Users must restart the local TW5 node server to pickup the changes. Otherwise, multiple users of a group will overwrite each other’s work.

This line of thought is getting pretty far afield from the intention of the issue (to support CommonMark). If we think a Serverless TW5 is of value, let's spawn another issue and reference back to this thread.

[danielo515]

@n8sabes that is a quite cool idea. Using any replication mechanism as a distributed system sounds very interesting, and a bit geekie if you ask me (which is not bad, of course). But having to raise node command is something that not everyone is ready for. See what is my approach below.

That same lack of synchronization and usability under groups of user was what motivated me to create NoteSelf. It has a very easy on-boarding process ( just visit https://noteself.github.io/online and you are done) and activating the synchronization between devices is as easy as creating an account on cloudant (which only takes two minutes ). I am also creating a tool for easy config of cloudant to allow group of users in NoteSelf. Please take a look at https://noteself.github.io It uses sync mechanism based on your browsers local storage as cache and then syncs to a central server when available. I think that having it accessible online is much more convenient than spawing a command on your own computer.

[n8sabes]

@danielo515 Noteself is a very cool implementation of TW5, and neat idea with PouchDB -- I've always liked PouchDB (Nolan and gang did a nice job, especially with Couch+Couchbase syncing protocol). I would still like a stand-alone disk-base store that can be used via git for a team of developers. I'll play with your noteself more tomorrow and direct anything related over on that project.

CommonMark It would be high value to get CommonMark working with all the momentum building in Github and across IDEs. I'm not the right person to drive CommonMark integration. Is this something of high enough value for someone to take on?

[pmario]

I'm planning to play with markdown-it on the weekend. It seems to be a stable project. .. only 2 open issues !? .. many plugins. ... So we'll see.

[pmario]

Just to tease you a bit. ... prototype seems to be working.

Expect the first alpha late today :)

[danielo515]

Msrkdoen-it has lots of plugins, yes, but it provides you little control over what's being rendered. As a library user all you can do is register plugins and parse markdown. It does not gives you access to the node tree for example, which is a very desirable feature . I don't recommend it for tiddlywiki because the lack of hackhability it has (unless you write a plugin for it )

[pmario]

@danielo515 ... have a closer look at the API docs. It's super flexible.

You can add new tokens and rendering rules if needed. Every instance can use its own settings. So it should be possible to do crazy stuff like the following in a tiddler type: text/vnd.tiddlywiki

```.gfm
github specific markdown comes here
```

```.commonmark
commonmark-down comes here
```

The html sanitizer plugin is a bit too basic. But it should be possible to improve it. ... For my first alpha I'll use the default one.

[pmario]

For v0.0.1 we'll get 4 different dialects.

• gfm
• commonmark
• markdown-it
• zero ... which basically disables everything.

v0.0.2 should get

• pandoc specific markdown elements, if possible. eg:
  • subscript
  • superscript ... and so on

---

edit: added markdown-it dialect

[pmario]

But the cool thing, that I actually want, and that took my interest is: here ... The last paragraph above the Summary

You also can write your own renderer to generate other formats than HTML, such as JSON/XML... You can even use it to generate AST.

Which means 2 things:

1. We can use the token stream and map it to the TW AST. That's how the existing markdown.js implementation works. (Not much use for this project)
2. parse markdown and render it to tiddlywiki syntax instead of html output.

No 2 is the thing, that I really like. ... I don't know yet, how to implement it. ... but we'll see.

But may be

```.gfm
```

is already good enough.

[n8sabes]

I'm no expert on the matter, but does this mean you cannot use language identifiers for syntax highlighting as GFM does with code fences?

[pmario]

v0.0.1 alpha 1: https://wikilabs.github.io/editions/nightly/markdown-it.html

[pmario]

I'm no expert on the matter, but does this mean you cannot use language identifiers for syntax highlighting as GFM does with code fences?

markdown is no syntax highlighting library. ... we'll need a different library eg: highlighter.js. ... but the plugin doesn't detect / use it atm.

[n8sabes]

@pmario, Nicely Done! 😄

As a test, I copied the markup from RSB readme.md into the TW5 alpha test and it rendered super well. The only visual differences with GitHub seems to be in CSS Styling and the :icon: elements. Totally workable!

[pmario]

just as a reference:

• markdown mime/type spec: https://tools.ietf.org/html/rfc7763
• Guidance on Markdown: Design Philosophies, Stability Strategies, and Select Registrations, mentioned in the spec: https://tools.ietf.org/html/rfc7764

[jlazarow]

@pmario Awesome. This solves one of my big gripes about TW --- I should not have to surrender my content format. Having it all in Markdown should allow the use of pandoc and such when exporting.

I was even able to modify your code to have markdown-it-katex integrated. Now one can actually use KaTeX and Markdown.

[pmario]

@jlazarow ... link to your modifications?

[kepano]

@pmario any thoughts on supporting GFM task lists?

[pmario]

@pmario any thoughts on supporting GFM task lists?

IMO this functionality is part of the github UI and not necessarily part of the markup syntax. ...

TW can handle this usecase, in a much more flexible way, out of the box. ... No development needed. See: http://tiddlywiki.com/#TaskManagementExample ... Also doing it the TW way will open up the system to all the possibilities that the "everything is a tiddler" paradigm has.

[kepano]

@pmario I hear what you are saying, though I think this is a case where the Markdown and TW philosophies may be slightly at odds. The syntax for TW task lists is more cumbersome and less portable than GFM, particularly for quick lists that you quickly want to jot down.

[pmario]

The existing plugins https://tiddlywiki.com/plugins/tiddlywiki/markdown/ and https://wikilabs.github.io/editions/markdown-it/ work for me.

[amitnovick]

I'm using @pmario 's commonmark plugin v0.0.3

and Highlight.js plugin v5.1.17

I have some feedback for it based on just few minutes of playing around with it:

1. The syntax highlighting doesn't work for me on text/x-markdown contentType tiddlers.

Variants I tried in $:/config/markdown/variant are: {commonmark, gfm, markdown-it}

Particularly this syntax doesn't get highlighted:

(3 back-ticks)lang
(3 back-ticks)

where lang values I tried are: {javascript, css, html, python}

2. With the markdown-it variant, it seems that :icon: Emojis(?) aren't rendered, particularly I tried the :wink: icon.

My priority

In terms of priority, for me, the code highlighting feature is especially important (I can live without Emojis in my notes).

Why I appreciate this plugin

This back-tick fenced code-block syntax from the CommonMark spec is the chief reason why I'd rather use this plugin over the Gruber/Maruku variants found in the standard plugin.

[jlazarow]

The issue with this plugin is that the markdown-it renderer "overwhelms" every other plugin i.e. the output of this plugin is raw HTML so it serves as the last processing step. That being said, the markdown-it community is pretty good about having markdown-it-* plugins e.g. you need https://github.com/valeriangalliat/markdown-it-highlightjs and then the specific feature will work once added to the markdown-it pipeline.

It's probably not as useful since it's by no means "usable" without extracting the relevant parts, but you can see here that I've heavily modified markdown-it for a variety of purposes... one being adding support for highlghtjs.

https://github.com/jlazarow/TiddlyWiki5/tree/master/plugins/tiddlywiki/markdown-it

I would like to contribute this back at some point once time permits (you might see a variety of hacks within that code that aid my PhD research e.g. PDFJS integration).

[amitnovick]

Thank you for sharing your knowledge :)

[...] you need https://github.com/valeriangalliat/markdown-it-highlightjs and then the specific feature will work once added to the markdown-it pipeline.

So the way to enable Syntax Highligting to work with markdown-it plugin is to integrate @pmario 's markdown-it plugin with markdown-it-highlightjs?

If that's true, then one of the two issues I brought above will be fixed:

1. The syntax highlighting doesn't work for me on text/x-markdown contentType tiddlers.