Should we go back to GFM rather than CommonMark? (The general Markdown bug)

[ara4n]

One fix to the irritating bug https://github.com/vector-im/riot-web/issues/4674 is to go back to GFM from CommonMark. This bug exists to track the comparison i'm building up between them:

https://docs.google.com/spreadsheets/d/1pXWhED9CPtQTnQ5QGrCFSXuSi1yyHV9Gd-dNFIUVXIs/edit#gid=0

It follows on from the conversation at https://github.com/vector-im/riot-web/issues/4809#issuecomment-355158896

[ara4n]

Ironically, github have been maintaining the GFM spec and reference implementation as a diff against cmark (the C ref impl of commonmark): https://github.com/commonmark/cmark/compare/master...github:master (thanks to @maxgrenderjones to pointing it out). So one solution could be to 'just' port the same patches commonmark.js, and then we end up with best of both worlds?

[pafcu]

See also https://github.com/vector-im/riot-web/issues/5674

CommonMark spec seems to allow code fences interrupting a paragraph (unlike GFM), but the actual reference implementation seems to have the GFM behaviour.

[dkanada]

I'd like to bump this issue because CommonMark has no support for strikethrough which I consider a very useful tool that is unavailable at the moment. Has there been any movement in this discussion internally or is the idea mostly dead by now?

[jryans]

@ara4n Could we just return to Marked in Riot Web, or would we really want the C reference impl of GFM? The 2017 PR to move to CommonMark (https://github.com/matrix-org/matrix-react-sdk/pull/575) only says that "Marked has some annoying bugs", so it's not clear to me what those were / how annoying they were / if they still exist.

[mpfaff]

CommonMark is missing so many simple but incredibly useful things. Please switch to GFM

[ShadowJonathan]

Do libraries exist that can efficiently transform html back to GFM? That's the biggest issue i see right now, and also where markdown style is erased (e.g. *italics* -> _italics_). So maybe a new format, e.g. m.style.gfm is needed to compensate for this?

This'd make markdown more predictable, instead of janky through it being converted to HTML and back.

[onemorepereira]

This is a major inconvenience. Coming from GitHub, Slack, and other platforms that support simple (I will refrain from saying standard) markdown to Element/Riot is frustrating to say the least.

I suspect it's also a good reason for more casual users to drop it as a client --and therefore being turned off the entire ecosystem.

[AgentOak]

I really think this should have its priority raised, not reduced. The text formatting is the major pain point of Element for most users in my circle. It is generally seen as the worst composer of any messaging client and most significant drawback to using Matrix/Element. Hands down, I would strongly prefer plain text only messaging to the current markdown implementation.

The formatting kicks in at unwanted times (single-item lists), message source gets messed up when editing messages (escaping backslashes are lost, quotes get formatted completely differently), quotes with empty lines just don't work, and the list goes on... And there is no escape from this hell, because there is no option to permanently disable Markdown.

[ShadowJonathan]

I'd like to mention that comrak seems to be a good library to support this feature with, as element is already largely migrating towards rust.

[ara4n]

Thanks to https://github.com/vector-im/element-web/issues/4674 and https://github.com/matrix-org/matrix-react-sdk/pull/7200 we look to be drifting towards a GFM-flavoured dialect of commonmark anyway?

[t3chguy]

I think lack of tilde strikethrough and table syntax remain glaring differences

[ptman]

Missing tables wouldn't be that bad if html is passed through, but I think it's stripped?

[Hugueprime]

Have you considered using markdown-it which has the advantage to have some compatibility with common mark but without the lack of customization ?

Otherwise have you considered doing it our-self to have a product that fully meets element needs ?

[t3chguy]

@Hugueprime that looks to be a JS implementation only, this issue relates to all Element client codebases given they must stay consistent. So needs JS + Kotlin + Swift implementations, or a reusable Rust implementation.

[ell1e]

In https://github.com/vector-im/element-web/issues/14898 I explained why I think using the dialect commonly used in messengers makes more sense, and GFM doesn't seem to implement that either unless I'm mistaken. If that's true, then it seems to me a custom addition, for example using either GFM or CommonMark but with a custom previous pass for some corner cases like for more lenient backtick handling, might be the way to go.

[ShadowJonathan]

From https://github.com/vector-im/element-web/issues/20724: we'd (at the very least) like to be able to use emoticons like >_> or >.< without them being interpreted as block quotes, so something like requiring a space after a > (like discord and other messengers do) would be wanted.

[Johennes]

If I'm not mistaken, both matrix-rust-sdk and matrix-rich-text-editor are using https://github.com/raphlinus/pulldown-cmark which supports some of the GFM extensions over CM:

Further, it optionally supports parsing footnotes, Github flavored tables, Github flavored task lists and strikethrough.

In fact, both Element X iOS and Android support composing GFM strikethrough and tables already. They don't correctly render the tables but the formatted HTML representation is correct.

Meanwhile Element Web remains stuck on its own CM extension. Changing that doesn't seem trivial due to the magic we're doing in https://github.com/matrix-org/matrix-react-sdk/blob/develop/src/Markdown.ts.

However, in general I think there's little point to stick to CM on just one platform and ergonomic strikethrough and tables, while being somewhat exotic, seem like highly desirable features to me.