Add support for Markdown MyST

[lrq3000]

⚠️ This issue respects the following points: ⚠️

• [X] This is a enhancement/feature request. Not a bug or question.
• [X] The topic is not already reported at Issues. (I've searched it).
• [X] Markor is up to date. See Releases for the latest version. Updates are available from F-Droid and GitHub.
• [X] The wanted feature/enhancement is not present in the latest development version (git master). (Please download and try the test version of Markor, named Marder. Don't worry; Markor and Marder appear as completely separate applications. You can install both side-by-side, and Markor's settings are not touched. If your desired feature is present, you don't need to open this issue. The change will be part of the next Markor update.)

Description

Markedly Structured Text (MyST) is a superset of CommonMark Markdown markup language enriched with ReStructured Text (RST) directives system, so that MyST supports advanced structuration directives (including custom ones) that allow for the writing of scientific manuscripts and books. For example, it supports citations and bibtex, whereas Markdown does not.

MyST can be seen as an alternative to AsciiDoc, but whereas AsciiDoc is mainly compatible with the Ruby ecosystem, MyST is mainly compatible with the scientific Python ecosystem (ie, can integrate with Jupyter Lab). Nevertheless, its standard implementation is in Javascript (MyST.js), so that it does not require Python, contrary to AsciiDoc.

TODO:

• [x] Add support in edit mode: MyST files are Markdown files, with a similar base syntax and filenames being .md, so editing MyST files is supported by Markor out of the box.
• [ ] Add MyST syntax highlighting in edit mode: MyST is a superset of Markdown CommonMark, so Markor currently correctly highlights the base syntax of MyST, but not advanced ones, especially specific directives and roles. But since they have the same syntax all the time, it should need only a simple if-elif/switch statement to cover all specific cases, and then a generic else to cover all other cases of custom/unrecognized directives and roles.
• [ ] MyST View-Mode support using MyST.js : MyST.js is the standard implementation to process MyST files into a rendered document. Hence, we could use a similar approach to PR #2022 by using TextConverter to call MyST.js to render a MyST document.

IMHO a minimal implementation of MyST in Markor should implement the View-Mode support. Syntax highlighting can be implemented later, because anyway there is already partial syntax highlighting that is enough for edition, but what matters is that we should be able to view the rendered document and especially citations.

Information

Android version: 11 Device: Huawei P40 Pro App Version: latest stable

Source

Google Play

Format / File type

Something else (-> Additional info)

Additional info / Log

-

[harshad1]

Note: Currently it is a bit painful to add a new format. I have plans to simplify this - move all the functionality into the Format Registry instead of spread out in 10 different files.

[lrq3000]

@harshad1 Could you provide me some pointers please about which files should be modified? Maybe another more complete PR than the one I linked to, because it seems the AsciiDoc PR only implements partial support?

[gsantner]

View-Mode is gonna be more difficult. Markdown is converted at Java side to Markdown. What you want to do is convert by JavaScript, thus requiring the unmodified text on javascript side.

Though, is it really something that many people use? I don't feel like it's a good idea to support someone's todays favorite reinvented brew of Markdown and tomorrow another. CommonMark/GFM/GitLab-Markdown is what the parser supports. I think that should be more then enough. Don't get me wrong and it shouldn't be rude - but hope you get the point what I mean.

At the end of the day, also there needs to be somebody to maintain everything and unfortunately it is rather the exception that people keep contributing after "making it once". Adding something is only the first mile - it needs to be maintained as long it's part of the software.

Instead what I suggest to be a better idea is to support i.e. ReStructured Text on it's own.

[lrq3000]

@gsantner Thank you for your suggestions.

Yes MyST is quite a new standard compared to other more established ones, but it is supported by an active and competent team with tight links with the Jupyter devs. I do not know them personally, but I know the works of one dev (rowanc1) who is a prolific developer of explorable explanations/interactive scientific visualization frameworks since several years.

Anyway, I propose MyST because there are to my knowledge only 2 light markup languages that provide sufficient support of features for scientific publications, especially citations: AsciiDoc, and MyST. And for AsciiDoc, it's not even totally correct, as a plugin is required to support bibtex files, whereas they are natively supported in MyST. Hence, comparing with CommonMark/GFM/GitLab-Markdown is like comparing apples and oranges, those are great for note taking, but very poor for scientific works.

If the issue is maintenance, if I can succeed in implementing it, you can rest assured that I will keep maintaining it for at least a few years but likely much more, as I intend to use MyST as my primary format for scientific works. You can look at my history to see that I am a maintainer of lots of softwares, some over more than a decade.

In summary: I understand your concerns, MyST is still quite a new standard. But I do think it is going to have its userbase, because there are simply no real alternative for scientists who need mobile solutions like me. In any case, I will maintain the feature if I succeed in implementing it, and I have a track record that shows I am an experienced maintainer.

Please let me know if I can go ahead.

[lrq3000]

PS: Usually I'm quite good at detecting which softwares have a great future, because I am quite meticulous with my choices since I need solutions that are future-proof.

For example, MyST has a huge feature that is super useful imho: not only does it support bibtex, but it supports bibtex from links online! So for example we could imagine linking to a bibtex in Zotero. Hence, mobile scientists could have a fully mobile solution that covers both scientific writing and scientific sourcing. This is the kind of thing that can make such a standard blow up once the userbase matures the usecases.

Anyway, I propose MyST because as a mobile scientist I determined it is the only solution that fits my needs, it is not replaceable with AsciiDoc. I am willing to do the work, but I need the greenlight, I won't do it if in any case it won't be merged.

[lrq3000]

PS2: ReStructured Text is not adequate for scientific publishing (no support for citations) so it does not interest me, although I can understand why you would like support for it. But this is way beyond my abilities and available time, RST is a lot more complicated than MyST (which is basically Markdown mixed with some RST).

[gsantner]

I understand your concerns, MyST is still quite a new standard.

So my take away is: It's new and give it time. If it shows enough adoption at other software it might get something of interest. But honestly put, I don't think it should be added at this point.

Anyway, I propose MyST because there are to my knowledge only 2 light markup languages that provide sufficient support of features for scientific publications

Markdown shines at taking notes, in writing blog posts, in commenting, making social media posts etc. Yes you can make formulas (KaTex), yes you can make links and footnotes. It may be enough for basic needs of scientific publishing, but it's not made for it. I think you should not forcefully try to turn something into something it is not and on it's way much change how the former used to be. Markdown is a simple format with limited amount of syntax that everybody can understand in short amount of time.

At the end, I can only suggest: To use something widely supported for scientific - and I think i.e. LaTex does a very good job here to cater to all needs. Markdown fits and works for many things, but I don't think it should be used for everything.

[lrq3000]

@gsantner I'm not trying to force Markdown into something it is not: MyST is another standard, and I am not making it.

It seems you are very opinionated about what Markdown, and Markdown inspired markup languages, should be. While I appreciate your suggestions, and I know LaTeX perfectly well (and it is awesome! But NOT mobile, syntax and plugin system is too complex for reasonable implementations with enough features for scientific publishing to work on mobile), I did not come here to ask for your opinion on what I should use, I know perfectly well my needs and I am aware of the limitations of using newer softwares. Trying to assume my needs is bound to fail, especially given I have disabilities that require specific tooling. To be more precise: I need to be able to write scientific articles from my bed, since I am often bedridden due to illness. Hence, I don't care what you think Markdown should be, if a superset can make my life and others with similar disabilities better, I go for it, plain and simple.

I also do believe that, more generally, mobile/agile scientific tools have a bright future. It is necessary for me already, but I'm sure it can be of use to others without disabilities as well.

I understand you are not open to merging such changes, and it seems it's not just because it is new, but because of MyST being an extension of Markdown. That's fine, I can make my own fork for my own needs. I just would appreciate not being patronized.

I would hence appreciate the guidance I asked for (where should I look for to implement a new markup language) as a fellow OSS developer. Thank you in advance.

[Benko111]

I actually like that proposal very much and do think it would absolutely be a suitable addition. From a users point of view that is. I'm into writing stuff, from books to articles or essays, I do so much more with Markor than using it just as a plain note taking app, I've got drafts of entire books both completed and uncompleted stored in Markor. I'd definitely welcome a solution enriching Markors format capabilities while at the same time still remain Markdown. Just my thoughts.

[gsantner]

I understand you are not open to merging such changes

Not quite right. I just don't feel it's viable yet. Give it a year and the situation might be well different. Might be also that other parsers support it by then and we don't need to rely on something that only can handle MyST.

I can't spot any request at the flexmark parser yet at last, which is what Markor uses. It's the one I know to have the biggest amount of compatible syntax/modes/markdown-specs when it comes to Java/Android world.

I would hence appreciate the guidance I asked for

There is something going on ith asciidoc, it's not merged yet. But once that is in it might get easier - like by swapping out asciidoc.js file and changing the converter association at Markdown in i.e. a temporary fork.

[lrq3000]

@gsantner Thank you for clarifying your position and for the guidance you provided, I appreciate it. I understand why you may want the standard to mature before merging it in.

I will nevertheless try to implement it as you suggested, and open a PR, even if it's not intended to be merged anytime soon.

Also congratulations about the AsciiDoc implementation, it's good to have such an advanced authoring standard supported in Markor!

[gsantner]

Current situation: I use Markdown everyday, really. Also read a lot of related articles, forums, blogs, Markdown tools, other Markdown software. Yet this is the only place I ever heard of MyST.

So need to say, it's not seems a good fit for Markor. I don't feel like it's a good idea to add just cater to a handful of people's personal tool preference.

Every work it's right tool, humbly said Markdown is not the "tool for everything". It has it's place what it can and can't do. There are other formats/syntax/languages that may fit better to have more features builtin, instead of added-on-top.

[lrq3000]

Sorry @gsantner my e-mail provider (gmail) decided to classify as spam all e-mails from github due to a wave of crypto spams... bummer, but now solved (I hope).

Thank you for your reconsidering my request. You are right that MystMD is not well known. This is because this is a component, the next-gen parser, of the much better known Jupyter Books, so you will get many more results looking for the latter.

Currently, there is a wave of new open-source markup languages and publishing ecosystems for scientific writing, with MystMD/Jupyter-Book, Typst and Quarto (and Asciidoc to a lesser extent but it's more intended for general writing than scientific writing). This is very exciting and they each offer very interesting new takes on scientific writing (for example Typst aims to be a simpler LaTeX, and a lot of people are convinced by the claim). And interestingly, all these tools use Markdown or a Markdown-like markup syntax.

But all these projects started or at least were refactored in their current form around 2019-2020, so they are arguably still very young. Yet, they each have reached 3-5k stars or even 28k stars for typst (!) on their respective github repositories, which is as much as the reference Asciidoc compiler Asciidoctor, which was started in 2014! (EDIT: if we look beyond developers communities, on Reddit the Typst community is much bigger than Asciidoc's - Quarto and MystMD/Jupyter-Book have no reddit communities). This shows that these projects have great traction despite being young.

I am very convinced this is not a fashion and that they really add a lot of value for scientific publications, but I agree these are still young and are dedicated to scientific writing. I would dispute that these are not a "handful of people", at least not fewer than Asciidoc's userbase (and for sure much more for Typst!), but it's not the majority of users either. And although they all have Markdown as their basis, they each have different, unique features and implementation challenges.

I would like instead to suggest to implement a plugins architecture, so that 3rd-party developers like myself could offer support for additional markup syntaxes without touching the core of Markor, and without you having the burden to maintain these additions.

Markor has an awesome UX, it is extremely well made and polished. It is only natural that so many users would like to extend it to support more syntaxes, just like CKEditor for web apps.