Clarify the Markdown scenarios

[alerque]

Per #1610 closing a bunch of Markdown related issues I think we have some documentation updates to do. I'd like to

• [ ] Look into adopting markdown.sile into this GitHub org as semi-official.
• [x] Prominently document approaches to handling Markdown based document workflows in the manual and README.
• [ ] Also document an approach to handling minimal Markdown strings without a full inputter / class if we have such a thing available.

[alerque]

@Omikhleia Just a heads up one of the big hindrances for me in adopting / recommending markdown.sile is the current hard dependency on silex.sile. Given the latter's path handling, it ends up taking over core SILE stuff. This may be fine for some folks and some projects, but it definitely isn't appropriate as a default way for people to bolt on Markdown input support.

Given that I've started using the develop branch in production and we've dropped the old Markdown support there, I've had to patch it back into SILE just for rendering a few simple strings.

I think we need to reconsider bringing Markdown parsing back into SILE's core without introducing too many new dependencies. I realize the libraries we ripped out were a good riddance, but I'd like to get markdown.sile's robustness a lot easier to reach for without dragging along a too many other things.

[Omikhleia]

I think we need to reconsider bringing Markdown parsing back into SILE's core without introducing too many new dependencies

As for dependencies: without tables, language support and smart quotes, proper super sub- and superscript, I am not sure how it can be achieved with SILE's (current) core without losing functionality... I'm certainly willing to discuss it, but it's not an obvious topic. Perhaps we could have an open discussion on our Gitter (even privately if needed first), at least to consider possible strategies? Also note that markdown.sile goes beyond "mere" Markdown, with Djot etc.

... the current hard dependency on silex.sile

Some things from silex.sile could be sorted out and ported here upstream. I had to move "fast" to be able to make book(s), but there would be a way for some features. On the same vein as I proposed #1853, we could work on proposing other backports. But beside my limited time, my main concern is that I am unsure how to build the new develop branch, and most of all when you'd want 0.15 out. I'd need some perspectives and an idea of the roadmap, to organize my time here.

I've had to patch (the old Markdown support) back into SILE just for rendering a few simple strings.

Of course, I don't think this is a good move -- I'm sure we can do better than reviving that old broken/limited solution.

[Omikhleia]

BTW, unless I missed something, the only required "hard dependency" in markdown.sile on silex.sile are

• silex.lang = a trimmed down variant of #1641 so if we could progress on it, this would no longer be needed
• silex.ast = some generic AST utilities which should, IMHO, be part of SILE...
• EDIT: and proper em-nesting and ragged-environment nesting -- but IMHO that should also be part of SILE core.

[Omikhleia]

BTW, unless I missed something, the only required "hard dependency" in markdown.sile on silex.sile are

Of course I'm missing the current dependencies...

• embedders.sile = doesn't depend on silex (and only exists because SILE's "converters" are lame, and lilypond.sile is non-existent....) :heavy_check_mark:
• ptable.sile = does depend on silex.compat, so this could go away at some point. --> Done :heavy_check_mark:
• smartquotes.sile = does depend on silex, but also because of silex.lang = #1641 so as above, moving forward with the latter would make it unnecessary.
• textsubsuper.sile = does depend on silex, but #1853 would make it unnecessary. --> Done :heavy_check_mark:
• labelrefs.sile = doesn't depend on silex, strictly speaking (though it behaves poorly with multiple package re-instantiation, but maybe this is now a moot point, or at least we made progress towards a solution) :heavy_check_mark:

There's nothing impossible here ;)

[alerque]

Sorry if that came out wrong, I wasn't blaming you for the way you went about things. I've had to move fast and fix things downstream for my own production work too, I know how it goes. Yes we can talk this over. I'm moving this week so any comments I drop will be a bit spastic. My only intent was to give you a heads up that I was reconsidering v0.15.0 shipping with no level of Markdown inputer support.

[Omikhleia]

Sorry if that came out wrong (...) My only intent was to give you a heads up that I was reconsidering v0.15.0 shipping with no level of Markdown inputer support.

No problem - and you might indeed have to do so, depending on your target release date for 0.15.

I am just pointing that there's nevertheless a way forward for removing the dependency on silex.sile from markdown.sile and its dependencies:

• extracting some fixes and refactors = #1913, #1914
• completing here some PRs which were adopted as early backports/workarounds = #1641, #1853

As I wrote, it doesn't seem impossible or inaccessible.

Then on the other side, I'd need to make a new (major) release of these components, without the silex dependency and officially supporting 0.15 (and possibly also fixing things for Lua 5.1 as per https://github.com/Omikhleia/markdown.sile/pull/101), and doing tests do ensure we are not missing anything (abstracting various hacks to silex.sile was a difficult decision, but at least it means my other components are not that cluttered with them).

So there's a bit of a chicken and egg situation, but it all seems doable with a few well-planned iterations...

... #1641 is probably the toughest point - the draft PR is kind of a workaround - but we could also try to address the minimal need and open new tickets for refactors we could address separately later (= such as deciding if some stuff eventually needs to be moved in cldr, splitting hyphenation patterns and better refactoring i18n, etc.).

[Omikhleia]

A few months later... With markdown.sile 1.5.2 + silex.sile 0.4.1, the latter no longer aggressively "takes over core SILE stuff" (i.e. excepted the bare minimum: language override for BCP47 support and AST utilities). The whole of silex's replacements is now only loaded when using my resilient classes. This addresses one of the point raised here. A possible negative impact of the change is that the multiple package instantiation model introduced in SILE 0.13-0.14 may cause some havoc in some workflows not using a resilient class. One can't make an omelette without breaking eggs.

[Omikhleia]

Now that the "silex" issue is solved, out of the three check-boxes on this issue

• "Look into adopting ..." - Dunno what's at stakes here ;)
• "Prominently document approaches to handling Markdown" - FWIW, I plan rewriting my own guide to markdown.sile this year, but I'm unsure what help I can be here... But it's somehow related to the previous point.
• "Document an approach to handling minimal Markdown strings without a full inputter" --> This one may need clarification. What is "minimal Markdown" and how is it expected to work? I'm interested, because I'd want to have some capabilities for Markdown/Djot in many places (say, in the YAML-based resilient document master file...) but there are some difficult points here.

[Omikhleia]

... but there are some difficult points here.

And then I am wondering what I meant here, lol! -- but before I forget again, here it is:

• There's an opened issue on djot.lua on how to process inline content only (not linking it here, since I already linked it elsewhere)
• On the lunamark side (for Markdown), it would require a small patch or hack too...

In both cases, the issue at stakes is how to embed Markdown/Djot in another (e.g. say SIL) document, where we know from the surrounding context whether we need to support inline elements only (e.g. in the course of a paragraph) or block elements (on their own paragraphs). Not having a clean way for passing this information to the inputter (and eventually to the parsing engine) makes it harder to manage (though not impossible[^1]). The use case at stakes is, for instance: ... Some \strong{SIL} content, \raw[type=markdown]{Some *Markdown* content}, and let's be back to SIL in that same paragraph.

[^1]: We could let it parse the content, which would currently always end up being wrapped in a paragraph, and if in inline surrounding context, check if the parser returned one single paragraph (otherwise, raise an error) and remove it (using only its inline content).

[Omikhleia]

This issue = Sept. 12, 2023

Regarding markdown.sile as 3rd-party collection:

• v1.5.0 = Oct. 2023 (small feature release)
• v1.5.1 = Dec. 2023 (Lua 5.1 / LuaJIT compatibility, one of the points at stake regarding SILE's expectations)
• v1.5.2 = Jan. 2024 (minimal dependency on silex.sile, one of the points at stake in the above discussions, reported as a "major hindrance")
• v2.0.0 = March 2024 (big feature release)
• v2.1.0 = Aug. 2024 (SILE 0.15 upgrade)
• v2.2.0 = Oct. 2024 (small feature release)

I still don't have any idea what this issue is about or what would be expected here, to plan ahead ;)