Option to ignore unexpected characters in markdown

[jakobkmar]

I have an application where I query markdown from a third party API, and this is valid markdown syntax in most of the cases. Therefore I am rendering the returned markdown text using mordant, and this works in most of the cases.

But I encountered the following problem where the mordant MarkdownRenderer encounters an unexpected "token":

Exception in thread "main" java.lang.IllegalStateException: Unexpected token when parsing inlines: org.intellij.markdown.ast.LeafASTNode@5c63e02a; [Markdown:~:'~'}]
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.parseInlines(MarkdownRenderer.kt:272)
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.innerInlines(MarkdownRenderer.kt:280)
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.innerInlines$default(MarkdownRenderer.kt:278)
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.parseStructure(MarkdownRenderer.kt:134)
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.parseStructure(MarkdownRenderer.kt:96)
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.parseFile(MarkdownRenderer.kt:87)
    at com.github.ajalt.mordant.markdown.MarkdownRenderer.render(MarkdownRenderer.kt:81)
    at com.github.ajalt.mordant.markdown.Markdown.document(Markdown.kt:25)
    at com.github.ajalt.mordant.markdown.Markdown.render(Markdown.kt:31)
    at com.github.ajalt.mordant.rendering.Widget$DefaultImpls.render$default(Widget.kt:8)
    at com.github.ajalt.mordant.terminal.Terminal.render(Terminal.kt:242)
    at com.github.ajalt.mordant.terminal.Terminal.println(Terminal.kt:201)

The corresponding text was:

"![comparison.png](https://i.imgur.com/KarVVbr.png)\n\n_Frame rate comparison between vanilla Minecraft and Sodium at a render distance of 32 chunks. You can find a world download with this exact scene [here](https://cdn.discordapp.com/attachments/705601849188679712/731924127476219914/Demo.zip) for your own comparison against this reference. Mileage may vary depending on how powerful your hardware is._\n\n<br>\n\nSodium is a free and open-source rendering engine replacement for the Minecraft client that greatly improves frame rates, reduces micro-stutter, and fixes graphical issues in Minecraft. It boasts wide compatibility with the Fabric mod ecosystem when compared to other mods and doesn't compromise on how the game looks, giving you that authentic block game feel.\n\n<br>\n\nIf you're coming from Optifine, you can generally expect a significant improvement to performance over it, but you'll be missing some small features while the Fabric community builds other free and open-source alternatives. For a quick list of replacement features (such as zoom), take a look [here](https://gist.github.com/modmuss50/deff1658c4550ca8b16cb5d40ceaa468). **Sodium and Optifine are incompatible with one another. You must pick one.**\n\n<br>\n\nYou can find more comparisons for various hardware configurations, such as...\n\n- [Intel i5-7200U @ 2.5GHz / Intel HD 620](https://i.imgur.com/0JrlAuf.png) (37-&gt;69fps)\n- [Intel i7-3770 @ 4.0GHz / GTX 970 (user-submitted)](https://i.imgur.com/gthEOTt.png) (27-&gt;152fps)\n- [Intel i3-6100 / GTX 750 Ti (user-submitted)](https://i.imgur.com/tGuLKNN.png) (10-&gt;102fps)\n- [Intel i7-8700K @ 5.0GHz / RTX 2080 Ti (user-submitted)](https://i.imgur.com/Q9z0vbB.png) (87-&gt;368fps)\n- [AMD Ryzen 5 2600 / RX 580 (user-submitted)](https://i.imgur.com/6WHjQR9.png) (133-&gt;586fps)\n\nYou can even find some exotic and low-end systems running Sodium:\n\n- [Raspberry Pi 4B / 4GB Variant (user-submitted)](https://i.imgur.com/RIGL7xp.png) (17-&gt;36fps)\n- [AMD Athlon X2 QL-45 / ATI Radeon 4530 (user-submitted)](https://i.imgur.com/7skXO7M.png) (18-&gt;49fps)\n\n \n**Note:** Sodium is mostly stable at this point, but it does not yet contain support for the Fabric Rendering API, which a small number of mods currently use. If you try to use these mods with Sodium, your game may crash or behave unexpectedly.\n\n# Features\n\n- A modern OpenGL rendering pipeline for chunk rendering that takes advantage of multi-draw techniques, allowing for a significant reduction in CPU overhead (~90%) when rendering the world. This can make a huge difference to frame rates for most computers that are not bottle-necked by the GPU or other components. Even if your GPU can't keep up, you'll experience much more stable frame times thanks to the CPU being able to work on other rendering tasks while it waits.\n<br>\n\n- Vertex data for rendered chunks is made much more compact, allowing for video memory and bandwidth requirements to be cut by almost 40%.\n<br>\n\n- Nearby block updates now take advantage of multi-threading, greatly reducing lag spikes caused by chunks needing to be updated. ([before](https://streamable.com/lm5sp5), [after](https://streamable.com/nsdl0r))\n<br>\n\n- Chunk faces which are not visible (or facing away from the camera) are culled very early in the rendering process, eliminating a ton of geometry that would have to be processed on the GPU only to be immediately discarded. For integrated GPUs, this can greatly reduce memory bandwidth requirements and provide a modest speedup even when GPU-bound.\n<br>\n\n- Plentiful optimizations for chunk loading and block rendering, making chunk loading significantly faster and less damaging to frame rates. ([before](https://streamable.com/3taw22), [after](https://streamable.com/4pesh2))\n<br>\n\n- Many optimizations for vertex building and matrix transformations, speeding up block entity, mob, and item rendering significantly for when you get carried away placing too many chests in one room.\n<br>\n\n- Many improvements to how the game manages memory and allocates objects, which in turn reduces memory consumption and lag spikes caused by garbage collector activity.\n<br>\n\n- Many graphical fixes for smooth lighting effects, making the game run better while still applying a healthy amount of optimization. For example, take this [before and after](https://i.imgur.com/lYmlmgq.png) of a white concrete room in vanilla, or this [comparison while underwater](https://i.imgur.com/QQMuOTy.png).\n<br>\n\n- Smooth lighting for fluids and other special blocks. ([comparison](https://i.imgur.com/z9HBcvq.png))\n<br>\n\n- Smooth biome blending for blocks and fluids, providing greatly improved graphical quality that is significantly less computationally intensive.  ([comparison](https://i.imgur.com/Fud5oyF.png))\n<br>\n\n- Animated textures which are not visible in the world are not updated, speeding up texture updating on most hardware (especially AMD cards.)\n<br>\n\n... and much more, this list is still being written after the initial release.\n\n## Installation\n\nMake sure you have the latest version of [Fabric Loader](https://fabricmc.net/use/) installed. Afterwards, all you need to do is simply drop the mod into your mods folder. No other mods (not even the Fabric API!) are required in order to use Sodium. You do not need to create new worlds in order to take advantage of the mod.\n\n## Configuration\n\nSodium replaces the video settings screen with a new and improved user interface that contains all the bells and whistles for configuring Sodium. Out of the box, Sodium will enable all optimizations which are supported on your system.\n\n## Reporting Issues\n\nPlease use the [issue tracker](https://github.com/jellysquid3/sodium-fabric/issues) linked at the top of the page to report bugs, crashes, and other issues.\n\n## Common questions\n\n**Will you port to Minecraft 1.15 and older? What about Minecraft 1.8.9?** \n\nNo. Sodium is a modern mod designed for modern versions of Minecraft. Please stop asking.\n\n<br>\n\n**Will you add Forge support?**\n\nNo. Forge compatibility is not being considered. You can read my thoughts [here](https://gist.github.com/jellysquid3/629eb84a74ab326046faf971150dc6c3) on why that decision was made.\n\n<br>\n\n**Does Sodium support Optifine shader packs?**\n\nNo. However, experimentation is being done and it is possible in the future that Sodium could provide its own shader pack system. No promises are being made.\n\n<br>\n\n**Is Sodium a replacement for Optifine?**\n\nIf you need the best performance out of your game, it very well could be. The primary focus of Sodium is optimization and improving rendering quality right now, not features. If you're looking for all the other features of which Optifine provides, check out [this list](https://gist.github.com/LambdAurora/1f6a4a99af374ce500f250c6b42e8754) for some Fabric-based alternatives (such as zoom functionality) which are compatible with Sodium. "

Is it possible to add an option to ignore these unexpected characters / tokens?

[ajalt]

Thanks for the stack trace and reproducer.

The very first line in the CommonMark spec is

Any sequence of characters is a valid CommonMark document.

So I don't think that mordant should ever throw an exception when rendering markdown. I'll take a look at this bug, and if you find any other inputs that throw a different stack trace, please open up a new issue.