Custom executables for display.Markdown [feature request]

[kiwi0fruit]

It would be nice to be able to set custom Markdown to HTML converters for display.Markdown(...) in Atom settings. Like path to executable with arguments that reads from stdin and writes to stdout (like pandoc --from gfm or pandoctools --profile Jupyter).

With this it would be easier to write Stitch/Knotr and Knitty/Pandoctools documents that put Markdown code inside Python code (all these programs use Pandoc filter that runs code blocks via Jupyter kernels, grabs outputs, parses and inserts them into Pandoc abstract syntax tree for future processing).

Putting all Markdown into Python code would allow using f-strings for variables substitution and would utilize existing Hydrogen capabilities without additional modifications (like running Markdown cells in Hydrogen too - but that's a separate feature request).

[kiwi0fruit]

I would appreciate pointing to the right place in Hydrogen code to make changes so that I could make a PR (if there is no objections to it).

[wadethestealth]

@kiwi0fruit I am unsure what you mean by display.Markdown() but with IPython.display you can already put Markdown in python code and output it

also stitch is for putting Python in Markdown, which is the opposite of hydrogen ^Not True Oops Hydrogen works with .md as well

Edit: I took a look at the Knitty docs looks like you are reffering to the IPython.display, which we do not have control over, and thats for jupyter not us. In fact IPython may have already added this to their own module. Here is the docs

[kiwi0fruit]

I mean IPython.display.Markdown(). This creates special Markdown object that is then parsed and displayed by Hydrogen (other Jupyter clients also do this). So I'm talking about overriding client behavior. Not server behavior. Server sends Markdown object and is OK with that.

[wadethestealth]

@kiwi0fruit The kernel returns a data stream which we copy to an output component called OutputStore. This OutputStore is a prop of a Display component inside ResultView for inline results and is a prop of an Output component inside of History. Both History and ResultView are exported from files in hydrogen/lib/components/result-view, and Output and Display are imported from @nteract/display-area, which is outdated and there has been inside talk of updating this. So technically Hydrogen doesn't do any parsing, but its dependency does. However, I may be able to help you better if I understood what you were trying to do. Also markdown cells are now runnable as of today, if you are using this repo instead of the released version from apm.

[kiwi0fruit]

Thanks! I'll dig into this. And nice to hear that Markdown cells work now.

[kiwi0fruit]

So technically Hydrogen doesn't do any parsing, but its dependency does

I'm going to add an if switch where Hydrogen uses it's dependency to parse Markdown and add option to use external parser instead (specified in a new Atom setting to add) - do a subprocess stdin-stdout call.

[kiwi0fruit]

Subprocess interaction would be like in unix-filter Atom package.

[wadethestealth]

@kiwi0fruit a little note is that you wouldn't be able to just replace Display or Output with a Markdown to Html parser since both of those support many MIME types

I also think that your original problem could be solved by just creating a python or whatever language module that parses markdown into custom html, this way you have much more flexibility.

[kiwi0fruit]

If it's so then the simplest solution might be to plug Markdown to Markdown converter (from user desired flavour to standard Jupyter flavor) if MIME type is Markdown. And user sets this converter in Atom settings. Luckily Pandoc is great at md to md conversion.

This feature request is convenient for writing any desired Markdown flavour and instantly previwing it right via Hydrogen (instead of separate pane with Markdown Preview Plus).

your original problem could be solved by just creating a python or whatever language module

This is the simplest solution but with drawbacks. It's not general: it would work only with Python kernels. It would put some code that should properly be on client side to kernel side - that doesn't look like good idea (there is already matplotlib with it's quirks and kernel behaviour depending on client type).

Also unfortunately at the moment there is a bug/feature when high dpi images rendered via Markdown() are displayed too big - wider than preview area. I guess this might be useful for matplotlib output and likes but it spoils using Hydrogen for Markdown editing. Is there a way to fix it so that all usage cases are convenient?

[wadethestealth]

@kiwi0fruit I am starting to understand better, so one way would be to create a Hydrogen Plugin For Markdown Flavors basically you could add a command to your own package that first calls our plugin api function getCellRange() You can then use the atom editor abilities to directly edit the text into jupyter flavor. Or you could not change the text but use builtin hydrogen methods (obvious not going to be guaranteed version to version) to run your jupyter version. Also for the images too big issue, why not just set a size for the image in your markdown code

[kiwi0fruit]

Oh. I completely forgot about Hydrogen plugins. Sounds like a great idea. Just to confirm: Is it possible for a Hydrogen plugin to override standard Hydrogen behaviour and add additional operation between when Hydrogen receives MIME object and sends it further for conversion?

As about setting image size manually: it's a deal breaker for use case of instant previewing Markdown that is not intended to have image size set. Setting it specifically for Hydrogen then removing it is unacceptable.

[wadethestealth]

@kiwi0fruit Technically it is easily possible to call hydrogen functions and reroute any and all behavior inside a plugin, but It is not officially supported meaning version to version your plugin may break and need to be rewritten around internal changes.

[kiwi0fruit]

After all I find creating the extension is too much additional work. PR or auto-patching fork is much easier to implement.

[wadethestealth]

@kiwi0fruit I am currently working on updating our output dependency, which will give the ability to add custom markdown flavors very simply, by creating a new react component which takes the text and parses it to your desired markdown flavor. Also I believe that while I work on that its probably best to do an auto patch instead

[kiwi0fruit]

Sounds great. By the way. Now Hydrogen is able to render Markdown cells in # %% .py (and other kernels) mode. What if to expand this functionality to .md documents? Like it can render Markdown text that is in between code blocks in Markdown (that can be run in Hydrogen now). Maybe even optionally split Markdown to pieces via <!-- %% -->.

But this all make sense if to adapt Hydrogen to be used instead of separate Markdown preview pane. Is it a good user experience for writing Markdown documents? I find it to be in between separate preview pane UX and in place rendering like in Typora. It sounded like good idea to me. Is it really convenient?

[wadethestealth]

@kiwi0fruit This is definitely a neat idea. I will look into that next.

[wadethestealth]

@kiwi0fruit I am closing this issue because it is quite cluttered, but you are free to open a new issue about this once #1641 is merged