Use markdown in place of embedded html for doc

[cmnbroad]

There is a fair amount of discussion in this thread

[magicDGS]

Suggestion: https://github.com/atlassian/commonmark-java, which supports:

• Markdown -> HTML
• Markdown -> Plain text

I would love to have this support in Barclay, so maybe I can contribute to add this to the codebase. Because I'm doing my documentation with Markdown + jekyll, it will be nice if the support for the HelpDoclet is customizable and could render to HTML or let it as it is.

Let me know if you want me to work on this.

[cmnbroad]

@magicDGS We're definitely interested in markdown, and would of course welcome the help, but I'm not sure we'll have much time to devote to it in the short term (and I don't know anything about commonmark-java - for starters it would have to have a compatible license).

[vdauwera]

At a glance the license looks like some flavor of BSD, which would be totally fine.

Do either of you have a sense of how much work would be involved in getting this integrated?

[magicDGS]

I had a look to the commonmark-java library and it looks super-simple to use, although it depends on how much do you want to integrate markdown in Barclay. My idea, which is maybe too simple, is to allow markdown formatted strings in:

• @Argument.doc()
• @CommandLineProgramGroup.getDescription()
• @CommandLineProgramProperties.summary()
• @CommandLineProgramProperties.oneLineSummary()
• @CommandLineProgramProperties.usageExample()
• @PositionalArguments.doc()
• @ArgumentCollection.doc() (I would like to have this displayed at some point, see #7 for discussion)

The returned String for this method will be parsed with the org.commonmark.parser.Parser in both clp/docgen classes, and rendered differently:

• CLP will render it to plain text using org.commonmark.renderer.text.TextContentRenderer to output in the terminal.
• DocGen will render it to HTML using org.commonmark.renderer.html.HtmlRenderer, which is quite flexible to include custom features with extensions.
• DocGen should have the possibility to use a different Renderer for custom implementations or no rendering at all: this will allow to output markdown formatted files without the need of rendering here, but with a clean command line.

If this support is enough for start with the integration of Markdown, I think that this won't take too much time to integrate unless you have specific needs for rendering markdown to html. Let me know if you are interested in this contribution.

[vdauwera]

This sounds super valuable and quite simple as you say, thank you. I'll defer to @cmnbroad if he thinks there is a reason not to do this right now (eg if it would interfere with developments he has planned or if there is higher value work that should be done first) but if he has no objections then I would say please go ahead and do it!

[magicDGS]

Cool! I forgot to mention that the @DocumentedFeature.groupSummary() and @DocumentedFeature.summary() should be also rendered in the DocGen.

[magicDGS]

If there is any reason for delay this, I'm fine with it. Anyway, I added a quick draft in my repository for backup my main idea: https://github.com/magicDGS/barclay/tree/dgs_markdown_integration

Feel free to have a look and to tell me to upload a PR if you wanna discuss the changes and keep track of them.

[vdauwera]

From my perspective, sooner is better than later, because we are making a big push to update and clean up all the tool documentation ahead of the release. I would rather write any new stuff directly in markdown rather than HTML then have to convert it later.

[cmnbroad]

Interesting - this is a bit different than what I was thinking, which was that the main reason we would embrace markdown would be as a replacement for embedded html in javadoc, rather than introducing another embedded syntax in places where we currently only support plain text. Maybe I was the only one thinking that, though. I'd prefer to have a resolution/solution to that issue before we introduce yet another embedded syntax into the mix. I think there are doclets out there that do allow that for javadoc but I've not really looked at them. @vdauwera thoughts ?

[cmnbroad]

BTW, I wrote that comment before I seeing @magicDGS' comment that he had a branch, so I haven't looked at that yet.

[vdauwera]

@cmnbroad that was also my original purpose but if it's possible to support markdown elsewhere without too much difficulty it empowers us to eg include active links in argument definitions.

[magicDGS]

Ok, I misunderstood the issue, but I think that include Markdown into the argument definitions or summaries will be good for have a clean command line help and a formatted online documentation without much more issues. I also generate a normal javadoc, and I usually format it in HTML.

Anyway, I believe that my branch changes will allow to render the javadoc to HTML if it is written in Markdown, and I think that commonmark-java will do nothing to HTML formatted code (I haven't tested). If both HTML and Markdown could be included in the javadoc, it will be great.

Maybe have a look to my branch and tell me if you think that it is a good addition. I'm planning to incorporate something similar to my toolkit, but the CLP is less flexible to add a renderer as I did in my draft...

[vdauwera]

I'll look at your branch, @magicDGS

[vdauwera]

@magicDGS This looks great to me! I really like that you can use eg bold and code markup, and especially include links in argument docs!!

Could you please also show this working on a tool that has a documentation block ahead of the class, like https://github.com/broadinstitute/gatk/blob/master/src/main/java/org/broadinstitute/hellbender/tools/walkers/bqsr/BaseRecalibrator.java ?

This is where we'll get really high value from Markdown support.

[magicDGS]

Thanks @vdauwera! With respect to the example of BaseRecalibrator, do you mean to parse Markdown and render to HTML the javadoc text? I'll try that in my next comment...

[magicDGS]

Hi @vdauwera and @cmnbroad, the current state of the branch includes what you wanted to do: javadoc formatted in Markdown is rendered to HTML for the doclet. In addition, the javadoc could still be written in HTML and rendered correctly.

I think that this is a good addition, in my case because I would like to use Markdown in the argument docs and in your case to write simpler tool help by using Markdown in javadoc. Should I open a PR for further discussion about details?

[vdauwera]

Yes please open a pull request, that would be wonderful. Thank you!