[23] JEP 467 - Compiler changes for supporting markdown Javadoc comments

jarthana commented 3 months ago

What it does

Compiler changes for supporting markdown Javadoc comments (https://github.com/eclipse-jdt/eclipse.jdt.core/issues/2429)

How to test

Author checklist

[ ] I have thoroughly tested my changes
[ ] The change is following the coding conventions
[ ] I have signed the Eclipse Contributor Agreement (ECA)

stephan-herrmann commented 2 months ago

Hi @jarthana, this looks very promising!

Do you already have ideas how clients can detect the style (html / markdown) of a given org.eclipse.jdt.core.dom.Javadoc?

Once this information is available, I might be able to start playing with the JDT/UI side of it :)

stephan-herrmann commented 2 months ago

I couldn't hold my horses :) and made some experiments towards populating DOM's Javadoc with parsed markdown comments. Turned out I only needed to tweak one scanner method to achieve first results:

diff --git a/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/Scanner.java b/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/Scanner.java
index 263d1c0..7ebfaf0 100644
--- a/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/Scanner.java
+++ b/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/Scanner.java
@@ -3124,6 +3124,21 @@
    int stopPosition = this.currentPosition;
    switch (token) {
        case TokenNameCOMMENT_LINE:
+           if (commentStart+2 < stopPosition && this.source[commentStart+2] == '/') { // check for third '/'
+               if (this.commentPtr > -1) {
+                   // is this a continuation of a previous markdown comment?
+                   int prevStop = this.commentStops[this.commentPtr];
+                   while (CharOperation.isWhitespace(this.source[prevStop])) {
+                       if (++prevStop == commentStart) {
+                           // extend markdown region, FIXME: need to check if previous actually is a markdown comment
+                           this.commentStops[this.commentPtr] = stopPosition;
+                           return;
+                       }
+                       if (prevStop >= this.source.length) break; // safety, shouldn't happen
+                   }
+               }
+               break; // add new markdown comment below (positive positions to signal javadoc!)
+           }
            // both positions are negative
            commentStart = -this.startPosition;
            stopPosition = -this.lastCommentLinePosition;

This addresses two issues:

only positive positions in commentStarts and commentStops are recognized as javadoc positions, but /// comments were treated as line comments (due to start with // :))
based on treatment as line comment, we only saw lines, not multi-line regions

Does this fit your design?

For handling of tags, also DocCommentParser will need to implement parseMarkdownLinks(). For a quick and dirty experiment a copy from JavadocParser looked promising, perhaps it can simply be pushed up to AbstractCommentParser, or do you envision any differences between use cases?

Both hacks together resulted in some Javadoc that I can play with, but I should mention that the result wasn't yet complete. In particular I had a @param tag with description, but no TagElement.fragment was created for the description, so perhaps some "end-of-region" action isn't happening or such.

jarthana commented 2 months ago

Hi @jarthana, this looks very promising!

Do you already have ideas how clients can detect the style (html / markdown) of a given org.eclipse.jdt.core.dom.Javadoc?

Once this information is available, I might be able to start playing with the JDT/UI side of it :)

Stephan, I was thinking of taking a step-by-step approach. Hence the limited change to the dom* code. I have asked @noopur2507 to spend some time and tell us what the jdt.ui needs/wants from JDT core APIs. I plan to take this patch to completion and take up the DOM changes immediately in another PR. Thanks for spending time on this - this helps!

jarthana commented 2 months ago

I couldn't hold my horses :) and made some experiments towards populating DOM's Javadoc with parsed markdown comments. Turned out I only needed to tweak one scanner method to achieve first results:

Does this fit your design?

I forgot that we have to replicate the scanner code from getNextToken0() in jumpOverMethodBody() as well. I have also added an explicit case in the recordComment now for the markdown.

For handling of tags, also DocCommentParser will need to implement parseMarkdownLinks(). For a quick and dirty experiment a copy from JavadocParser looked promising, perhaps it can simply be pushed up to AbstractCommentParser, or do you envision any differences between use cases?

I just went by what we do with parseTag(), where we seem to have different implementation for these two parsers. Perhaps we can abstract out few things to AbstractCommentParser, but don't yet know.

Both hacks together resulted in some Javadoc that I can play with, but I should mention that the result wasn't yet complete. In particular I had a @param tag with description, but no TagElement.fragment was created for the description, so perhaps some "end-of-region" action isn't happening or such.

eclipse-jdt / eclipse.jdt.core