Support for Java 11

[chrisvest]

The javadoclet APIs changed a lot in Java 9. And is now much more locked down. Adding support (in this case for Java 11) for asciidoclet unfortunately requires a lot of hacks, but things appear to work stably.

The doclet APIs now work by processing the abstract syntax tree from the java compiler. This is presented to the doclet via the DocletEnvironment interface. In order to avoid duplicating the large body of functionality that exists in the standard doclet, we pass a filtered environment to the standard doclet, which post-processes the javadoc comments, so the standard doclet sees asciidoctor output in place of the javadoc that is written in the source files.

This is fine in principle, but it turns out that the standard doclet is not written to the interfaces, but rather the internal APIs of javac. This immediately forces us to break the module encapsulation via --add-exports. We have to either take the path of hacks in order to reuse the standard doclet, or reimplement all of the standard doclet, or post-process the javadoc HTML that the standard doclet produces. I opted for hacks, since that appeared to be the path of least resistance.

It gets worse, though. Due to a mix of visibility constraints, immutability, memoisation, and lack of extension points, we can't just present a modified view of the AST to the standard doclet. We have to actually dig into, and mutate, the AST to make it look like a person wrote asciidoctor output in their javadoc comments. This messes up error reporting and linting, since javadoc will now be producing warnings for source locations that don't actually exist. This mess is concentrated in LazyDocCommentTableProcessor and AsciidocComment.

The related issue is #76

[chrisvest]

What's currently missing is 1) tests to prove that the stylesheets and templates are copied to the correct places and used, 2) bringing back the AsciidoctorRendererTest, and as part of that 3) fix the handling of javadoc tags like @param, @throws, @author, etc.

[chrisvest]

Javadocs should be parsed correctly now, though some tags are not going through and I'm not sure why. Resources such as stylesheets are still not being copied correctly. Also, attributes such as {project_name} are not being replaced - maybe they are not being loaded correctly.

[chrisvest]

@msgilligan Have you had a chance to look at this?

[LightGuard]

@johncarl81 take a look. I think this is a good first step in getting a doclet that works with Java 11.

[johncarl81]

@LightGuard - I've been following, and yes, this is an excellent first step. I think we should probably make a new repo or a hard version break at least for this next version. Fantastic work @chrisvest!

[LightGuard]

I think a hard version break is probably best, it doesn't change anything but the version number. Maybe we need to create a new branch for a 2.x version. I think it would at least be good to get this out as a preview and see how it works in the wild.

[LightGuard]

Well, when I say it doesn't change anything, I mean the rest of the gav.

[chrisvest]

Attributes are being replaced correctly, but resources such as stylesheets are still not being copied correctly to the destination directory, because the javadoc tool does not pass the destination directory argument through to the doclet.

[chrisvest]

Stylesheets are now copied correctly, so coderay highlighting works again. A Java 11 stylesheet has also been added. There are two problems left, AFAIK: 1) the overview.adoc is somehow not processed through asciidoctor, so javadoc complains about the strange syntax, and 2) asciidoctor (in the version currently integrated) is producing HTML elements that javadoc says is not part of HTML5, such as data-lang and rel attributes, and tables without summary or caption.

Would an asciidoctor upgrade help with the second problem? These HTML5 issues are reported as errors, which makes javadoc fail the build.

[chrisvest]

Adding <additionalJOption>-Xdoclint:none</additionalJOption> to the javadoc configuration makes the build pass, but the overview is still not rendered properly.

[chrisvest]

Overview files are now processed correctly. The HTML problems have been ignored. So now the javadocs are generated correctly, and without failing the build, given the right options are provided.

@johncarl81 I think this has reached feature and functional parity with the asciidoclet for pre-modular Java versions, so now would be a good time to figure out how and where this gets integrated. And then find a cutting point between this PR and future work.

[chrisvest]

This PR is an example of the sort of changes people will have to do, to upgrade to an asciidoclet with these changes: https://github.com/chrisvest/stormpot/pull/118

[LightGuard]

Any further update on this? It would be nice to get a release cut, even if it's a beta.

[chrisvest]

@LightGuard I'm using this via jitpack to my fork at https://github.com/chrisvest/asciidoclet/tree/chrisvest:

<repositories>
  <repository>
    <!-- Required for the asciidoclet fork. -->
    <id>jitpack.io</id>
    <url>https://jitpack.io</url>
  </repository>
</repositories>
...
<groupId>com.github.chrisvest</groupId>
<artifactId>asciidoclet</artifactId>
<version>2.0.0</version>

[chrisvest]

It looks like newer versions of the Maven Javadoc plug-in now refuses to download doclet dependencies from anywhere other than Maven Central, which means the JitPack trick above is now ineffective. Unless you use an older version of the plug-in.

Alternatively, you can vendor the binary and refer to it via docletPath.

See https://github.com/chrisvest/asciidoclet/releases/tag/2.0.0 for more information.

[cradloff]

The pull request is now more than a year old. Will there ever be support for Java 11 or is this project dead? And if it's dead, is there an alternative?

[msgilligan]

I'm interested in getting Java 11 support finished and released. I apologize for not being involved for the last year, but I've just had way too much on my plate. I have more free time now and I have a pressing need to upgrade my asciidoclet-using project ConsensusJ to Java 11, so I have extra motivation and a real-world test case.

I'm not capable of doing this all single-handedly, but maybe some of the there maintainers will be able to help as well. @johncarl81 @mojavelinux ?

@chrisvest Are you still available to help?

[msgilligan]

@msgilligan Have you had a chance to look at this?

Sorry for not responding until now, but I was spread very thin for the last year. I'll look at the code this weekend.

[msgilligan]

I think a hard version break is probably best, it doesn't change anything but the version number. Maybe we need to create a new branch for a 2.x version.

Yes. I agree we should make a 2.x version that requires Java 9+. Note that people will still be able to compile code for earlier versions of Java, they'll just need to use JDK 9+ when building the JavaDoc.

I also think the 2.x version should use all the latest upstream AsciiDoctor tooling (though we may want to get this PR merged on a 2.x branch before we try updating the other jars)

I think it would at least be good to get this out as a preview and see how it works in the wild.

I agree and I can create a branch of my ConensusJ project that uses Asciidoclet for some real-world testing.

[chrisvest]

@chrisvest Are you still available to help?

Yes.

I've been using my own fork for half a year now. I'd also like to see the asciidoctor version upgraded to the latest, and to have asciidoctor-diagram support added. Someone already did that work based off of my fork, but unfortunately not in a way that allow me to apply those patches to my own fork.

[msgilligan]

Thanks @chrisvest. I'll try to push this forward, but my availability is limited and I don't have the authority to create a release, so I'm definitely hoping others will help as well.

[bentolor]

So much great work has already done in this PR and based on the comments the results are good.

How about just merging this in a 2.0-alpha release & branch or something release so people can actually use it and then trying to get the other things pinpointed/done?

AFAIK mostly upgrade of asciidoctor4j should also go in a 2.0 release, right?

Am I missing further work to be done?

[johncarl81]

@bentolor, I'd like to review it first... I think it's nearly there, @chrisvest just needs to respond to my comments. In the mean time you're free to use any code you wish.

[chrisvest]

@johncarl81 Replied to your comments.

[abelsromero]

Hi :wave: I'd like to continue the work from this PR. Does anyone have any objection?

[LightGuard]

Go for it.

[abelsromero]

Closing since https://github.com/asciidoctor/asciidoclet/pull/108 was merged. Many thanks to all!