search

davidB / scala-maven-plugin

The scala-maven-plugin (previously maven-scala-plugin) is used for compiling/testing/running/documenting scala code in maven.
https://davidb.github.io/scala-maven-plugin/
The Unlicense
560 stars 151 forks source link

Generating and Attaching of Scaladocs are broken with Scala3 #604

Open chr78rm opened 2 years ago

chr78rm commented 2 years ago

The production of Scaladoc APIs are broken when using Scala3, e.g. with Scala 3.1.2:

<profile>
    <id>scala3-failure</id>
    <properties>
        <scala-maven-plugin.version>4.6.1</scala-maven-plugin.version>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>${scala-maven-plugin.version}</version>
                <executions>
                    <execution>
                        <id>Scalac</id>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                            <goal>add-source</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-deprecation</arg>
                                <arg>-release:17</arg>
                                <arg>-encoding</arg>
                                <arg>utf-8</arg>
                            </args>
                        </configuration>
                    </execution>
                    <execution>
                        <id>Scaladoc-jar</id>
                        <goals>
                            <goal>doc-jar</goal>
                        </goals>
                        <phase>package</phase>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala3-library_3</artifactId>
            <version>3.1.2</version>
        </dependency>
    </dependencies>
</profile>

This gives

$ mvn clean install -P scala3-failure
...
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ scaladoc-jar-test ---
[INFO] Building jar: C:\Users\Developer\Git\Git-2.36.0-64-bit\projects\scaladoc-jar-test\target\scaladoc-jar-test-0.0.1-sources.jar
[INFO]
[INFO] >>> scala-maven-plugin:4.6.1:doc-jar (Scaladoc-jar) > generate-sources @ scaladoc-jar-test >>>
[INFO]
[INFO] --- scala-maven-plugin:4.6.1:add-source (Scalac) @ scaladoc-jar-test ---
[INFO]
[INFO] <<< scala-maven-plugin:4.6.1:doc-jar (Scaladoc-jar) < generate-sources @ scaladoc-jar-test <<<
[INFO]
[INFO]
[INFO] --- scala-maven-plugin:4.6.1:doc-jar (Scaladoc-jar) @ scaladoc-jar-test ---
bad option '-doc-format:html' was ignored
bad option '-doc-title' was ignored
source file not found: scaladoc-jar-test 0.0.1 API
2 warnings found
1 error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...

The issue can be reproduced with the build profile scala3-failure of the test project. The test project can be cloned by

git clone https://github.com/chr78rm/scaladoc-jar-test.git

The mentioned profile uses the currently promoted version (4.6.1) of the plugin and tries to attach scaladocs as depicted on the project site. I have investigated this a bit further. One reason why this cannot work is the wrong qualified class name given by the the method apidocMainClassName(). The correct qualified class name would be dotty.tools.scaladoc.Main. Using Scala3 this class is even separated from the scala compiler, that is an additional dependency (scaladoc_3) is needed to resolve this class. Unfortunately a potential fix in accordance with the outline of the given program structure of the plugin runs into classloader issues. scaladoc_3 uses the [java.lang.ClassLoader.getResourceAsStream(String)](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getClassLoader()) method but it seems that the invocations prepared by e.g. JavaMainCallerByFork don't provide a class loader when calling [getClassLoader()](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getClassLoader()). I am not an expert in classloader issues but it might well be that this issue is inevitable when using the currently provided invocations within the scala_maven_executions package.

Nevertheless, I have prepared a possible fix within the forked repo

https://github.com/chr78rm/scala-maven-plugin

of your plugin. Checkout the branch scaladoc3. The prepared fix uses the ProcessBuilder of the JDK to invoke a forked process and reflects as far as possible the given outline of the plugin code structure. The mentioned classloader issue can be triggered by checking out the branch classloader-issue. Both mentioned branches of the fork produce an artifact of the plugin with version 4.6.2-EXPERIMENTAL. This version is used by the other build profiles of the test project.

chr78rm commented 2 years ago

I have updated the test project to use the currently promoted version 4.6.2 within the build profile scala3-failure. Your latest commits have been integrated into the fork of your plugin. The branches scaladoc3 and classloader-issue are producing the version 4.6.3-EXPERIMENTAL now. Again, these version is referenced by the other build profiles of the reproducer.

slandelle commented 2 years ago

@chr78rm Thanks for reporting. Sadly, I can't review your fork:

I'll only be able to review if you can provide a PR that fixes the above points (all the more as I personally use neither Scala 3 nor Scaladoc). Regards

chr78rm commented 2 years ago

Thank you for your answer. I didn't want to put too much effort into this beforehand.

Please don't read too much into some of the commits on the branches. I needed to research your code and hence I had included some tracing statements at some time. After I understood the inner workings of the plugin well enough I had removed them again. Besides, I had updated the master branch of the forked repo to your latest commit (eac2c9bb). After that, I integrated your latest commits including the fix for issue #600 into the scaladoc3 branch by working on the output of git diff master..scaladoc3 --name-only. That is to say the files listed by git diff master..scaladoc3 --name-only contain the relevant changes for the possible fix at this instant. All your commits up to (eac2c9bb) are included in scaladoc3 now.

$ git diff master..scaladoc3 --name-only
pom.xml
src/main/java/scala_maven/ScalaDocMojo.java
src/main/java/scala_maven/ScalaMojoSupport.java
src/main/java/scala_maven_dependency/ArtifactIds.java
src/main/java/scala_maven_dependency/ArtifactIds4Scala2.java
src/main/java/scala_maven_dependency/ArtifactIds4Scala3.java
src/main/java/scala_maven_dependency/Context.java
src/main/java/scala_maven_dependency/Context4ScalaHome.java
src/main/java/scala_maven_dependency/Context4ScalaRemote.java
src/main/java/scala_maven_executions/ScalaDoc3Caller.java

I have attached a file with the full diffs to this comment but you might want to use an IDE for a better visual experience.

Additionally, I will replay the relevant diffs into a new branch rooted at your latest commit. I haven't looked into how you are organizing your integration tests yet. I'm using the mentioned test project as reference at this stage. E.g. the test project contains a build profile scala2 which can be used to verify that everything works in Scala2 as before. Another build profile works with Scala3.

Of course I can provide a pull request for this in due course.

diff-scaladoc3-master.txt

hohonuuli commented 1 year ago

Is there a way to disable arguments that are used to execute the scaladoc? Under the hood the Scala-maven-plugin scaladoc execution is running a command like:

 /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java -Xbootclasspath/a:/Users/brian/.m2/repository/com/google/protobuf/protobuf-java/3.7.0/protobuf-java-3.7.0.jar:[...] -classpath /Users/brian/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.8.0/scala-maven-plugin-4.8.0.jar scala_maven_executions.MainWithArgsInFile dotty.tools.dotc.Main /private/var/folders/d7/4ps3zfnx08n8gxq0wc9ydddw0000gq/T/scala-maven-2306955749493420450.args

The problem I seem to hit is that the temp file (e.g. /private/var/folders/d7/4ps3zfnx08n8gxq0wc9ydddw0000gq/T/scala-maven-2306955749493420450.args) contains the following lines:

-doc-format:html
-doc-title
"scommons 0.0.4_3 API"

These flags are are no longer appropriate for scaladoc 3 and when included produce the following:

bad option '-doc-format:html' was ignored
bad option '-doc-title' was ignored
source file not found: scommons 0.0.4_3 API
2 warnings found
1 error found

When I manually edit those out of the args file and rerun the command above, it works fine. So being able to strip those out of the args file might resolve this issue.

hohonuuli commented 1 year ago

Code that produces the bad flag is here

mnd999 commented 1 year ago

The flags are old and deprecated but they shouldn't break anything according to https://github.com/lampepfl/dotty/issues/11907. The problem seems to be the newline that gets generated between -doc-title and the module name itself. If the offending lines in the file become as follows, we get warnings but it doesn't fail anymore:

-doc-format:html
-doc-title "scommons 0.0.4_3 API"

This would seem like a reasonably quick workaround to do.

jam01 commented 2 months ago

Hi all, just hit this in 4.9.1. Is there any user-level workarounds?

edit: from what I've found so far, it seems in Scala 3 the plugin should be using a different class dotty.tools.scaladoc.Main? This is what gradle is using also

edit2: getting a little further with:

configuration:
  scaladocClassName: dotty.tools.scaladoc.Main
  dependencies:
    - groupId: org.scala-lang
      artifactId: scaladoc_3
      version: 3.3.3

forgive the yaml syntax (trying out polyglot)

I get the following with displayCmd 

... scala_maven_executions.MainWithArgsInFile dotty.tools.scaladoc.Main /tmp/scala-maven-8998361167489940263.args
bad option '-doc-format:html' was ignored
scaladoc supports only .tasty and .jar files, following files will be ignored: ...
java.lang.NullPointerException: Cannot invoke "java.lang.ClassLoader.getResourceAsStream(String)" because the return value of "java.lang.Class.getClassLoader()" is null
    at dotty.tools.scaladoc.renderers.Resources.renderResource(Resources.scala:557)
    at dotty.tools.scaladoc.renderers.Resources.renderResource$(Resources.scala:20)
    at dotty.tools.scaladoc.renderers.Renderer.renderResource(Renderer.scala:20)
    at dotty.tools.scaladoc.renderers.HtmlRenderer.renderResources$$anonfun$1(HtmlRenderer.scala:74)
    at scala.collection.immutable.List.flatMap(List.scala:293)
    at scala.collection.immutable.List.flatMap(List.scala:79)
    at dotty.tools.scaladoc.renderers.HtmlRenderer.renderResources(HtmlRenderer.scala:74)
    at dotty.tools.scaladoc.renderers.HtmlRenderer.render(HtmlRenderer.scala:48)
    at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:239)
    at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:69)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.Option.map(Option.scala:242)
    at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:73)
    at dotty.tools.scaladoc.Main.run(Main.scala:8)
    at dotty.tools.scaladoc.Main$.main(Main.scala:14)
    at dotty.tools.scaladoc.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at scala_maven_executions.MainHelper.runMain(MainHelper.java:156)
    at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:30)

edit3: adding args: ['-nobootcp'] prevents the error and the goal succeeds, but there's no actual docs in the -javadoc.jar. I'm confident it's because all the files passed listed are .scala which are not supported...

edit4: working configuration for scala 3.3.3

- id: attach-javadocs
  goals: ['doc-jar'] # on package
  configuration:
    args: ['-nobootcp'] # avoid NPE from getClass.getClassLoader
    scaladocClassName: dotty.tools.scaladoc.Main # scala3 class
    sourceDir: '${project.build.outputDirectory}' # use compiled classes
    includes: ['**/*.tasty'] # only tasty
    dependencies:
      - groupId: org.scala-lang
        artifactId: scaladoc_3
        version: 3.3.3
chr78rm commented 2 months ago

I can confirm that the configuration above does the trick for now. But it seems unreliable at best because it depends on an unsupported option (-nobootcp). Indeed I am getting the output: "bad option '-nobootcp' was ignored", but it wasn't actually ignored since removing the option causes the ClassLoader bug again.

The Compiler Options Lookup Table notes this option as not yet available in Scala 3.3.x and you cannot find this option within the ScalaDoc Settings at all. The same applies when listing the available options via scalac -help or scaladoc -help on the command line.

I had submitted a pull request (#611) two years ago which would have fixed this issue but the maintainers had chosen to ignore it.

The ScalaDoc generation additionally suffers from this bug at present: Scaladoc generates broken local links.

slandelle commented 2 months ago

I had submitted a pull request (https://github.com/davidB/scala-maven-plugin/pull/611) two years ago which would have fixed this issue but the maintainers had chosen to ignore it.

@chr78rm This is both not correct and offending. I requested that you clean up your PR (squash commit, remove extra dependency to one of your personal projects) and rebase it as it was conflicting with the main branch before I could review. This has not been done yet.

Please understand that I have some professional interest in maven + Scala 2 but have none for maven + Scala 3 and never will. Still, I'm willing to spend some of my personal time to help solve other people's problems. But only to a certain extend. Help me help you.

chr78rm commented 2 months ago

Please, calm down @slandelle . I have simply stated the fact that you chose to ignore the pull request, speak: to not integrate it - for whatever reasonable or unreasonable grounds you might have had. I really don't know how this comes as offending. Bye, Bye.

slandelle commented 2 months ago

I haven't chosen to ignore it. I've requested you to perform some changes before I could review it, which you haven't.

johanhaleby commented 2 months ago

I'm having the very same problem as we speak. I can't release my open-source project (rest-assured) because of this :( Will try the workarounds by @jam01 .

slandelle commented 2 months ago

@johanhaleby IMHO, if you want to provide modules for Scala 3, you should build and release them with sbt.

johanhaleby commented 2 months ago

@slandelle Yeah that might be so, but it would complicate my build process even further and I've not used sbt in 10+ years so I'm very rusty. I have other scala modules as well that use Scala 2, and I can generate javadoc there without any issues.

johanhaleby commented 2 months ago

It compiles and builds just fine btw, it's just javadoc generation that I struggle with. And I can't publish to maven central because of that.

jam01 commented 2 months ago

Thank you for your work @slandelle .

I gave sbt a try and wow ... Literally every project I've seen is configured differently. Basically my decision with my project was, if Maven can do it I'll use Maven, if not I'll figure out sbt. Thankfully I was able to figure it out through this project.

Additionally, the upcoming Maven 4 features should really leave little to be missed between Maven and other build tools (other than the official support from the Scala team).

Right now the only thing I'm missing is the ability to test Scala.js code in a JS environment (which sbt supports). But I can live with that. (Also someone figured it out in Gradle so I might venture a PR at some point if I really needed it).

johanhaleby commented 2 months ago

To reproduce my problems:

  1. Checkout the repo: git clone git@github.com:rest-assured/rest-assured.git
  2. mvn clean install -Dmaven.test.skip=true
  3. cd modules/scala-extensions
  4. mvn clean package -Prelease

It'll end this error:

[INFO] --- scala:4.9.2:doc-jar (attach-javadocs) @ scala-extensions ---
bad option '-doc-format:html' was ignored
bad option '-doc-title' was ignored
source file not found: scala-extensions 5.4.1-SNAPSHOT API
johanhaleby commented 2 months ago

I've also tried porting @jam01's workaround to maven like this:

<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.9.2</version>
<executions>   
    <execution>
        <id>attach-javadocs</id>
        <goals>
            <goal>doc-jar</goal>
        </goals>
        <configuration>
            <scaladocClassName>dotty.tools.scaladoc.Main</scaladocClassName>
            <sourceDir>${project.build.outputDirectory}</sourceDir>
            <args>-nobootcp</args>
            <includes>
                <include>**/*.scala</include>
            </includes>
            <additionalDependencies>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scaladoc_3</artifactId>
                    <version>3.4.2</version>
                </dependency>
            </additionalDependencies>
        </configuration>
    </execution>
</executions>

But now it fails with:

ava.lang.ClassNotFoundException: dotty.tools.scaladoc.Main
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at scala_maven_executions.MainHelper.runMain(MainHelper.java:144)
        at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:30)
slandelle commented 2 months ago

Let me say things bluntly (my opinion only).

This project is currently understaffed and the current's staff interests don't align with yours.

We are currently 2 persons:

I will never ever have an interest in using this plugin for scaladoc, Scala 3, Scala.js, Scala native or Spark. I'm willing to spend a bit of personal time reviewing clean PRs on topics that are of no concern to me but I won't investigate and fix such issues myself.

This code is open-source, if you don't want this plugin to die, I see 2 solutions:

jam01 commented 2 months ago

That's fair, this is the nature of open source. I appreciate your disclaimer.

Scala 3, Scala.js (minus tests in JS) and scaladoc (with my overriding config) are already supported out of the box. The only reason I didn't submit a PR is because I found my config to be pretty simple to merit the PR work for me at this point (again the nature of open source).

Perhaps you'd consider closing this issue since there's a workaround.

jam01 commented 2 months ago

@johanhaleby I tried to give your project a shot, but based on an error I got at compilation I think it requires Java 8 which I don't have locally.

That Main class is there on 3.4.2 so perhaps there's a dependency management issue? Also change your includes to be **/*.tasty instead of .scala.

johanhaleby commented 2 months ago

@slandelle I'm the first one to respect this, I know how difficult it can be to maintain an open-source project ❤️

johanhaleby commented 2 months ago

@jam01 Yeah it's still using Java 8. But the code in the module is a .scala file. Should I still use **/*.tasty?

jam01 commented 2 months ago

@johanhaleby oh I think your issue is you're using the wrong "dependencies" element, it shoud be dependencies and not additionalDependencies see here.

Yes, you should still use tasty.

johanhaleby commented 2 months ago

@jam01 Nice find, thank you! I changed to this:

 <execution>
    <id>attach-javadocs</id>
    <goals>
        <goal>doc-jar</goal>
    </goals>
    <configuration>
        <scaladocClassName>dotty.tools.scaladoc.Main</scaladocClassName>
        <sourceDir>${project.build.outputDirectory}</sourceDir>
        <args>-nobootcp</args>
        <includes>
            <include>**/*.tasty</include>
        </includes>
        <dependencies>
            <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scaladoc_3</artifactId>
                <version>3.4.2</version>
            </dependency>
        </dependencies>
    </configuration>
</execution>

but now it fails with:

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala_maven_executions.MainHelper.runMain(MainHelper.java:156)
        at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:30)
Caused by: java.lang.AbstractMethodError: Method dotty/tools/scaladoc/ScaladocSettings.dotty$tools$dotc$config$YSettings$_setter_$YccNoAbbrev_$eq(Ldotty/tools/dotc/config/Settings$Setting;)V is abstract
        at dotty.tools.scaladoc.ScaladocSettings.dotty$tools$dotc$config$YSettings$_setter_$YccNoAbbrev_$eq(ScaladocSettings.scala)
        at dotty.tools.dotc.config.YSettings.$init$(ScalaSettings.scala:316)
        at dotty.tools.scaladoc.ScaladocSettings.<init>(ScaladocSettings.scala:7)
        at dotty.tools.scaladoc.Scaladoc$.extract(Scaladoc.scala:99)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:53)
        at dotty.tools.scaladoc.Main.run(Main.scala:8)
        at dotty.tools.scaladoc.Main$.main(Main.scala:14)
        at dotty.tools.scaladoc.Main.main(Main.scala)

😢

johanhaleby commented 2 months ago

My bad! I had a different version of scaladoc_3 than the scala compiler. Now it freaking works!!!

Thanks A LOT!!! You definitely made my day.

mcallisto commented 1 week ago

Hello @jam01

can you please elaborate further on Scala.js support out of the box?

Scala 3, Scala.js (minus tests in JS) and scaladoc (with my overriding config) are already supported out of the box.

If scala-maven-plugin can compile and package Scala.js can you provide / point to a sample POM file or any useful reference?

jam01 commented 1 week ago

See if this helps https://github.com/jam01/json-schema/blob/0.1.0/js/pom.yaml

Also, consider opening a new issue as to not pollute this one that's about docs.