com-lihaoyi / mill

Mill is a fast JVM build tool that supports Java and Scala. 2-4x faster than Gradle and 5-10x faster than Maven for common workflows, Mill aims to make your project’s build process performant, maintainable, and flexible
https://mill-build.org/
MIT License
2.14k stars 341 forks source link

Error in IntelliJ about Multiple 'scala3-library*.jar' files when using scala 3.x and scalajs #3344

Closed malcolmredheron closed 4 weeks ago

malcolmredheron commented 2 months ago

When I use GenIdea in a mill/scalajs project and then build in intellij, it works fine if build.sc uses scala 2.13.14 but for every version of scala 3 that I have tried IJ gives me errors of the form scala: Multiple 'scala3-library*.jar' files (scala3-library_3-3.3.1.jar, scala3-library_sjs1_3-3.3.1.jar) in Scala compiler classpath in Scala SDK scala-SDK-3.3.1

https://github.com/malcolmredheron/mill-scalajs-experiment contains code and exact steps that demonstrate the problem. This happens for me with mill 0.11.8, 0.11.10, 0.11.11 and 0.11.11-1-066ea7, which is the most recent commit that I could find a build for

lefou commented 2 months ago

Thank you for this report. We introduced the "scala-SDK" entry in Mill 0.11.8. It looks like some handling for ScalaJS is missing.

lefou commented 2 months ago

I just checked your example project.

> mill --version
Mill Build Tool version 0.11.11
Java version: 17.0.11, vendor: Eclipse Adoptium, runtime: /opt/openjdk-bin-17.0.11_p9
Default locale: de_DE, platform encoding: UTF-8
OS name: "Linux", version: 6.1.90-gentoo-x86_64, arch: amd64

> mill mill.idea.GenIdea/idea
[1/1] mill.idea.GenIdea.idea 
Analyzing modules ...
[build.sc] [1/1] mill.idea.GenIdea.idea > [7/9] sources.super.mill.scalalib.JavaModule.sources 
Writing 111 IDEA project files to /tmp/mill-scalajs-experiment/.idea ...

> mill show __.scalaCompilerClasspath
[1/1] show 
[
  "qref:v1:878e902e:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.3.1/scala3-compiler_3-3.3.1.jar",
  "qref:v1:7d4aa14c:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_sjs1_3/3.3.1/scala3-library_sjs1_3-3.3.1.jar",
  "qref:v1:14776eac:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.3.1/scala3-interfaces-3.3.1.jar",
  "qref:v1:3a303a0e:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.3.1/scala3-library_3-3.3.1.jar",
  "qref:v1:4543c999:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.3.1/tasty-core_3-3.3.1.jar",
  "qref:v1:ab49893c:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.5.0-scala-1/scala-asm-9.5.0-scala-1.jar",
  "qref:v1:9614657d:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler-interface/1.3.5/compiler-interface-1.3.5.jar",
  "qref:v1:4ec30e0c:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-reader/3.19.0/jline-reader-3.19.0.jar",
  "qref:v1:83615674:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal/3.19.0/jline-terminal-3.19.0.jar",
  "qref:v1:606a0c92:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal-jna/3.19.0/jline-terminal-jna-3.19.0.jar",
  "qref:v1:e1bf2dc6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-library_2.13/1.12.0/scalajs-library_2.13-1.12.0.jar",
  "qref:v1:a94bfc28:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar",
  "qref:v1:a26d3af1:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.7.0/protobuf-java-3.7.0.jar",
  "qref:v1:c491d6fc:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/util-interface/1.3.0/util-interface-1.3.0.jar",
  "qref:v1:b412aca9:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.3.1/jna-5.3.1.jar",
  "qref:v1:283320ce:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-javalib/1.12.0/scalajs-javalib-1.12.0.jar"
]

> cat .idea/libraries/scala_SDK_3_3_1.xml 
<component name="libraryTable">
    <library name="scala-SDK-3.3.1" type="Scala">
        <properties>
            <language-level>Scala_3_3</language-level>
            <compiler-classpath>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.7.0/protobuf-java-3.7.0.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.3.1/jna-5.3.1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-reader/3.19.0/jline-reader-3.19.0.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal-jna/3.19.0/jline-terminal-jna-3.19.0.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal/3.19.0/jline-terminal-3.19.0.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-javalib/1.12.0/scalajs-javalib-1.12.0.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-js/scalajs-library_2.13/1.12.0/scalajs-library_2.13-1.12.0.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.5.0-scala-1/scala-asm-9.5.0-scala-1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.3.1/scala3-compiler_3-3.3.1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.3.1/scala3-interfaces-3.3.1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.3.1/scala3-library_3-3.3.1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_sjs1_3/3.3.1/scala3-library_sjs1_3-3.3.1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.3.1/tasty-core_3-3.3.1.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler-interface/1.3.5/compiler-interface-1.3.5.jar"/>
                <root url="file://$USER_HOME$/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/util-interface/1.3.0/util-interface-1.3.0.jar"/>
            </compiler-classpath>
        </properties>
    </library>
</component>

I think the scala-SDK entry should be a scala-js-SDK, to avoid collisions in projects that cross compile to JVM and JS.

But the compiler classpath that Mill uses looks at a first glance identical to the one in the .idea/libraries/scala_SDK_3_3_1.xml, except the ordering.

So the question is, who is wrong? Obviously Mill compiles successfully. So is IntelliJ reporting false here or should the classpath indeed contain less entries? @lolgab, WDYT, anything redundant or too much here?

lefou commented 2 months ago

From looking at the .idea/libraries/scala_SDK_3_3_1.xml file, it's most likely, that the <library> type should not be Scala but some ScalaJS specific type.

<component name="libraryTable">
    <library name="scala-SDK-3.3.1" type="Scala">
        <properties>
            <language-level>Scala_3_3</language-level>
            <compiler-classpath>
<!-- -->
            </compiler-classpath>
        </properties>
    </library>
</component>

@unkarjedy, do you have any insights?

unkarjedy commented 2 months ago

It's most likely, that the type should not be Scala but some ScalaJS specific type.

There is no any ScalaJS specific library type. Also Scala JS support is pretty limited: https://youtrack.jetbrains.com/issues/SCL?q=%22scala%20js%22%20%23Unresolved%20


Scala Plugin has the best support for SBT projects. If in doubts I would suggest to create a similar SBT project with similar structure and see how the project structure looks there. If something can't be done for SBT, most likely it can't be done for other build tools.

To fix this particular issue, just don't add Scala JS specific jars to the compiler classpath. E.g. sbt show scalaInstance doesn't mix Scala JS libraries into the SDK

lefou commented 2 months ago

@unkarjedy Thank you! Does that also mean, IntelliJ isn't using any JS runtime to run tests?

malcolmredheron commented 2 months ago

Ok, I'll try with bsp and report back

In the meantime, if mill stops adding scalajs jars to the class path (" just don't add Scala JS specific jars to the compiler classpath") then how will it be possible to build code that uses any browser apis?


Sent with Superhuman ( https://superhuman.com/refer/cmsqjaxn ). Please favor brief email ( http://three.sentenc.es/ ) over chat, especially for tasks. For chat (+1-415-235-0687), WhatsApp is best. Never Signal unless there are legal issues.

On Wed, Aug 07, 2024 at 3:48 AM, Tobias Roeser < @.*** > wrote:

@ unkarjedy ( https://github.com/unkarjedy ) Thank you! Does that also mean, IntelliJ isn't using any JS runtime to run tests?

— Reply to this email directly, view it on GitHub ( https://github.com/com-lihaoyi/mill/issues/3344#issuecomment-2273178678 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAAOIXL6OMBHHSWFMQSCNFLZQH3OLAVCNFSM6AAAAABMDUTUGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZTGE3TQNRXHA ). You are receiving this because you authored the thread. Message ID: <com-lihaoyi/mill/issues/3344/2273178678 @ github. com>

lefou commented 2 months ago

Would be nice if someone with sbt experience can prepare a sample project.

malcolmredheron commented 2 months ago

I tried with bsp with these steps:

IJ compiles everything fine but I remembered why I don't like using bsp: it only shows compilation errors (or removes them when they are fixed) when I manually compile. Using GenIdea I get immediate feedback on errors. Unless there's a fix for this, I'd love to get GenIdea working

Also, my real project uses Mill (0.11.8, but I think that it works with 0.11.11 too), Scala 3.3.1 and ScalaJS 1.6.0, and even has a module that's shared between the JS and JVM builds. It works great (after a lot of trouble figuring out how to do the shared module). So GenIdea can work even with Scala 3.x. The problems started when I tried to upgrade to Scala 3.4. Pairing things back to find the source of the error led to this. I don't yet understand why my real project works with Scala 3.3.1 when this one does not

unkarjedy commented 2 months ago

Does that also mean, IntelliJ isn't using any JS runtime to run tests?

Yes

malcolmredheron commented 1 month ago

I've updated my demo repo to Mill 0.11.12 and get the same error as before. Is that expected? I can't tell whether https://github.com/com-lihaoyi/mill/pull/3345 was meant to fix all of this

malcolmredheron commented 1 month ago

Strangely, my real project uses mill 0.11.8 and a mixture of ScalaJs and regular Scala, and builds fine, both in mill and in IntelliJ (using GenIdea to make the project). But upgrading to 0.11.12 gets me the same problems that my example project demonstrates

hombre commented 4 weeks ago

With mill version 0.11.12 - 0.12.0-RC2 the example project [mill-scalajs-experiment] compiles in IntelliJ IDEA 2024.2.3 (Ultimate Edition) by removing the scala-library from scala_js_SDK.

mill -j 0 mill.idea.GenIdea/idea; sed -i '/scala3-library_3/d' .idea/libraries/scala_jsSDK*.xml

lefou commented 4 weeks ago

With mill version 0.11.12 - 0.12.0-RC2 the example project [mill-scalajs-experiment] compiles in IntelliJ IDEA 2024.2.3 (Ultimate Edition) by removing the scala-library from scala_js_SDK.

mill -j 0 mill.idea.GenIdea/idea; sed -i '/scala3-library_3/d' .idea/libraries/scala_jsSDK*.xml

So, you mean removing the scala3-library_3 or scala-library or both?

lefou commented 4 weeks ago

Although I think IDEA is misbehaving, if this solves the issue, I can implement this specific logic.

hombre commented 4 weeks ago

Only scala3-library.

malcolmredheron commented 1 week ago

Thank you! I can confirm that this fixed my real project, in addition to the sample that I linked to. (And sorry about the slow reply. I had to figure out some path issues caused by one of the other changes between 0.11.11 and 0.12-M3.)