ClosedFileSystemException caused by null in ZipFileSystem.ensureOpen in IntelliJ 2024.2 with Azul JDK for project and Temurin JDK for compile server

fanf commented 1 month ago

steps

download IntelliJ 2024.2 community
clone rudder repository: git clone https://github.com/Normation/rudder.git
cd to rudder/webapp/sources and run ./rudder/rudder-web/src/main/build.sh to init the js/css env so that to avoid latter compilation errors unrelated to the problem
create a new project in IntelliJ in intellij from maven pom located in path_of_clone/rudder/webapp/sources/pom.xml
download Temurin 21 (https://adoptium.net/), Azul 21 (https://www.azul.com/downloads/#zulu) and JetBrains JDK (https://github.com/JetBrains/JetBrainsRuntime/releases), unarchive, and add them as a JDK in IntelliJ: Project Structure... > Platform Settings > SDK, then + and select the path to the previous root directories of the SDK ; then in "Project Settings > Project", choose either azul or jbr SDK ; Set the language level to 17
open IntelliJ settings and :
- change the incremental compiler to Zinc in Settings... > Build, Exec, Deploy > Compiler > Scala Compiler top of the screen.
- force the Scala Compile Server to Temurin SDK in the drop down list in Settings... > Build, Exec, Deploy > Compiler > Scala Compiler > Scala Compile Server then JVM > JDK and change stack size to -Xss8m ;
try a compilation: CTRL+F9 : it fails with compilation errors.

If you change the incremental compiler to IDEA it works. It used to work with IntelliJ 2024.1.x. It works if Temurin is chosen.

problem

Note: I started by opening a ticket on JetBrains bug tracker (https://youtrack.jetbrains.com/issue/SCL-22939), but the root cause might be in Zinc thus that ticket too.

The compilation fails on the last project (rudder-web) with two stack traces which seems to indicate that the problem might be in Zinc (see at the end).

I found an other project which had a very similar problem: https://github.com/jobrunr/jobrunr/issues/427 The cause was that the classpath resource manager was not thread safe as show in the correction: https://github.com/jobrunr/jobrunr/commit/248e098345d49eabd85bd267f0183c99ece49847

Exception 1 :

scala: ## Exception when compiling 140 sources to /home/fanf/java/workspaces/rudder-project/rudder/webapp/sources/rudder/rudder-web/target/classes
java.nio.file.ClosedFileSystemException
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
java.base/java.nio.file.Files.readAttributes(Files.java:1853)
java.base/java.nio.file.Files.size(Files.java:2462)
sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
scala.util.Try$.apply(Try.scala:217)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
com.facebook.nailgun.NGSession.run(NGSession.java:198)

Exception 2 :

scala: Compilation failed when compiling to: /home/fanf/java/workspaces/rudder-project/rudder/webapp/sources/rudder/rudder-web/target/classes
  null
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
  jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
  jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
  java.base/java.nio.file.Files.readAttributes(Files.java:1853)
  java.base/java.nio.file.Files.size(Files.java:2462)
  sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
  sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
  sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
  sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
  sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
  sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
  sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
  sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
  java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
  sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
  sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
  sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
  scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
  java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
  scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
  sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
  sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
  scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
  scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
  scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
  sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
  scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
  scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
  scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
  sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
  sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
  sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
  sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
  sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
  sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
  sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
  sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
  sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
  sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
  sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
  sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
  sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
  sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
  sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
  sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
  sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
  scala.util.Try$.apply(Try.scala:217)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
  org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
  org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
  org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
  org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
  org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
  org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
  org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
  org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
  org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
  java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base/java.lang.reflect.Method.invoke(Method.java:580)
  com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
  com.facebook.nailgun.NGSession.run(NGSession.java:198)

vasilmkd commented 1 month ago

Since Maven also uses Zinc through scala-maven-plugin`, it might be possible to reproduce the exception by just using Maven in the terminal with the JetBrains JDK as above.

fanf commented 1 month ago

And... No. But maven plugin is extremely linear and non concurrent contrary to intellij that tries to parallelize as much as possible (for our joy), so if it's a threadsafety problem, it might be consistant (but it's just a guessing game here).

eed3si9n commented 1 month ago

download JetBrains JDK (https://github.com/JetBrains/JetBrainsRuntime/releases)

If this doesn't happen on Zulu or Eclipse Temurin, I don't think we'd work on this issue. If there's some logical error in the concurrency handling, we'd be happy to review pull requests though.

fanf commented 1 month ago

@eed3si9n Oh ! I didn't test with azul, and actually I have the same error ! So I retested with Temurin... And now I get the error too. In the meantime, I deleted my whole .idea and started from a fresh one, so perhaps changed with that. Let me try a with a fresh user, fresh clone, fresh jvm install so that the reproduction step and qualification is actually OK.

fanf commented 1 month ago

@eed3si9n so, fresh linux user, fresh clone, fresh install of intellij, fresh everything. I can reproduce the problem with Azul zulu JDK with the updated steps in top post, ie with setting the scala compile server to use temurin 21 and the project to use zulu 21. If I set the scala compile server to zulu too, then the error disappear. So it seems that the problem arise when the compiler server JDK is not the same as the project.

I can't test setting JBR as a scala compile server JDK because then I get a NoClassDefFoundError: INIT (perhaps a limitation of JBR).

So I don't know if the problem is in IntelliJ or zinc, but I does not seems to be directly linked to JBR.

Friendseeker commented 18 hours ago

For any guys investigating the issue but got hit with the following error when running ./rudder/rudder-web/src/main/build.sh

npm error code 1
npm error path /Users/jiahuitan/rudder/webapp/sources/rudder/rudder-web/src/main/node_modules/elm
npm error command failed
npm error command sh -c node install.js
npm error -- ERROR -----------------------------------------------------------------------
npm error
npm error I tried to get `elm` from @elm_binaries/darwin_arm64, but something went wrong.
npm error This can happen if you use the "--omit=optional" or "--no-optional" npm flag, or
npm error if your "node_modules" folder was copied over from a different computer (or VM).

Try first do a npm install elm under the directory rudder/webapp/sources/rudder/rudder-web/src/main/ and that bypasses the issue for me

sbt / zinc

ClosedFileSystemException caused by null in ZipFileSystem.ensureOpen in IntelliJ 2024.2 with Azul JDK for project and Temurin JDK for compile server #1383

steps

problem