search

eclipse-jdt / eclipse.jdt.core

Eclipse Public License 2.0
156 stars 123 forks source link

BatchCompilerTest.test248_jar_ref_in_jar fails on Windows #2766

Closed iloveeclipse closed 3 weeks ago

iloveeclipse commented 1 month ago

After merge of https://github.com/eclipse-jdt/eclipse.jdt.core/pull/2551 we see that BatchCompilerTest.test248_jar_ref_in_jar fails on Windows.

https://download.eclipse.org/eclipse/downloads/drops4/I20240725-1800/testresults/html/org.eclipse.jdt.core.tests.compiler_ep433I-unit-win32-java17_win32.win32.x86_64_17.html

Unexpected problems [out: ][err: ---------- 1. ERROR in C:\Users\genie.releng\AppData\Local\Temp\comptest\run.1721963082240\regression\src\p\X.java (at line 4) A a; ^ A cannot be resolved to a type ---------- 2. ERROR in C:\Users\genie.releng\AppData\Local\Temp\comptest\run.1721963082240\regression\src\p\X.java (at line 5) G g; ^ G cannot be resolved to a type ---------- 2 problems (2 errors) ]

junit.framework.AssertionFailedError: Unexpected problems [out: ][err: ----------
1. ERROR in C:\Users\genie.releng\AppData\Local\Temp\comptest\run.1721963082240\regression\src\p\X.java (at line 4)
A a;
^
A cannot be resolved to a type
----------
2. ERROR in C:\Users\genie.releng\AppData\Local\Temp\comptest\run.1721963082240\regression\src\p\X.java (at line 5)
G g;
^
G cannot be resolved to a type
----------
2 problems (2 errors)
]
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest.runTest(AbstractBatchCompilerTest.java:651)
at org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest.runConformTest(AbstractBatchCompilerTest.java:478)
at org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest.test248_jar_ref_in_jar(BatchCompilerTest.java:8979)
iloveeclipse commented 1 month ago

On Windows I see that the tests start to misbehave after test test245_jar_ref_in_jar - the test succeeds, but report this warning at the end and following 6 tests are failing.

org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest.test245_jar_ref_in_jar - 1.8

WARNING in test: org.eclipse.jdt.core.tests.compiler.regression.AbstractRegressionTest - tearDown
    - problems occured while deleting C:\tmp\comptest\run.1722611893110\lib\lib8.jar
    - stack trace:
        -> org.eclipse.jdt.core.tests.util.Util.waitUntilFileDeleted(Util.java:1357)
        -> org.eclipse.jdt.core.tests.util.Util.delete(Util.java:536)
        -> org.eclipse.jdt.core.tests.util.Util.flushDirectoryContent(Util.java:800)
        -> org.eclipse.jdt.core.tests.compiler.regression.AbstractRegressionTest.tearDown(AbstractRegressionTest.java:4143)
    - lib file info: read, write, exist, dir
        + children: 
            - lib8.jar file info: read, write, exist, file
            - lib9.jar file info: read, write, exist, file
            - s file info: read, write, exist, dir
                + children: 
                    - lib6.jar file info: read, write, exist, file

    !!! ERROR: C:\tmp\comptest\run.1722611893110\lib\lib8.jar was never deleted even after having waited 10000ms!!!

org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest.test246_jar_ref_in_jar - 1.8

I assume org.eclipse.jdt.internal.compiler.batch.Main.performCompilation() doesn't properly clean up opened nested jars and that seem to happen now after update to 1.8 because we probably run into some code path that was never entered with Java 1.4 compilation mode.

The code is called from org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest.invokeCompiler(PrintWriter, PrintWriter, Object, TestCompilationProgress) and the failing tests in question have rather complex project libraries setup involving nested jar libraries.

The test failures are actually follow up problems of the test setup/teardown being unable to delete / create required jars.

iloveeclipse commented 1 month ago

On Linux, thanks to lsof, I see that the code in question works as expected, and the opened file descriptor is closed in org.eclipse.jdt.internal.compiler.batch.ClasspathJar.reset() for the lib/lib8.jar with this stack:

ClasspathJar.reset() line: 281  
FileSystem.cleanup() line: 430  
Main.performCompilation() line: 4726    
Main.compile(String[]) line: 1711   
BatchCompilerTest(AbstractBatchCompilerTest).invokeCompiler(PrintWriter, PrintWriter, Object, AbstractBatchCompilerTest$TestCompilationProgress) line: 672  
BatchCompilerTest(AbstractBatchCompilerTest).runTest(boolean, String[], Object, String, String, boolean, AbstractBatchCompilerTest$TestCompilationProgress) line: 589   
BatchCompilerTest(AbstractBatchCompilerTest).runConformTest(String[], String, String, String, boolean) line: 475    
BatchCompilerTest.test245_jar_ref_in_jar() line: 8902

So at least from Linux point of view, JDT works as expected...

jukzi commented 3 weeks ago

It is an error in JDK that JarFile and its underlying ZIpFile ist never closed when System.getSecurityManager()==null in jdk.internal.loader.URLClassPath.JarLoader.checkJar(JarFile)

stacktrace when it happens (with JDK 21.0.2+13):

Thread [main] (Suspended)   
    jdk.internal.loader.URLClassPath$JarLoader.checkJar(java.util.jar.JarFile) line: 798    
    jdk.internal.loader.URLClassPath$JarLoader.getJarFile(java.net.URL) line: 808   
    jdk.internal.loader.URLClassPath$JarLoader$1.run() line: 774    
    jdk.internal.loader.URLClassPath$JarLoader$1.run() line: 768    
    java.security.AccessController.executePrivileged(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext, java.lang.Class<?>) line: 809  
    java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext) line: 714   
    jdk.internal.loader.URLClassPath$JarLoader.ensureOpen() line: 767   
    jdk.internal.loader.URLClassPath$JarLoader.<init>(java.net.URL, java.net.URLStreamHandler, java.util.HashMap<java.lang.String,jdk.internal.loader.URLClassPath.Loader>, java.security.AccessControlContext) line: 734   
    jdk.internal.loader.URLClassPath$3.run() line: 497  
    jdk.internal.loader.URLClassPath$3.run() line: 479  
    java.security.AccessController.executePrivileged(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext, java.lang.Class<?>) line: 809  
    java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext) line: 714   
    jdk.internal.loader.URLClassPath.getLoader(java.net.URL) line: 478  
    jdk.internal.loader.URLClassPath.getLoader(int) line: 446   
    jdk.internal.loader.URLClassPath$1.next() line: 342 
    jdk.internal.loader.URLClassPath$1.hasMoreElements() line: 353  
    java.net.URLClassLoader$3$1.run() line: 662 
    java.net.URLClassLoader$3$1.run() line: 660 
    java.security.AccessController.executePrivileged(java.security.PrivilegedAction<T>, java.security.AccessControlContext, java.lang.Class<?>) line: 778   
    java.security.AccessController.doPrivileged(java.security.PrivilegedAction<T>, java.security.AccessControlContext) line: 400    
    java.net.URLClassLoader$3.next() line: 659  
    java.net.URLClassLoader$3.hasMoreElements() line: 684   
    java.lang.CompoundEnumeration<E>.next() line: 2756  
    java.lang.CompoundEnumeration<E>.hasMoreElements() line: 2765   
    java.util.ServiceLoader$LazyClassPathLookupIterator<T>.nextProviderClass() line: 1210   
    java.util.ServiceLoader$LazyClassPathLookupIterator<T>.hasNextService() line: 1228  
    java.util.ServiceLoader$LazyClassPathLookupIterator<T>.hasNext() line: 1273 
    java.util.ServiceLoader$2.hasNext() line: 1309  
    java.util.ServiceLoader$3.hasNext() line: 1393  
    org.eclipse.jdt.internal.compiler.apt.dispatch.BatchAnnotationProcessorManager.discoverNextProcessor() line: 177    
    org.eclipse.jdt.internal.compiler.apt.dispatch.RoundDispatcher.round() line: 118    
    org.eclipse.jdt.internal.compiler.apt.dispatch.BatchAnnotationProcessorManager(org.eclipse.jdt.internal.compiler.apt.dispatch.BaseAnnotationProcessorManager).processAnnotations(org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration[], org.eclipse.jdt.internal.compiler.lookup.ReferenceBinding[], boolean) line: 172    
    org.eclipse.jdt.internal.compiler.Compiler.processAnnotations() line: 972   
    org.eclipse.jdt.internal.compiler.Compiler.compile(org.eclipse.jdt.internal.compiler.env.ICompilationUnit[], boolean) line: 449 
    org.eclipse.jdt.internal.compiler.Compiler.compile(org.eclipse.jdt.internal.compiler.env.ICompilationUnit[]) line: 425  
    org.eclipse.jdt.internal.compiler.batch.Main.performCompilation() line: 4710    
    org.eclipse.jdt.internal.compiler.batch.Main.compile(java.lang.String[]) line: 1711 
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest).invokeCompiler(java.io.PrintWriter, java.io.PrintWriter, java.lang.Object, org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest$TestCompilationProgress) line: 672   
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest).runTest(boolean, java.lang.String[], java.lang.Object, java.lang.String, java.lang.String, boolean, org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest$TestCompilationProgress) line: 589  
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(org.eclipse.jdt.core.tests.compiler.regression.AbstractBatchCompilerTest).runConformTest(java.lang.String[], java.lang.String, java.lang.String, java.lang.String, boolean) line: 475  
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest.test245_jar_ref_in_jar() line: 608 
    java.lang.invoke.LambdaForm$DMH.0x000001d130234000.invokeVirtual(java.lang.Object, java.lang.Object) line: not available    
    java.lang.invoke.LambdaForm$MH.0x000001d130098800.invoke(java.lang.Object, java.lang.Object) line: not available    
    java.lang.invoke.Invokers$Holder.invokeExact_MT(java.lang.Object, java.lang.Object, java.lang.Object) line: not available   
    jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.lang.Object, java.lang.Object[]) line: 153  
    jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.lang.Object, java.lang.Object[]) line: 103  
    java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 580    
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(junit.framework.TestCase).runTest() line: 177  
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(org.eclipse.jdt.core.tests.junit.extension.TestCase).runTest() line: 972   
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(junit.framework.TestCase).runBare() line: 142  
    junit.framework.TestResult$1.protect() line: 122    
    junit.framework.TestResult.runProtected(junit.framework.Test, junit.framework.Protectable) line: 142    
    junit.framework.TestResult.run(junit.framework.TestCase) line: 125  
    org.eclipse.jdt.core.tests.compiler.regression.BatchCompilerTest(junit.framework.TestCase).run(junit.framework.TestResult) line: 130    
    org.eclipse.jdt.core.tests.compiler.regression.RegressionTestSetup(junit.framework.TestSuite).runTest(junit.framework.Test, junit.framework.TestResult) line: 241   
    org.eclipse.jdt.core.tests.compiler.regression.RegressionTestSetup(junit.framework.TestSuite).run(junit.framework.TestResult) line: 236 
    org.eclipse.jdt.core.tests.compiler.regression.RegressionTestSetup(org.eclipse.jdt.core.tests.util.CompilerTestSetup).run(junit.framework.TestResult) line: 59  
    junit.framework.TestSuite.runTest(junit.framework.Test, junit.framework.TestResult) line: 241   
    junit.framework.TestSuite.run(junit.framework.TestResult) line: 236 
    org.junit.internal.runners.SuiteMethod(org.junit.internal.runners.JUnit38ClassRunner).run(org.junit.runner.notification.RunNotifier) line: 90   
    org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(org.eclipse.jdt.internal.junit.runner.TestExecution) line: 93    
    org.eclipse.jdt.internal.junit.runner.TestExecution.run(org.eclipse.jdt.internal.junit.runner.ITestReference[]) line: 40    
    org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner(org.eclipse.jdt.internal.junit.runner.RemoteTestRunner).runTests(java.lang.String[], java.lang.String, org.eclipse.jdt.internal.junit.runner.TestExecution) line: 530 
    org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner(org.eclipse.jdt.internal.junit.runner.RemoteTestRunner).runTests(org.eclipse.jdt.internal.junit.runner.TestExecution) line: 758   
    org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner(org.eclipse.jdt.internal.junit.runner.RemoteTestRunner).run() line: 453   
    org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner.main(java.lang.String[]) line: 83 
    org.eclipse.pde.internal.junit.runtime.CoreTestApplication.start(org.eclipse.equinox.app.IApplicationContext) line: 28  
    org.eclipse.equinox.internal.app.EclipseAppHandle.run(java.lang.Object) line: 208   
    org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(java.lang.Object) line: 143 
    org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(java.lang.Object) line: 109  
    org.eclipse.core.runtime.adaptor.EclipseStarter.run(java.lang.Object) line: 439 
    org.eclipse.core.runtime.adaptor.EclipseStarter.run(java.lang.String[], java.lang.Runnable) line: 271   
    java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, java.lang.Object, java.lang.Object) line: not available   
    java.lang.invoke.LambdaForm$MH.0x000001d130005c00.invoke(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object) line: not available    
    java.lang.invoke.LambdaForm$MH.0x000001d130006000.invokeExact_MT(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object) line: not available  
    jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.lang.Object, java.lang.Object[]) line: 155  
    jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.lang.Object, java.lang.Object[]) line: 103  
    java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 580    
    org.eclipse.equinox.launcher.Main.invokeFramework(java.lang.String[], java.net.URL[]) line: 668 
    org.eclipse.equinox.launcher.Main.basicRun(java.lang.String[]) line: 605    
    org.eclipse.equinox.launcher.Main.run(java.lang.String[]) line: 1481    
    org.eclipse.equinox.launcher.Main.main(java.lang.String[]) line: 1454

And even the Cleaner does not automatically close the ZipFile as soon as it become only Phantomrechable (is only referenced by PhantomCleanable) but only if - by chance - a GC happens: image Manually triggering a GC in that state using VisualVM invokes the cleaner. That can be also used as workaround by calling System.gc() when waiting for the File to be deleted.

jukzi commented 3 weeks ago

see https://github.com/openjdk/jdk/blob/38bd8a36704a962f0ad1052fd2ec150a61663256/src/java.base/share/classes/jdk/internal/loader/URLClassPath.java#L780

jukzi commented 3 weeks ago

errors gone in https://download.eclipse.org/eclipse/downloads/drops4/I20240814-1800/testResults.php

jukzi commented 3 weeks ago

reported to JDK https://bugs.openjdk.org/browse/JDK-8338445