Deadlock during startup between classloader/bundle activator/GradleClasspathContainerInitializer/workspace

[abstratt]

Expected Behavior

Eclipse should start up with no lock-ups.

Current Behavior

Eclipse some times freezes during startup. Deadlock is only broken due to OSGi classloader deadlock workaround (see attached log).

Context

Sometimes, Eclipse with Buildship will freeze for 10s of seconds during startup. Eventually, it will unfreeze, and a timeout exception will be logged.

From looking at the thread dump at the time of the freeze (see attached), it seems there is a multiway lock-up that involves:

• Buildship's DefaultModelPersistence "local cache"
• Equinox classloading and Buildship's CorePlugin startup
• JDT classpath container initialization
• Workspace resource delta notification
• Workspace refresh job

Steps to Reproduce

Start Eclipse with a workspace that contains a Gradle-based Java project. If it doesn't happen, try again (I may be missing some required UI-related step, but hopefully the thread dump attached will be helpful).

Your Environment

As shown on about dialog: Eclipse Version: 2021-09 (4.21.0) Build id: 20210910-1417

As shown on Eclipse log header:

eclipse.buildId=4.21.0.I20210906-0500 java.version=15.0.1 java.vendor=AdoptOpenJDK BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US Framework arguments: -product org.eclipse.epp.package.committers.product Command-line arguments: -os linux -ws gtk -arch x86_64 -product org.eclipse.epp.package.committers.product

Buildship: Eclipse Plug-ins for Gradle 3.1.5.v20210113-0929 org.eclipse.buildship.feature.group Eclipse Buildship

buildship-freeze.stacktrace.txt buildship-freeze.log

[UdoW]

This happens regularily in my installation. Buildship Version 3.1.5, Eclipse versions from 2021-09 until 2022-03.

I have to wait several minutes. Sometimes Eclipse recovers and I can continue working, sometimes I have to kill it and try to restart it again.

Very annoying.

I attached my stack traces from the Eclipse Error Log:

exception_stack_trace1.txt exception_stack_trace2.txt exception_stack_trace3.txt exception_stack_trace4.txt exception_stack_trace5.txt exception_stack_trace6.txt

[vogella]

Not sure if platform can do something about this. cc @jkubiz @iloveeclipse

@donat This seems to be a issue which really harms the usage of the Eclipse IDE and Buildship.

[vogella]

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=579369#c11

[gk10gk]

Same problem in my installation, timeout after 6 minutes!

!MESSAGE While loading class "org.eclipse.buildship.core.internal.preferences.PersistentModelConverter$12", thread "Thread[Worker-6: Load persistent model for all projects,5,main]" timed out waiting (30493ms) for thread "Thread[Worker-2: Building,5,main]" to finish starting bundle "org.eclipse.buildship.core_3.1.5.v20210113-0929 [360]". To avoid deadlock, thread "Thread[Worker-6: Load persistent model for all projects,5,main]" is proceeding but "org.eclipse.buildship.core.internal.preferences.PersistentModelConverter$12" may not be fully initialized.

Versions: eclipse.buildId=4.23.0.I20220308-0310 java.version=11.0.14.1 Buildship: 3.1.5.v20210113-0929

stacktrace.txt

[iloveeclipse]

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=579369#c12 explanation. Stacks here show that Buildship bundle loads something on start() of the bundle, hangs and so the entire workspace is blocked because the code in which buildship bundle is initialized performs workspace refresh and so locks the workspace for any other thread. This blocking load of whatever it needs should be done in a job, and NOT on start() directly.

ThreadId: 55 ThreadName: Worker-4: Refreshing workspace ThreadState: WAITING
  Blocked On: com.google.common.util.concurrent.SettableFuture@316871c3 LockOwnerId: -1 LockOwnerName: null
  Synchronizers Locked: 
    java.util.concurrent.locks.ReentrantLock$NonfairSync@5490905b
  Monitors Locked: none
  Stack Trace: 
    java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
    java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:537)
    com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:104)
    com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240)
    com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3582)
    com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2175)
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2162)
    com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
    com.google.common.cache.LocalCache.get(LocalCache.java:3951)
    com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
    com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
    com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4941)
    org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence.loadModel(DefaultModelPersistence.java:67)
    org.eclipse.buildship.core.internal.CorePlugin.scheduleSynchronizationForAbsentModels(CorePlugin.java:223)
    org.eclipse.buildship.core.internal.CorePlugin.start(CorePlugin.java:120)
    org.eclipse.osgi.internal.framework.BundleContextImpl.lambda$2(BundleContextImpl.java:808)
    org.eclipse.osgi.internal.framework.BundleContextImpl$$Lambda$62/0x000000080013d840.run(Unknown Source)
    java.base@11.0.13/java.security.AccessController.doPrivileged(Native Method)
    org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:802)
    org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:759)
    org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1019)
    org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
    org.eclipse.osgi.container.Module.doStart(Module.java:605)
    org.eclipse.osgi.container.Module.start(Module.java:468)
    org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:404)
    org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
    org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)
    org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:345)
    org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:388)
    org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:467)
    org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:170)
    java.base@11.0.13/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:616)
    org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:196)
    org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:920)
    org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:246)
    org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:63)
    org.eclipse.jdt.core.JavaCore.computeClasspathContainerInitializer(JavaCore.java:3824)
    org.eclipse.jdt.core.JavaCore.getClasspathContainerInitializer(JavaCore.java:3800)
    org.eclipse.jdt.internal.core.JavaModelManager.initializeContainer(JavaModelManager.java:3125)
    org.eclipse.jdt.internal.core.JavaModelManager$10.run(JavaModelManager.java:3028)
    org.eclipse.jdt.internal.core.JavaModelManager.initializeAllContainers(JavaModelManager.java:3088)
    org.eclipse.jdt.internal.core.JavaModelManager.getClasspathContainer(JavaModelManager.java:2087)
    org.eclipse.jdt.core.JavaCore.getClasspathContainer(JavaCore.java:3777)
    org.eclipse.jdt.internal.core.JavaProject.resolveClasspath(JavaProject.java:3243)
    org.eclipse.jdt.internal.core.JavaProject.resolveClasspath(JavaProject.java:3407)
    org.eclipse.jdt.internal.core.JavaProject.getResolvedClasspath(JavaProject.java:2491)
    org.eclipse.jdt.internal.core.ExternalFoldersManager.refreshReferences(ExternalFoldersManager.java:494)
    org.eclipse.jdt.internal.core.DeltaProcessor.resourceChanged(DeltaProcessor.java:2091)
    org.eclipse.jdt.internal.core.DeltaProcessingState.resourceChanged(DeltaProcessingState.java:501)
    org.eclipse.core.internal.events.NotificationManager$1.run(NotificationManager.java:305)
    org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
    org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:295)
    org.eclipse.core.internal.events.NotificationManager.handleEvent(NotificationManager.java:277)
    org.eclipse.core.internal.resources.Workspace.broadcastEvent(Workspace.java:376)
    org.eclipse.core.internal.resources.Resource.refreshLocal(Resource.java:1561)
    org.eclipse.core.internal.refresh.RefreshJob.runInWorkspace(RefreshJob.java:166)
    org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:42)
    org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

[donat]

The relevant code that causes the deadlock seems innocent enough: it reads a simple preference file: DefaultModelPersistence.loadModel(). This code is called at multiple places: in the plugin's start() method and in the Gradle classpath container initializer. It can very well be that these are called at the same time, causing the deadlock.

The loadModel() implementation uses a Guava LoadingCache which produces the deadlock.

[vogella]

@donat so you try to fix this deadlock by synchronizing the loadModel method?

[donat]

I'm still investigating, but yes, if I don't find a more elegant solution, I'll force the synchronization.

[donat]

Possible fix on master: https://github.com/eclipse/buildship/commit/72f938ebe03d963355a5ca53db04450de1a54f0b. I'm going to publish a snapshot release soon with the change.

[vogella]

Would be nice if you can also add the snapshot update side to https://github.com/eclipse/buildship/blob/master/README.md similar to the Maven tooling https://github.com/eclipse-m2e/m2e-core/blob/master/README.md#-installation

[vogella]

Possible fix on master: 72f938e. I'm going to publish a snapshot release soon with the change.

Thanks

[donat]

Hi everyone! The changes are available in the latest snapshot release. Please give it a try and let us know if it fixed this issue.

[vogella]

Hi everyone! The changes are available in the latest snapshot release. Please give it a try and let us know if it fixed this issue.

https://download.eclipse.org/buildship/updates/latest/ is also updated?

[donat]

No, the latest update site is updated during a final release.

You can get the latest 3.x snapshot from an update site like this one: https://download.eclipse.org/buildship/updates/e422/snapshots/3.x/

[vogella]

No, the latest update site is updated during a final release.

You can get the latest 3.x snapshot from an update site like this one: https://download.eclipse.org/buildship/updates/e422/snapshots/3.x/

Would it be possible to add a https://download.eclipse.org/buildship/updates/snapshot update site which contains always the latest and greatest snapshot? Make it easier to point to it for testing.

[donat]

@vogella I've created a separate bug report for that: https://github.com/eclipse/buildship/issues/1153

[vogella]

@abstratt or gk10gk or @UdoW could you check if the change fixes the deadlock for you?

[UdoW]

In my installation with Eclipse 2022-03 and a workspace with about 20 projects that use Gradle I cannot reproduce the deadlocks anymore. I will investigate further especially in my own RCP product which uses the Buildship API in my own plug-ins to modify the import of Gradle projects.

The only error from the Buildship plug-in I could see is this font leakage bug which is described here #1109 . I got it after closing the preference page of Gradle.

[donat]

@UdoW I managed the fix the font leak issue. I'm planning a release very soon.

[donat]

Closing as fixed based on https://github.com/eclipse/buildship/issues/1148#issuecomment-1119305635