deadlock during startup

[martinlippert]

A colleague reported a problem during startup, the thread dump reveals a deadlock with Buildship involved:

Found one Java-level deadlock:
=============================
"Worker-3: Load persistent model for all projects":
  waiting for ownable synchronizer 0x000000070072dac8, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "Worker-4: Refreshing workspace"

"Worker-4: Refreshing workspace":
  waiting to lock monitor 0x00006000000568a0 (object 0x000000073ba67d30, a java.lang.Object),
  which is held by "Worker-3: Load persistent model for all projects"

Java stack information for the threads listed above:
===================================================
"Worker-3: Load persistent model for all projects":
    at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
    - parking to wait for  <0x000000070072dac8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@21/LockSupport.java:269)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21/AbstractQueuedSynchronizer.java:756)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(java.base@21/AbstractQueuedSynchronizer.java:1038)
    at java.util.concurrent.locks.ReentrantLock$Sync.tryLockNanos(java.base@21/ReentrantLock.java:168)
    at java.util.concurrent.locks.ReentrantLock.tryLock(java.base@21/ReentrantLock.java:479)
    at org.eclipse.osgi.container.Module.lockStateChange(Module.java:309)
    at org.eclipse.osgi.container.Module.start(Module.java:419)
    at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:513)
    at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
    at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:570)
    at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:335)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:397)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:500)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
    at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
    at java.lang.ClassLoader.loadClass(java.base@21/ClassLoader.java:526)
    at org.eclipse.buildship.core.internal.preferences.PersistentModelConverter.toModel(PersistentModelConverter.java:142)
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence.doLoadModel(DefaultModelPersistence.java:130)
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence$1.load(DefaultModelPersistence.java:62)
    - locked <0x000000073ba67d30> (a java.lang.Object)
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence$1.load(DefaultModelPersistence.java:57)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
    - locked <0x000000073bb9f570> (a com.google.common.cache.LocalCache$StrongEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence$2.run(DefaultModelPersistence.java:193)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
"Worker-4: Refreshing workspace":
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence$1.load(DefaultModelPersistence.java:62)
    - waiting to lock <0x000000073ba67d30> (a java.lang.Object)
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence$1.load(DefaultModelPersistence.java:57)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
    - locked <0x000000073ba67f08> (a com.google.common.cache.LocalCache$StrongEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4941)
    at org.eclipse.buildship.core.internal.preferences.DefaultModelPersistence.loadModel(DefaultModelPersistence.java:70)
    at org.eclipse.buildship.core.internal.CorePlugin.scheduleSynchronizationForAbsentModels(CorePlugin.java:223)
    at org.eclipse.buildship.core.internal.CorePlugin.start(CorePlugin.java:120)
    at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:818)
    at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)
    at java.security.AccessController.executePrivileged(java.base@21/AccessController.java:809)
    at java.security.AccessController.doPrivileged(java.base@21/AccessController.java:571)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:810)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:767)
    at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1032)
    at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:371)
    at org.eclipse.osgi.container.Module.doStart(Module.java:605)
    at org.eclipse.osgi.container.Module.start(Module.java:468)
    at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:513)
    at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
    at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:570)
    at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:335)
    at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:397)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:500)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
    at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
    at java.lang.ClassLoader.loadClass(java.base@21/ClassLoader.java:526)
    at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:622)
    at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:196)
    at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:920)
    at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:246)
    at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:63)
    at org.eclipse.jdt.core.JavaCore.computeClasspathContainerInitializer(JavaCore.java:3874)
    at org.eclipse.jdt.core.JavaCore.getClasspathContainerInitializer(JavaCore.java:3850)
    at org.eclipse.jdt.internal.core.JavaModelManager.initializeContainer(JavaModelManager.java:3190)
    at org.eclipse.jdt.internal.core.JavaModelManager$10.run(JavaModelManager.java:3093)
    at org.eclipse.jdt.internal.core.JavaModelManager.initializeAllContainers(JavaModelManager.java:3153)
    at org.eclipse.jdt.internal.core.JavaModelManager.getClasspathContainer(JavaModelManager.java:2152)
    at org.eclipse.jdt.core.JavaCore.getClasspathContainer(JavaCore.java:3827)
    at org.eclipse.jdt.internal.core.JavaProject.resolveClasspath(JavaProject.java:3272)
    at org.eclipse.jdt.internal.core.JavaProject.resolveClasspath(JavaProject.java:3436)
    at org.eclipse.jdt.internal.core.JavaProject.getResolvedClasspath(JavaProject.java:2521)
    at org.eclipse.jdt.internal.core.ExternalFoldersManager.refreshReferences(ExternalFoldersManager.java:494)
    at org.eclipse.jdt.internal.core.DeltaProcessor.resourceChanged(DeltaProcessor.java:2093)
    at org.eclipse.jdt.internal.core.DeltaProcessingState.resourceChanged(DeltaProcessingState.java:501)
    at org.eclipse.core.internal.events.NotificationManager$1.run(NotificationManager.java:321)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
    at org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:311)
    at org.eclipse.core.internal.events.NotificationManager.handleEvent(NotificationManager.java:295)
    at org.eclipse.core.internal.resources.Workspace.broadcastEvent(Workspace.java:453)
    at org.eclipse.core.internal.resources.Resource.refreshLocal(Resource.java:1609)
    at org.eclipse.core.internal.refresh.RefreshJob.runInWorkspace(RefreshJob.java:228)
    at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:43)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

Found 1 deadlock.

[donat]

Do you have a reproducer, maybe?

[martinlippert]

Do you have a reproducer, maybe?

Unfortunately not... 🤷🏼‍♂️

[donat]

Got it, no problem. We'll take a look anyway ;)

[martinlippert]

Awesome, many thanks for looking into this!!!

[donat]

No problem. I'll kick off a snapshot release with the changes. Could you or your colleague test it?

[martinlippert]

Sure, I will ask him. Let me know once the snapshot build is around and where to find it

[donat]

You can install the latest snapshot from this update site: https://download.eclipse.org/buildship/updates/latest-snapshot/

[martinlippert]

Awesome, I integrated this into our CI builds, so that folks can test this right out-of-the-box. Since you never really know when a deadlock will happen, not sure about how long to keep this open though... ;-)