search

eclipse-archived / ceylon-ide-eclipse

Eclipse Plugin for Ceylon
http://ceylon-lang.org/documentation/ide
Eclipse Public License 1.0
59 stars 28 forks source link

Deadlock in warmup job #400

Closed FroMage closed 12 years ago

FroMage commented 12 years ago

For some reason I've had this deadlock three times since yesterday, for absolutely no obvious reason (every time I noticed it only when it would not let me save the file because it's waiting on completion).

I used jvisualVM to get a thread dump and the culprit seems to be:


"Worker-172" prio=10 tid=0x00007fec74001000 nid=0x7b59 runnable [0x00007fed6a8f6000]
   java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.getEntry(HashMap.java:364)
    at java.util.HashMap.containsKey(HashMap.java:352)
    at com.redhat.ceylon.compiler.loader.AbstractModelLoader.lookupClassMirror(AbstractModelLoader.java:200)
    at com.redhat.ceylon.compiler.loader.AbstractModelLoader.convertToDeclaration(AbstractModelLoader.java:602)
    at com.redhat.ceylon.eclipse.core.model.loader.JDTModelLoader.convertToDeclaration(JDTModelLoader.java:459)
    at com.redhat.ceylon.eclipse.core.model.loader.JDTModelLoader.loadPackage(JDTModelLoader.java:329)
    at com.redhat.ceylon.compiler.loader.model.LazyPackage.getMembers(LazyPackage.java:130)
    at com.redhat.ceylon.eclipse.core.builder.WarmupJob.run(WarmupJob.java:34)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

   Locked ownable synchronizers:
    - None

That's very weird, because the thread is marked as running, doesn't own any lock and containsKey seems to just be hung.

Anyone has any idea how to debug that? I think this requires proper attention by everyone that can help, such as @quintesse and @tombentley as this essentially freezes the IDE.

FroMage commented 12 years ago

Full thread dump:


2012-09-12 12:32:42
Full thread dump OpenJDK 64-Bit Server VM (22.0-b10 mixed mode):

"RMI TCP Connection(idle)" daemon prio=10 tid=0x00007fec84003800 nid=0x7c1a waiting on condition [0x00007fed6bffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007fbc20258> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"JMX server connection timeout 640" daemon prio=10 tid=0x00007fecdc006800 nid=0x7c17 in Object.wait() [0x00007fed6aaf8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007fbc206d8> (a [I)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
    - locked <0x00000007fbc206d8> (a [I)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"RMI Scheduler(0)" daemon prio=10 tid=0x00007fecdc005000 nid=0x7c15 waiting on condition [0x00007fed6abf9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007fbc21aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"RMI TCP Connection(1)-0:0:0:0:0:0:0:1" daemon prio=10 tid=0x00007fec84002800 nid=0x7c14 runnable [0x00007fed820fb000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    - locked <0x00000007fbd1a050> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - <0x00000007fbc20448> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007fec880b8800 nid=0x7c11 runnable [0x00007fed88f95000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
    at java.net.ServerSocket.implAccept(ServerSocket.java:522)
    at java.net.ServerSocket.accept(ServerSocket.java:490)
    at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"Attach Listener" daemon prio=10 tid=0x00007fed48001800 nid=0x7c0f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Worker-174" prio=10 tid=0x00007fed00185000 nid=0x7b68 in Object.wait() [0x00007fed6adfa000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000780237228> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
    - locked <0x0000000780237228> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)

   Locked ownable synchronizers:
    - None

"Worker-173" prio=10 tid=0x00007fec68001000 nid=0x7b67 in Object.wait() [0x00007fed6aefb000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007fa626630> (a java.lang.Object)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.core.internal.jobs.ThreadJob.waitForRun(ThreadJob.java:270)
    - locked <0x00000007fa626630> (a java.lang.Object)
    at org.eclipse.core.internal.jobs.ThreadJob.joinRun(ThreadJob.java:197)
    at org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:92)
    at org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:286)
    at org.eclipse.core.internal.utils.StringPoolJob.run(StringPoolJob.java:101)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

   Locked ownable synchronizers:
    - None

"Worker-172" prio=10 tid=0x00007fec74001000 nid=0x7b59 runnable [0x00007fed6a8f6000]
   java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.getEntry(HashMap.java:364)
    at java.util.HashMap.containsKey(HashMap.java:352)
    at com.redhat.ceylon.compiler.loader.AbstractModelLoader.lookupClassMirror(AbstractModelLoader.java:200)
    at com.redhat.ceylon.compiler.loader.AbstractModelLoader.convertToDeclaration(AbstractModelLoader.java:602)
    at com.redhat.ceylon.eclipse.core.model.loader.JDTModelLoader.convertToDeclaration(JDTModelLoader.java:459)
    at com.redhat.ceylon.eclipse.core.model.loader.JDTModelLoader.loadPackage(JDTModelLoader.java:329)
    at com.redhat.ceylon.compiler.loader.model.LazyPackage.getMembers(LazyPackage.java:130)
    at com.redhat.ceylon.eclipse.core.builder.WarmupJob.run(WarmupJob.java:34)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

   Locked ownable synchronizers:
    - None

"Worker-159" prio=10 tid=0x00007fed0020e000 nid=0x7b4b in Object.wait() [0x00007fed7016a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000780237228> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
    - locked <0x0000000780237228> (a org.eclipse.core.internal.jobs.WorkerPool)
    at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)

   Locked ownable synchronizers:
    - None

"Worker-140" prio=10 tid=0x00007fec8803b800 nid=0x7a8d in Object.wait() [0x00007fed7036c000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007fa626630> (a java.lang.Object)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.core.internal.jobs.ThreadJob.waitForRun(ThreadJob.java:270)
    - locked <0x00000007fa626630> (a java.lang.Object)
    at org.eclipse.core.internal.jobs.ThreadJob.joinRun(ThreadJob.java:197)
    at org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:92)
    at org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:286)
    at org.eclipse.ui.internal.ide.ContentTypeDecorator.decorate(ContentTypeDecorator.java:57)
    at org.eclipse.ui.internal.decorators.LightweightDecoratorDefinition.decorate(LightweightDecoratorDefinition.java:269)
    at org.eclipse.ui.internal.decorators.LightweightDecoratorManager$LightweightRunnable.run(LightweightDecoratorManager.java:81)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.ui.internal.decorators.LightweightDecoratorManager.decorate(LightweightDecoratorManager.java:365)
    at org.eclipse.ui.internal.decorators.LightweightDecoratorManager.getDecorations(LightweightDecoratorManager.java:347)
    at org.eclipse.ui.internal.decorators.DecorationScheduler$1.ensureResultCached(DecorationScheduler.java:371)
    at org.eclipse.ui.internal.decorators.DecorationScheduler$1.run(DecorationScheduler.java:331)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

   Locked ownable synchronizers:
    - None

"[ThreadPool Manager] - Idle Thread" daemon prio=10 tid=0x00007fed18003800 nid=0x6794 in Object.wait() [0x00007fed8160d000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000783e1f740> (a org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run(Executor.java:106)
    - locked <0x0000000783e1f740> (a org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor)

   Locked ownable synchronizers:
    - None

"DLTK indexing" daemon prio=10 tid=0x00007fed08479800 nid=0x678f in Object.wait() [0x00007fed88d1f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x000000078183e0d8> (a org.eclipse.dltk.core.search.indexing.IndexManager)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.dltk.internal.core.search.processing.JobManager.run(JobManager.java:445)
    - locked <0x000000078183e0d8> (a org.eclipse.dltk.core.search.indexing.IndexManager)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"Java indexing" daemon prio=10 tid=0x00007fed080a0800 nid=0x6789 in Object.wait() [0x00007fed8ac9b000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000781146ce0> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:382)
    - locked <0x0000000781146ce0> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"Worker-JM" prio=10 tid=0x00007fed1c17e800 nid=0x6785 in Object.wait() [0x00007fed8a754000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000780260a50> (a java.util.ArrayList)
    at org.eclipse.core.internal.jobs.InternalWorker.run(InternalWorker.java:58)
    - locked <0x0000000780260a50> (a java.util.ArrayList)

   Locked ownable synchronizers:
    - None

"Bundle File Closer" daemon prio=10 tid=0x00007fed1c108000 nid=0x6784 in Object.wait() [0x00007fed8a8fe000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007802373b8> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
    - locked <0x00000007802373b8> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)

   Locked ownable synchronizers:
    - None

"[Timer] - Main Queue Handler" daemon prio=10 tid=0x00007fed14006800 nid=0x6783 in Object.wait() [0x00007fed8ab9a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000780eed658> (a java.lang.Object)
    at org.eclipse.equinox.internal.util.impl.tpt.timer.TimerImpl.run(TimerImpl.java:141)
    - locked <0x0000000780eed658> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"Framework Event Dispatcher" daemon prio=10 tid=0x00007fed1c064000 nid=0x6781 in Object.wait() [0x00007fed8ad9c000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007802375a0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
    - locked <0x00000007802375a0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)

   Locked ownable synchronizers:
    - None

"Start Level Event Dispatcher" daemon prio=10 tid=0x00007feda4448000 nid=0x6780 in Object.wait() [0x00007fed8ae9d000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000780e56a00> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
    - locked <0x0000000780e56a00> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)

   Locked ownable synchronizers:
    - None

"State Data Manager" daemon prio=10 tid=0x00007feda44d6000 nid=0x677f waiting on condition [0x00007fed8af9e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.eclipse.osgi.internal.baseadaptor.StateManager.run(StateManager.java:297)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"Framework Active Thread" prio=10 tid=0x00007feda442b800 nid=0x677e in Object.wait() [0x00007fed90108000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000780b1df60> (a org.eclipse.osgi.framework.internal.core.Framework)
    at org.eclipse.osgi.framework.internal.core.Framework.run(Framework.java:1863)
    - locked <0x0000000780b1df60> (a org.eclipse.osgi.framework.internal.core.Framework)
    at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
    - None

"Service Thread" daemon prio=10 tid=0x00007feda4114000 nid=0x677a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"C2 CompilerThread1" daemon prio=10 tid=0x00007feda4111800 nid=0x6779 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"C2 CompilerThread0" daemon prio=10 tid=0x00007feda410e800 nid=0x6778 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Signal Dispatcher" daemon prio=10 tid=0x00007feda410c800 nid=0x6777 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Finalizer" daemon prio=10 tid=0x00007feda40b5800 nid=0x6776 in Object.wait() [0x00007fed9befd000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x000000078034b838> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    - locked <0x000000078034b838> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

   Locked ownable synchronizers:
    - None

"Reference Handler" daemon prio=10 tid=0x00007feda40b3000 nid=0x6775 in Object.wait() [0x00007fed9bffe000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x000000078034b490> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
    - locked <0x000000078034b490> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
    - None

"main" prio=10 tid=0x00007feda4008000 nid=0x6769 runnable [0x00007fedac518000]
   java.lang.Thread.State: RUNNABLE
    at org.eclipse.swt.internal.gtk.OS._gdk_window_get_origin(Native Method)
    at org.eclipse.swt.internal.gtk.OS.gdk_window_get_origin(OS.java:5387)
    at org.eclipse.swt.widgets.Display.map(Display.java:2820)
    at org.eclipse.swt.widgets.Control.drawBackground(Control.java:120)
    at org.eclipse.swt.widgets.Control.windowProc(Control.java:5091)
    at org.eclipse.swt.widgets.Display.windowProc(Display.java:4369)
    at org.eclipse.swt.internal.gtk.OS._gtk_main_do_event(Native Method)
    at org.eclipse.swt.internal.gtk.OS.gtk_main_do_event(OS.java:8295)
    at org.eclipse.swt.widgets.Display.eventProc(Display.java:1192)
    at org.eclipse.swt.internal.gtk.OS._g_main_context_iteration(Native Method)
    at org.eclipse.swt.internal.gtk.OS.g_main_context_iteration(OS.java:2332)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3177)
    at org.eclipse.ui.internal.dialogs.EventLoopProgressMonitor.runEventLoop(EventLoopProgressMonitor.java:123)
    at org.eclipse.ui.internal.dialogs.EventLoopProgressMonitor.isCanceled(EventLoopProgressMonitor.java:97)
    at org.eclipse.core.runtime.ProgressMonitorWrapper.isCanceled(ProgressMonitorWrapper.java:106)
    at org.eclipse.core.runtime.SubMonitor$RootInfo.isCanceled(SubMonitor.java:259)
    at org.eclipse.core.runtime.SubMonitor.isCanceled(SubMonitor.java:516)
    at org.eclipse.core.internal.jobs.ThreadJob.isCanceled(ThreadJob.java:144)
    at org.eclipse.core.internal.jobs.ThreadJob.waitForRun(ThreadJob.java:233)
    at org.eclipse.core.internal.jobs.ThreadJob.joinRun(ThreadJob.java:197)
    at org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:92)
    at org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:286)
    at org.eclipse.core.internal.resources.WorkManager.checkIn(WorkManager.java:118)
    at org.eclipse.core.internal.resources.Workspace.prepareOperation(Workspace.java:2282)
    at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2339)
    at org.eclipse.ui.actions.WorkspaceModifyOperation.run(WorkspaceModifyOperation.java:118)
    - locked <0x00000007fbd72a10> (a org.eclipse.ui.actions.WorkspaceModifyDelegatingOperation)
    at org.eclipse.ui.internal.editors.text.WorkspaceOperationRunner.run(WorkspaceOperationRunner.java:75)
    at org.eclipse.ui.internal.editors.text.WorkspaceOperationRunner.run(WorkspaceOperationRunner.java:65)
    at org.eclipse.ui.editors.text.TextFileDocumentProvider.executeOperation(TextFileDocumentProvider.java:456)
    at org.eclipse.ui.editors.text.TextFileDocumentProvider.saveDocument(TextFileDocumentProvider.java:772)
    at org.eclipse.ui.texteditor.AbstractTextEditor.performSave(AbstractTextEditor.java:5066)
    at org.eclipse.ui.texteditor.AbstractTextEditor.doSave(AbstractTextEditor.java:4855)
    at org.eclipse.ui.texteditor.AbstractTextEditor$TextEditorSavable.doSave(AbstractTextEditor.java:7198)
    at org.eclipse.ui.Saveable.doSave(Saveable.java:214)
    at org.eclipse.ui.internal.SaveableHelper.doSaveModel(SaveableHelper.java:346)
    at org.eclipse.ui.internal.SaveableHelper$3.run(SaveableHelper.java:193)
    at org.eclipse.ui.internal.SaveableHelper$5.run(SaveableHelper.java:274)
    at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:464)
    at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:372)
    at org.eclipse.ui.internal.WorkbenchWindow$13.run(WorkbenchWindow.java:1666)
    at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
    at org.eclipse.ui.internal.WorkbenchWindow.run(WorkbenchWindow.java:1663)
    at org.eclipse.ui.internal.SaveableHelper.runProgressMonitorOperation(SaveableHelper.java:282)
    at org.eclipse.ui.internal.SaveableHelper.runProgressMonitorOperation(SaveableHelper.java:261)
    at org.eclipse.ui.internal.SaveableHelper.saveModels(SaveableHelper.java:204)
    at org.eclipse.ui.internal.SaveableHelper.savePart(SaveableHelper.java:144)
    at org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.doSave(CompatibilityPart.java:404)
    at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.e4.core.internal.di.MethodRequestor.execute(MethodRequestor.java:56)
    at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:229)
    at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:235)
    at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:199)
    at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:89)
    at org.eclipse.e4.ui.internal.workbench.PartServiceImpl.savePart(PartServiceImpl.java:1154)
    at org.eclipse.ui.internal.WorkbenchPage.saveSaveable(WorkbenchPage.java:3459)
    at org.eclipse.ui.internal.WorkbenchPage.saveEditor(WorkbenchPage.java:3477)
    at org.eclipse.ui.internal.SaveAction.run(SaveAction.java:76)
    at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
    at org.eclipse.jface.commands.ActionHandler.execute(ActionHandler.java:119)
    at org.eclipse.ui.internal.handlers.E4HandlerProxy.execute(E4HandlerProxy.java:76)
    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.e4.core.internal.di.MethodRequestor.execute(MethodRequestor.java:56)
    at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:229)
    at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:210)
    at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:131)
    at org.eclipse.e4.core.commands.internal.HandlerServiceImpl.executeHandler(HandlerServiceImpl.java:171)
    at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.executeCommand(KeyBindingDispatcher.java:276)
    at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.press(KeyBindingDispatcher.java:494)
    at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.processKeyEvent(KeyBindingDispatcher.java:545)
    at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.filterKeySequenceBindings(KeyBindingDispatcher.java:366)
    at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher.access$0(KeyBindingDispatcher.java:313)
    at org.eclipse.e4.ui.bindings.keys.KeyBindingDispatcher$KeyDownFilter.handleEvent(KeyBindingDispatcher.java:82)
    at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
    at org.eclipse.swt.widgets.Display.filterEvent(Display.java:1483)
    at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1275)
    at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1300)
    at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1285)
    at org.eclipse.swt.widgets.Widget.sendKeyEvent(Widget.java:1312)
    at org.eclipse.swt.widgets.Widget.gtk_key_press_event(Widget.java:748)
    at org.eclipse.swt.widgets.Control.gtk_key_press_event(Control.java:3050)
    at org.eclipse.swt.widgets.Composite.gtk_key_press_event(Composite.java:738)
    at org.eclipse.swt.widgets.Widget.windowProc(Widget.java:1758)
    at org.eclipse.swt.widgets.Control.windowProc(Control.java:5116)
    at org.eclipse.swt.widgets.Display.windowProc(Display.java:4369)
    at org.eclipse.swt.internal.gtk.OS._gtk_main_do_event(Native Method)
    at org.eclipse.swt.internal.gtk.OS.gtk_main_do_event(OS.java:8295)
    at org.eclipse.swt.widgets.Display.eventProc(Display.java:1192)
    at org.eclipse.swt.internal.gtk.OS._g_main_context_iteration(Native Method)
    at org.eclipse.swt.internal.gtk.OS.g_main_context_iteration(OS.java:2332)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3177)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1022)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:916)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:86)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:585)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:540)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
    at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1414)

   Locked ownable synchronizers:
    - None

"VM Thread" prio=10 tid=0x00007feda40ab000 nid=0x6774 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007feda4013000 nid=0x676a runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007feda4015000 nid=0x676b runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007feda4016800 nid=0x676c runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007feda4018800 nid=0x676d runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007feda401a800 nid=0x676e runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007feda401c000 nid=0x676f runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007feda401e000 nid=0x6770 runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007feda4020000 nid=0x6771 runnable 

"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007feda4021800 nid=0x6772 runnable 

"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007feda4023800 nid=0x6773 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007feda411e800 nid=0x677b waiting on condition 

JNI global references: 1027
FroMage commented 12 years ago

Relevant code from HashMap:


      355     /**
      356      * Returns the entry associated with the specified key in the
      357      * HashMap.  Returns null if the HashMap contains no mapping
      358      * for the key.
      359      */
      360     final Entry<K,V> getEntry(Object key) {
      361         int hash = (key == null) ? 0 : hash(key.hashCode());
      362         for (Entry<K,V> e = table[indexFor(hash, table.length)];
      363              e != null;
      364              e = e.next) {
      365             Object k;
      366             if (e.hash == hash &&
      367                 ((k = e.key) == key || (key != null && key.equals(k))))
      368                 return e;
      369         }
      370         return null;
      371     }
FroMage commented 12 years ago

The thread in question is running, not waiting and doesn't have any monitor. There is no other code running the model loader at the same time.

So, the most probable explanation is that the thread in question is taking 100% CPU because it's walking a corrupted HashMap which has Entry that point to themselves in a circle.

That can easily happen if two threads try at once to use the model loader.

We can't make the model loader thread-safe easily. Two different instances of the compiler object will be entirely thread-safe from one another because there's zero static data shared, but the single model loader that is used by the IDE is not thread safe. Making it thread-safe would require significant rewrite and work.

The easiest solution is to make sure the IDE will not be using the model loader in a threaded environment, for example by not running the warmupJob while there is typechecking running.

FroMage commented 12 years ago

This is now in the hands of @gavinking

FroMage commented 12 years ago

I managed to block it 20 times in the last half hour, by adding small doc annotations to a lot of files.

gavinking commented 12 years ago

The easiest solution is to make sure the IDE will not be using the model loader in a threaded environment, for example by not running the warmupJob while there is typechecking running.

This "solution" is already implemented. In the latest rev of the IDE, every build and every warmup job obtains a workspace level lock before running. (See CeylonBuilder:588.)

But of course, lots of other things access the model, and due to the lazy loading stuff it's very hard to know what has the potential to cause loading to occur. We can't just go grabbing a workspace-level lock for every single user action.

FroMage commented 12 years ago

What sort of other actions access the model then?

It doesn't have to be per workspace by per project, though it's just as bad.

gavinking commented 12 years ago

What sort of other actions access the model then?

Pfff, CeylonParseController (the editor-level typechecker), every refactoring, every quickfix, the doc hover, the hierarchy popup—even stuff like the outline view and searching touches the model.

gavinking commented 12 years ago

It doesn't have to be per workspace by per project, though it's just as bad.

I think that in future we will need to start quite aggressively sharing models between projects for performance reasons, so I think it will need to be per workspace. Off the top of my head I can't remember why David chose to do a workspace-level lock instead of a project-level lock in the thread management stuff that is there now, but I think there was some reason.

FroMage commented 12 years ago

Well, I can try to make the model loader thread-safe, but it will mean adding locks and synchronisation blocks and shit. It will take me a while of not working on anything else. Did you already make the typechecker thread-safe? I have a hard time believing that you wouldn't have the same issues that the model loader has if you have a thread doing typechecking to update the model and another one that traverses it at the same time.

I mean, our current deadlock is because HashMap is not synchronised and one thread wrote to it while another was reading from it. The same sort of bug can happen in the typechecker.

gavinking commented 12 years ago

Did you already make the typechecker thread-safe? ... I mean, our current deadlock is because HashMap is not synchronised and one thread wrote to it while another was reading from it. The same sort of bug can happen in the typechecker.

WDYM? The "typechecker" itself isn't stateful....

I have a hard time believing that you wouldn't have the same issues that the model loader has if you have a thread doing typechecking to update the model and another one that traverses it at the same time.

I believe that the threading model that @davidfestal put in place was working great up until all this lazy loading shit went in. From that point I no longer understand how it is supposed to work and I certainly stopped being able to reason about it. We also started seeing bugs from that point on. Whether they are little silly bugs or the result of deep conceptual issues with the whole approach I wouldn't be able to say.

quintesse commented 12 years ago

Does the type checker need to be thread-safe though? It seems that it is particular to a certain compile job, while at the same time being able to use a shared model loader because everything it loads can be shared with other type checkers.

FroMage commented 12 years ago

The "typechecker" itself isn't stateful....

Does the type checker need to be thread-safe though?

Yes to both: it builds model classes, such as Package and Module, or updates them in the case of incremental compilation. Those operations modify the model (not the lazy model, but the from-source model), so if another thread tries to traverse (read) it while it's being updated, then the same sort of bug can happen.

gavinking commented 12 years ago

@FroMage What I've been trying to say is that the threading model ensures, or at least should ensure, that two typecheckers are never modifying the same model at the same time. The IDE was designed around that notion. (Of course, it might have bugs, in which case we need to find them.) But that in and of itself is not sufficient to eliminate any threading bugs, because of lazy loading.

FroMage commented 12 years ago

And what's I'm saying is that even if there is a single typechecker running at once, having any other thread read model that is being updated by that running typechecker can lead to the same sort of bug we ran into. It's a case of single-writer/reader in both cases.

FroMage commented 12 years ago

I mean, we can ignore the issue and only fix the lazy model, but I'm convinced the same bug is waiting to happen for model loaded from source (so not lazy) unless we also make the typechecker thread-safe.

gavinking commented 12 years ago

Perhaps. Again I don't recall off the top of my head, but my guess is that the model isn't made available to other threads until the typechecker finishes with it.

davidfestal commented 12 years ago

Afair, as soon as everything implying the typechecker model (classpath container initialization, build / typechecking, ...) was done under the root scheduling rule in non-UI threads, it seems we could ensure doing model changes sequentially and without deadlocks. Of course it was slower... But I'm not aware enough about recent changes to say more...

quintesse commented 12 years ago

Or with a swapping mechanism, where you have 2 models, one read-only used by the GUI elements and one being updated by the type checker. The moment everything is finished they get swapped inside a synchronized block.

gavinking commented 12 years ago

my guess is that the model isn't made available to other threads until the typechecker finishes with it.

Rm, I'm not certain that this is the case for an incremental build. Ask @davidfestal.

FroMage commented 12 years ago

but my guess is that the model isn't made available to other threads until the typechecker finishes with it

Just the act of updating a Package during incremental typechecking of a Class, or replace that Class is enough to derail any thread that reads that Package at the same time.

gavinking commented 12 years ago

@FroMage you and @davidfestal worked on this stuff. I don't have a good enough understanding of it.

FroMage commented 12 years ago

Right, for the typechecker, making it thread-safe is easier, since I assume it will only replace Units within Packages, and not update Declarations, but at the level of adding/removing Unit, Package and Module there needs to be explicit thread-safety.

For the model loader it's harder since just reading it causes shit to happen. But it's not like we have a choice, it has to be lazy. It's just that it was never designed for thread-safety. So it's not crap as I've heard ;) It's just misused at the moment. And we need to redesign it.

Having a crashing IDE is just not acceptable, so if we can't read/write to the model from a single thread in the IDE then I have to make the model loader thread-safe. What I hate is that testing this sort of thing is just next to impossible.

davidfestal commented 12 years ago

Le 12 sept. 2012 18:30, "Stéphane Épardaud" notifications@github.com a écrit :

Right, for the typechecker, making it thread-safe is easier, since I assume it will only replace Units within Packages, and not update Declarations, but at the level of adding/removing Unit, Package and Module there needs to be explicit thread-safety.

For the model loader it's harder since just reading it causes shit to happen. But it's not like we have a choice, it has to be lazy. It's just that it was never designed for thread-safety. So it's not crap as I've heard ;) It's just misused at the moment. And we need to redesign it.

Having a crashing IDE is just not acceptable, so if we can't read/write to the model from a single thread in the IDE

We should be able to ensure this in the IDE, using scheduling rules. I should review this as sooner as I come back to work !

then I have to make the model loader thread-safe. What I hate is that testing this sort of thing is just next to impossible.

— Reply to this email directly or view it on GitHub.

gavinking commented 12 years ago

We should be able to ensure this in the IDE, using scheduling rules.

I'm doubtful that this is really the right solution. A lot of access to the model necessarily occurs from the UI thread (for example, the doc hover, quick fixes/assists, and autocompletion. I don't see how we could possibly use scheduling rules to control that stuff.

gavinking commented 12 years ago

Right, for the typechecker, making it thread-safe is easier, since I assume it will only replace Units within Packages, and not update Declarations, but at the level of adding/removing Unit, Package and Module there needs to be explicit thread-safety.

I thought there already was. Is that not the case, @davidfestal?

gavinking commented 12 years ago

For the model loader it's harder since just reading it causes shit to happen. But it's not like we have a choice, it has to be lazy.

I reluctantly agree that this is probably true, but only because loading the entire model for the Java SDK is so expensive. For other modules, I'm pretty convinced that performance, stability, and overall user experience was better before we added the lazy loading stuff. The truth is that you wind up having to load so much of the model anyway, that you may as well load the whole thing—except in the special case of the bloated java module.

A different solution would be:

  1. "modularize" the Java SDK into smaller chunks java.core, java.swing, etc, and
  2. do model loading eagerly.

Unfortunately, I bet that 1 would be just really hard to do.

davidfestal commented 12 years ago

Yes, we had already worked to avoid concurrent mofification exceptions here Le 12 sept. 2012 20:08, "Gavin King" notifications@github.com a écrit :

Right, for the typechecker, making it thread-safe is easier, since I assume it will only replace Units within Packages, and not update Declarations, but at the level of adding/removing Unit, Package and Module there needs to be explicit thread-safety.

I thought there already was. Is that not the case, @davidfestalhttps://github.com/davidfestal ?

— Reply to this email directly or view it on GitHubhttps://github.com/ceylon/ceylon-ide-eclipse/issues/400#issuecomment-8504145.

davidfestal commented 12 years ago

I was speaking of write access actions Le 12 sept. 2012 20:07, "Gavin King" notifications@github.com a écrit :

We should be able to ensure this in the IDE, using scheduling rules.

I'm doubtful that this is really the right solution. A lot of access to the model necessarily occurs from the UI thread (for example, the doc hover, quick fixes/assists, and autocompletion. I don't see how we could possibly use scheduling rules to control that stuff.

— Reply to this email directly or view it on GitHubhttps://github.com/ceylon/ceylon-ide-eclipse/issues/400#issuecomment-8504121.

gavinking commented 12 years ago

Well OK but that's not the problem here (at least I don't think so). The problem is that "read" operations (which might actually do writes, but even if they didn't could still be a problem) are completely unsynchronized.

Sent from my iPhone

On Sep 12, 2012, at 3:01 PM, David Festal notifications@github.com wrote:

I was speaking of write access actions Le 12 sept. 2012 20:07, "Gavin King" notifications@github.com a écrit :

FroMage commented 12 years ago

So I committed a lot of code dealing with synchronisation, hopefully fixing the issue. If you still see a deadlock please report it here with a stack trace (use jvisualvm to obtain it).

FroMage commented 12 years ago

Never saw it again, so closing now.

gavinking commented 12 years ago

FTR, I have not observed it recently either.

davidfestal commented 12 years ago

Neither did I