leinardi / mypy-pycharm

A plugin providing both real-time and on-demand scanning of Python files with Mypy from within PyCharm/IDEA.
Apache License 2.0
188 stars 30 forks source link

Make plugin hot-reloadable #90

Closed intgr closed 2 years ago

intgr commented 2 years ago

Contributor checklist


Description

This allows the plugin to be (re)loaded without restarting the IDE, which is useful when developing the plugin and more convenient when upgrading.

There appear to be some issues in IntelliJ IDEA with dynamic unloading (IDEA-289241, IDEA-289243), but in the worst case, the user receives a message that they have to restart IDE to reload. Which was the norm previously.

Docs: https://plugins.jetbrains.com/docs/intellij/dynamic-plugins.html

Type of Changes

Type
:sparkles: New feature
intgr commented 2 years ago

Hot-reloading sometimes works, but other times causes errors. Haven't figured out what is causing this.

Throwable: Assertion failed ``` java.lang.Throwable: Assertion failed at com.intellij.openapi.diagnostic.Logger.assertTrue(Logger.java:218) at com.intellij.openapi.diagnostic.Logger.assertTrue(Logger.java:227) at com.intellij.ide.plugins.PluginSet.enablePlugin(PluginSet.kt:48) at com.intellij.ide.plugins.DynamicPlugins.loadPlugin(DynamicPlugins.kt:820) at com.intellij.ide.plugins.DynamicPlugins.loadPlugin(DynamicPlugins.kt:808) at com.intellij.ide.plugins.DynamicPluginVfsListener$prepareChange$2$afterVfsChange$1.run(DynamicPluginVfsListener.kt:77) at com.intellij.openapi.application.TransactionGuardImpl.runWithWritingAllowed(TransactionGuardImpl.java:214) at com.intellij.openapi.application.TransactionGuardImpl.access$200(TransactionGuardImpl.java:21) at com.intellij.openapi.application.TransactionGuardImpl$2.run(TransactionGuardImpl.java:196) at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:805) at com.intellij.openapi.application.impl.ApplicationImpl.lambda$invokeLater$4(ApplicationImpl.java:348) at com.intellij.openapi.application.impl.FlushQueue.doRun(FlushQueue.java:82) at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:131) at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:47) at com.intellij.openapi.application.impl.FlushQueue$FlushNow.run(FlushQueue.java:187) at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313) at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:776) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:746) at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:891) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:760) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$6(IdeEventQueue.java:447) at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:818) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$7(IdeEventQueue.java:446) at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:805) at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:492) at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90) ```
IllegalStateException: Couldn't get mypy plugin ``` 2022-02-19 15:09:30,807 [ 985772] ERROR - t.daemon.impl.ExternalToolPass - IdeaLoggingEvent[message=ExternalToolPass: , throwable=com.intellij.diagnostic.PluginException: annotator: com.leinardi.pycharm.mypy.MypyAnnotator@42419987 (class com.leinardi.pycharm.mypy.MypyAnnotator) [Plugin: com.leinardi.pycharm.mypy] at com.intellij.ide.plugins.PluginManagerCore.createPluginException(PluginManagerCore.java:270) at com.intellij.diagnostic.PluginProblemReporterImpl.createPluginExceptionByClass(PluginProblemReporterImpl.java:12) at com.intellij.diagnostic.PluginException.createByClass(PluginException.java:83) at com.intellij.codeInsight.daemon.impl.ExternalToolPass.processError(ExternalToolPass.java:272) at com.intellij.codeInsight.daemon.impl.ExternalToolPass.doAnnotate(ExternalToolPass.java:221) at com.intellij.codeInsight.daemon.impl.ExternalToolPass.doAnnotate(ExternalToolPass.java:212) at com.intellij.codeInsight.daemon.impl.ExternalToolPass$1.lambda$run$0(ExternalToolPass.java:190) at com.intellij.codeInsight.daemon.impl.ExternalToolPass.runChangeAware(ExternalToolPass.java:287) at com.intellij.codeInsight.daemon.impl.ExternalToolPass$1.lambda$run$2(ExternalToolPass.java:190) at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:188) at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:624) at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:698) at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:646) at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:623) at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:66) at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:175) at com.intellij.openapi.progress.util.BackgroundTaskUtil.runUnderDisposeAwareIndicator(BackgroundTaskUtil.java:277) at com.intellij.openapi.progress.util.BackgroundTaskUtil.runUnderDisposeAwareIndicator(BackgroundTaskUtil.java:255) at com.intellij.codeInsight.daemon.impl.ExternalToolPass$1.run(ExternalToolPass.java:189) at com.intellij.util.ui.update.MergingUpdateQueue.execute(MergingUpdateQueue.java:333) at com.intellij.util.ui.update.MergingUpdateQueue.execute(MergingUpdateQueue.java:323) at com.intellij.util.ui.update.MergingUpdateQueue.lambda$flush$1(MergingUpdateQueue.java:273) at com.intellij.util.ui.update.MergingUpdateQueue.flush(MergingUpdateQueue.java:287) at com.intellij.util.ui.update.MergingUpdateQueue.run(MergingUpdateQueue.java:241) at com.intellij.util.concurrency.QueueProcessor.runSafely(QueueProcessor.java:240) at com.intellij.util.Alarm$Request.runSafely(Alarm.java:385) at com.intellij.util.Alarm$Request.run(Alarm.java:374) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at com.intellij.util.concurrency.SchedulingWrapper$MyScheduledFutureTask.run(SchedulingWrapper.java:220) at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:246) at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:32) at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:225) at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:213) at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:214) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalStateException: Couldn't get mypy plugin at com.leinardi.pycharm.mypy.MypyAnnotator.plugin(MypyAnnotator.java:87) at com.leinardi.pycharm.mypy.MypyAnnotator.doAnnotate(MypyAnnotator.java:116) at com.leinardi.pycharm.mypy.MypyAnnotator.doAnnotate(MypyAnnotator.java:62) at com.intellij.codeInsight.daemon.impl.ExternalToolPass.doAnnotate(ExternalToolPass.java:218) ... 37 more ] ```
intgr commented 2 years ago

Sigh. After spending hours on this (even before creating this PR), trying out lots of things, many dead ends, I'm finally closer to understanding this. Just writing down notes for myself for future reference.

intgr commented 2 years ago

Looks like hot-reload works as long as:

:facepalm: I'm glad this is cleared up now. Seems like something IntelliJ should fix upstream.

Snapshot analysis outputs for the two cases:

com.intellij.analysis.problemsView.toolWindow.HighlightingWatcher.problems ``` 2022-02-19 17:26:49,160 [ 113120] INFO - lij.ide.plugins.DynamicPlugins - Snapshot analysis result: Root 1: ROOT: Global JNI com.intellij.ui.jcef.JBCefSourceSchemeHandlerFactory. com.intellij.ui.jcef.JBCefSourceSchemeHandlerFactory. com.intellij.util.lang.PathClassLoader.classes java.util.Vector.elementData java.lang.Object[] com.intellij.openapi.util.Disposer.ourTree com.intellij.openapi.util.ObjectTree.myObject2NodeMap it.unimi.dsi.fastutil.objects.Reference2ObjectOpenHashMap.key java.lang.Object[] com.intellij.analysis.problemsView.toolWindow.HighlightingWatcher.problems java.util.LinkedHashMap.tail java.util.LinkedHashMap$Entry.key com.intellij.openapi.editor.impl.RangeHighlighterImpl.myErrorStripeTooltip com.intellij.codeInsight.daemon.impl.HighlightInfo.quickFixActionMarkers com.intellij.util.containers.LockFreeCopyOnWriteArrayList.value java.lang.Object[] com.intellij.openapi.util.Pair.first com.intellij.codeInsight.daemon.impl.HighlightInfo$IntentionActionDescriptor.myAction com.leinardi.pycharm.mypy.intentions.TypeIgnoreIntention. com.leinardi.pycharm.mypy.intentions.TypeIgnoreIntention. * com.intellij.ide.plugins.cl.PluginClassLoader 2022-02-19 17:26:49,179 [ 113139] INFO - lij.ide.plugins.DynamicPlugins - Plugin com.leinardi.pycharm.mypy is not unload-safe because class loader cannot be unloaded. Memory snapshot created at /home/marti/unload-com.leinardi.pycharm.mypy-19.02.2022_17.26.43.hprof ```
Sticky class java.awt.KeyboardFocusManager.permanentFocusOwner ``` 2022-02-19 17:58:47,612 [ 423693] INFO - lij.ide.plugins.DynamicPlugins - Snapshot analysis result: Root 1: ROOT: Sticky class java.awt.KeyboardFocusManager.permanentFocusOwner com.intellij.openapi.editor.impl.EditorComponentImpl.myEditor com.intellij.openapi.editor.impl.EditorImpl.myDocumentMarkupModel com.intellij.openapi.editor.impl.EditorFilteringMarkupModelEx.myDelegate com.intellij.openapi.editor.impl.MarkupModelImpl.myListeners com.intellij.util.containers.LockFreeCopyOnWriteArrayList.value java.lang.Object[] com.intellij.analysis.problemsView.toolWindow.HighlightingWatcher.problems java.util.LinkedHashMap.tail java.util.LinkedHashMap$Entry.before java.util.LinkedHashMap$Entry.key com.intellij.openapi.editor.impl.RangeHighlighterImpl.myErrorStripeTooltip com.intellij.codeInsight.daemon.impl.HighlightInfo.quickFixActionMarkers com.intellij.util.containers.LockFreeCopyOnWriteArrayList.value java.lang.Object[] com.intellij.openapi.util.Pair.first com.intellij.codeInsight.daemon.impl.HighlightInfo$IntentionActionDescriptor.myAction com.leinardi.pycharm.mypy.intentions.TypeIgnoreIntention. com.leinardi.pycharm.mypy.intentions.TypeIgnoreIntention. * com.intellij.ide.plugins.cl.PluginClassLoader 2022-02-19 17:58:47,627 [ 423708] INFO - lij.ide.plugins.DynamicPlugins - Plugin com.leinardi.pycharm.mypy is not unload-safe because class loader cannot be unloaded. Memory snapshot created at /home/marti/unload-com.leinardi.pycharm.mypy-19.02.2022_17.58.41.hprof ```
intgr commented 2 years ago

Reported bugs to IntelliJ: https://youtrack.jetbrains.com/issue/IDEA-289241 https://youtrack.jetbrains.com/issue/IDEA-289243

leinardi commented 2 years ago

Thanks a lot for the investigation and for opening the bug reports upstream! I voted and started watching both of them.

BTW, let me know when you think there are enough changes to publish a new release :+1:

intgr commented 2 years ago

I've tested this thoroughly enough that I think this can be merged.

The dynamic unloading issues in IntelliJ aren't blockers for this: in the worst case, the user receives a message that they have to restart the IDE to reload the plugin. Which was the norm previously.

When it does work, it saves a significant amount of time during development and updating.

leinardi commented 2 years ago

@intgr I tested locally your changes and everything seems to work fine for me! I just submitted a new release with all your changes: https://github.com/leinardi/mypy-pycharm/releases

Thanks a lot for all the help and effort you are putting in!

BTW, not sure if you noticed or if I already told you but this plugin code is heavily inspired by Checkstyle-IDEA: I basically forked that project and adapted it to run Mypy. If you are looking for inspiration or if you want to check if there are some interesting changes that we could backport I suggest to take a look at their commit log: https://github.com/jshiell/checkstyle-idea/commits/main