If mypy fails and only prints to stdout, not stderr, this leads to a cryptic event log message without details or stacktrace

christian-steinmeyer commented 2 years ago

Step 1: Are you in the right place?

[x] I have verified there are no duplicate active or recent bugs, questions, or requests
[x] I have verified that I am using the latest version of the plugin.

Step 2: Describe your environment

Plugin version: 0.14.0
PyCharm/IDEA version: 2021.3.2 (Build 213.6777.50)
Mypy version: 0.931
Python version: 3.10.2 <-- note python 3.10

Step 3: Describe the problem:

Steps to reproduce:

use a match statement
use the plugin to run mypy on that file

Observed Results:

Error in event log (see below)

Expected Results:

plain error report of mypy result as with other mypy violations, instead of plugin crash
detailed information about what caused the error

Relevant Code:

MWE:

def print_hi(name):
    match name:
        case "Mypy":
            print("Doesn't work")
        case _:
            print(f'Hi, {name}')

if __name__ == '__main__':
    print_hi('PyCharm')

Mypy Plugin (Mypy exited abnormally)

The scan failed due to an exception: Mypy failed with code 2 
com.leinardi.pycharm.mypy.exception.MypyToolException: Mypy failed with code 2
   at com.leinardi.pycharm.mypy.mpapi.MypyRunner.runMypy(MypyRunner.java:324)
   at com.leinardi.pycharm.mypy.mpapi.MypyRunner.scan(MypyRunner.java:266)
   at com.leinardi.pycharm.mypy.checker.ScanFiles.scan(ScanFiles.java:109)
   at com.leinardi.pycharm.mypy.checker.ScanFiles.checkFiles(ScanFiles.java:100)
   at com.leinardi.pycharm.mypy.checker.ScanFiles.call(ScanFiles.java:74)
   at com.leinardi.pycharm.mypy.checker.ScanFiles.call(ScanFiles.java:46)
   at com.intellij.openapi.application.impl.ApplicationImpl$2.call(ApplicationImpl.java:294)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
   at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
   at java.base/java.security.AccessController.doPrivileged(Native Method)
   at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
   at java.base/java.lang.Thread.run(Thread.java:829)

intgr commented 2 years ago

There should be another message "Mypy failed abnormally" in your IDE that contains the error from mypy's stderr. Please post that as well.

Note that prior to Mypy plugin version 0.14, errors like this were just silently ignored (#92). So this is likely not a regression, but now the user is being informed in situations where mypy had been failing all along.

intgr commented 2 years ago

As for the version combination, I have also been using Python 3.10 and mypy 0.931 and newest IntelliJ for a while without issues.

christian-steinmeyer commented 2 years ago

Thanks for the quick response! I understand, that some messages were silent before but are no longer, but in my case, the mypy plugin doesn't seem to work anymore at all (at least for one project). When I use the mypy window and run a scan of a file, I get the following feedback:

The scan failed due to an error - please see the event log for more information

as well as these two messages in the event log:

13:41   Mypy Plugin (Mypy exited abnormally)

13:41   Unexpected Exception Caught
            The scan failed due to an exception: Mypy failed with code 2
            ... (show balloon)

The second of which I already posted in full above. As far as I can tell, that's all the info I have. Further, I get this entry in the idea.log:

2022-03-10 13:41:46,319 [6731692]   INFO - .pycharm.mypy.mpapi.MypyRunner - Running command: /Users/<VIRTUAL_ENV>/bin/mypy --show-column-numbers --follow-imports silent /Users/<PROJECT><SOME_FILE>.py

I deleted the mypy cache of that project and also verified that the plugin can find the mypy executable. Will now create a second project in the same environment and see if reproducible there or the project somehow got corrupted.

intgr commented 2 years ago

OK thanks. It's weird that the "Mypy failed abnormally" log entry does not contain extra details -- did mypy not print anything to its stderr?

Does mypy work from the command line if you run it in the same virtualenv, with the same arguments, same working directory?

christian-steinmeyer commented 2 years ago

If I run mypy --show-column-numbers --follow-imports silent <FILE> from the same virtual env and directory, mypy runs, and doesn't crash, but finds an error (Match statement not supported). Aaaaand there we go, that seems to be the issue.

In a MWE in the same environment, this yields the same error:

def print_hi(name):
    match name:
        case "Mypy":
            print("Doesn't work")
        case _:
            print(f'Hi, {name}')

if __name__ == '__main__':
    print_hi('PyCharm')

christian-steinmeyer commented 2 years ago

FYI: Looks like mypy's support for match statements might be released on Friday.

intgr commented 2 years ago

Oh I see, this particular error message is printed into stdout, not stderr.

match_stmt.py:2: error: Match statement is not supported
Found 1 error in 1 file (errors prevented further checking)

This is definitely something that can be improved in the plugin, to collect error messages from stderr and stdout.

christian-steinmeyer commented 2 years ago

Updated the issue to reflect our findings. I think there are two issues, actually:

What you mentioned already: missing information about the error cause
Even if this was present, the plugin should not have crashed, but rather report the mypy violation as any other, right?

intgr commented 2 years ago

Even if this was present, the plugin should not have crashed, but rather report the mypy violation as any other, right?

I think there is some room to improve the user experience (PRs welcome), but fundamentally it's not that the "plugin crashed", it's that mypy itself reports this situation as equivalent to a crash.

According to https://github.com/python/mypy/issues/6003 regular mypy inspections would be reported with exit code 1. But in this case mypy returns exit code 2, documented as "in case of a crash, bad arguments, and other non-standard conditions".

I assume mypy developers took this approach because the parser is incapable of handling match syntax. Without a valid parse tree of a file, it refuses to type check the file, nor any modules that import this file, etc.

So I believe the correct place to address (2) would be on the mypy side. If mypy were to implement a more graceful parser and handle this as a regular inspection, the plugin would handle it just fine.

christian-steinmeyer commented 2 years ago

That does make sense, thanks! If you point me to the correct file(s), I'd be happy to prepare a PR. Given, mypy's fix is just around the corner, this should be a non-issue soon then.

intgr commented 2 years ago

If you point me to the correct file(s), I'd be happy to prepare a PR.

Thanks, any help would be welcome. The error handling logic lives in the MypyRunner.runMypy() method -- see the diffs in https://github.com/leinardi/mypy-pycharm/pull/92/files

christian-steinmeyer commented 2 years ago

As I see it, the output of the process is already consumed and stored in issues at that point. So we could either handle them at that point as if they were normally found (e.g. through something like the ProcessResultsThread) or simply add some output to the error message based on the issues found (not sure about internationalization of the project, so not sure, how you'd do that in that case) - any preference?

pwmarcz commented 2 years ago

This issue makes Pycharm with Mypy pretty annoying to use - any syntax error makes Pycharm report that Mypy crashed (instead of just reporting the syntax error, which Mypy itself correctly points out).

According to https://github.com/python/mypy/issues/6003 regular mypy inspections would be reported with exit code 1. But in this case mypy returns exit code 2, documented as "in case of a crash, bad arguments, and other non-standard conditions".

This is a Github issue, not Mypy's documentation - the issue is not closed yet, and the comments point out that there are still some issues before the exit code is reliable. I don't think the current documentation says anything except implying that a non-zero exit code means failure. So, I would take that as Mypy actually not making any promises about 1 vs. 2 exit code just yet.

I agree that Mypy should do better here, but I think fixing this issue in the Pycharm plugin right now would be a big usability win for the users; I for one would certainly appreciate it.

EDIT: I see that a fix was discussed already (but no PR yet), I'll try to implement it later.

leinardi / mypy-pycharm