New erratic linting behavior

Behavior

New erratic behavior for On Type, On Save, and Terminal linting.

First, thank you. I'm new to this and you folks made the best linting app with incredible documentation that's really helped me figure a lot of things out!

It's been rock solid until 4 or 5 days ago. Since I'm new it seems highly likely I screwed up something. If so, I'm stumped.

Below are screenshots of error producing code that sum up the behavior pretty well. Basically, errors disappear when Pylance finally catches an error. However, in some circumstances both programs will share the same error (and hover). Which is expected and has been the norm until my recent issues. Whitespace is the only error from Ruff that persists when the other lint appears. A ruff check in the terminal produces the same diagnostics. If the Pylance lint is present, only the whitespace error is documented.

Code

ruff troubleshooting continued normal 24 03 23 15 240326-4

ruff troubleshooting continued below 24 03 23 15 240326-5

ruff troubleshooting continued above 24 03 23 15 240326-3

Settings

Settings via Output as well as VSCode User Settings Behavior happens with or without my pyproject.toml

24-03-25 23:36:30.502 [info] Running Ruff with: c:\Users\LL\macromodule\.venv\Scripts\ruff.exe ['check', '--force-exclude', '--no-cache', '--no-fix', '--quiet', '--output-format', 'json', '-', '--config=pyproject.toml', '--stdin-filename', 'c:\\Users\\LL\\macromodule\\desk_macropad\\simplemacropad.py']

// user settings
"ruff.path": [],
"ruff.interpreter": [],
"ruff.fixAll": true,
"ruff.organizeImports": true,
"ruff.ignoreStandardLibrary": false,
"ruff.importStrategy": "fromEnvironment",
"ruff.showNotifications": "onError",
"ruff.enable": true,
"ruff.lint.enable": true,
"ruff.lint.run": "onType",
"ruff.lint.args": [
  "--config=pyproject.toml"
],
"ruff.format.args": [
  "--config=pyproject.toml",
],
"ruff.codeAction.disableRuleComment": {
  "enable": true 
},
"ruff.codeAction.fixViolation": {
"enable": true,
},
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
}

ruff versions

It occurs with the vscode bundled ruff exe, a local venv ruff.exe, and an outside installation of ruff using Python 3.11.5.

Ruff versions:

0.3.0
0.3.1 (charliermarsh.ruff-2024.16.0-win32-x64\bundled)
0.3.4 (current workspace)
Environment

2024.16.0 ruff extension for Windows 64 VSCode on Windows 11

VSCode installation specs:

Version: 1.87.2 (user setup)
Commit: 863d2581ecda6849923a2118d93a088b0745d9d6
Date: 2024-03-08T15:20:17.278Z
Electron: 27.3.2
ElectronBuildId: 26836302
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: 11.8.172.18-electron
OS: Windows_NT x64 10.0.22631

Python

Python 3.11.8 Windows AMD64
Python 3.11.5 Foundation
Additional Notes

It also happens with Jedi (and Pylance turned off - as well as I could manage).

Also:

I've disabled extensions.
Rolled back versions for extensions (mspython, pylance, c/c++)
Reinstalled Ruff (via terminal and as an extension).
Adjusted encoding and looked for hidden chars
I hope this is just the right amount of information. :|
Let me know if you you'd like anything else.

First, thank you. I'm new to this and you folks made the best linting app with incredible documentation that's really helped me figure a lot of things out!

Thank you. We're glad that you found the documentation helpful.

Below are screenshots of error producing code that sum up the behavior pretty well.

Sorry, I looked at the screenshots, but I don't understand the specific issue you're running into. Would you mind going into more detail about what we can see on one of the screenshots? Is it that you see more errors than you used? Are there specific error (codes) that you're surprised to see (e.g. W293), or is it something with the interaction with pylance?

Hello. The first screenshot is the expected behavior. Typically a error caught by Pylance or other program would present side by side with ruff.

Now, as pylance picks up an error, not only is ruff not catching that particular error (in conjunction with Pylance) - the other errors elsewhere it had caught previously all just disappear (despite not being fixed yet).

In short, if pylance finds an error, 8 ruff errors just disappear. Further, a ruff check yields no errors found. Only the whitespace error and not the 8 others that are currently present and haven't been fixed. And not the error that pylance caught.

The second and third shot are to show that this happens above and below ruff's catches. And in the first shot you can see that Pylance and Ruff can and do report some of the same errors (as expected). But now certain Pylance errors seem to derail everything but Ruff formatting errors.

I want to blame Pylance.

Hope that makes it more clear and not more confusing . Thank you for your time.

Below are two sets of output from ruff check in the terminal for similar errors to the code shown above. First is with typical Ruff error checking.

If i add another error to my code (in this case forget the an = when declaring a constant) then NONE of the errors are highlighted. The second output is what I will see.

First Output

> ruff check C:\Users\LL\macromodule\desk_macropad\simplemacropad.py -q    
desk_macropad\simplemacropad.py:12:5: N807 Function name should not start and end with `__`
   |
10 | print("start ():", gc.mem_free())
11 |
12 | def __init__(self, bus: FourWire, **kwargs: Any):
   |     ^^^^^^^^ N807
13 |     super().__init__(bus, _INIT_SEQUENCE, **kwargs)
   |

desk_macropad\simplemacropad.py:12:5: ANN202 Missing return type annotation for private function `__init__`
   |
10 | print("start ():", gc.mem_free())
11 |
12 | def __init__(self, bus: FourWire, **kwargs: Any):
   |     ^^^^^^^^ ANN202
13 |     super().__init__(bus, _INIT_SEQUENCE, **kwargs)
   |
   = help: Add return type annotation: `None`

desk_macropad\simplemacropad.py:12:14: ARG001 Unused function argument: `self`
   |
10 | print("start ():", gc.mem_free())
11 |
12 | def __init__(self, bus: FourWire, **kwargs: Any):
   |              ^^^^ ARG001
13 |     super().__init__(bus, _INIT_SEQUENCE, **kwargs)
   |

desk_macropad\simplemacropad.py:12:25: F821 Undefined name `FourWire`
   |
10 | print("start ():", gc.mem_free())
11 |
12 | def __init__(self, bus: FourWire, **kwargs: Any):
   |                         ^^^^^^^^ F821
13 |     super().__init__(bus, _INIT_SEQUENCE, **kwargs)
   |

desk_macropad\simplemacropad.py:12:45: F821 Undefined name `Any`
   |
10 | print("start ():", gc.mem_free())
11 |
12 | def __init__(self, bus: FourWire, **kwargs: Any):
   |                                             ^^^ F821
13 |     super().__init__(bus, _INIT_SEQUENCE, **kwargs)
   |

desk_macropad\simplemacropad.py:13:27: F821 Undefined name `_INIT_SEQUENCE`   
   |
12 | def __init__(self, bus: FourWire, **kwargs: Any):
13 |     super().__init__(bus, _INIT_SEQUENCE, **kwargs)
   |                           ^^^^^^^^^^^^^^ F821
   |

desk_macropad\simplemacropad.py:26:8: F821 Undefined name `FONT`
   |
24 | WHITE_INV = BLACK
25 |
26 | font = FONT
   |        ^^^^ F821
27 |
28 | ORANGE = 0xFF8800
   |

desk_macropad\simplemacropad.py:425:9: ANN204 Missing return type annotation for special method `__call__`
    |
423 |                 mouse.release(mouse.LEFT_BUTTON)
424 |
425 |     def __call__(self):
    |         ^^^^^^^^ ANN204
426 |         self.send_code()
    |
    = help: Add return type annotation

Second Output

Note, I am adding whitespace and W293 simply to have an output to show in comparison. I can now confirm this occurs with Pylance Pyrite totally disabled and not highlighting errors. The extra error just makes my code look error free. The previous errors it noted are gone. And the missing = goes unnoticed. Same error laden code.

Previously, I had abundant error checking in all instances utilizing Ruff.

Now, certain errors in code (which are numerous basic errors) make it impossible to find ANY errors present in the code via Ruff. Those errors will not be highlighted or caught until i utilize another linter.

> ruff check C:\Users\LL\macromodule\desk_macropad\simplemacropad.py -q    
desk_macropad\simplemacropad.py:8:1: W293 [*] Blank line contains whitespace
   |
 6 | from keylayers import KT, enc_lyr, key_lyr
 7 | from simpler_macro_hardware import KEY_TO_PIXEL, SimplerMacropad
 8 |
   | ^ W293
 9 | gc.collect()
10 | print("start ():", gc.mem_free())
   |
   = help: Remove whitespace from blank line

After further checks, the behavior is also present with a fresh installation of Pylint enabled as the sole linter (Ruff and Pyrite/Pylance disabled).

I'm happy to close the issue since its upstream. If anyone has any advice on how best to pursue a solution or which portion of VSCode is the likely culprit, I'd greatly appreciate it. Regardless, thank you for your time and your awesome program. I'll be looking into other platforms to utilize ruff since my use case is pretty basic.

If i add another error to my code (in this case forget the an = when declaring a constant) then NONE of the errors are highlighted. The second output is what I will see.

This is a limitation of Ruff today. It can't parse files with syntax errors. That means most analyses are skipped if your code contains syntax errors. We're about to change this, but it will take us some time before we get there. However, ruff should highlight the first error for you

b.py:9:5: E999 SyntaxError: Unexpected token Newline
  |
7 | from ward import Scope, fixture  # pyright: ignore [reportUnknownVariableType]
8 | 
9 | from
  |      E999

I can now confirm this occurs with Pylance Pyrite totally disabled and not highlighting errors. The extra error just makes my code look error free. The previous errors it noted are gone. And the missing = goes unnoticed. Same error laden code.

I assume that's happening in VS Code? Do you see the error when running ruff check? Do you know if the errors that you used to see where from Pylance/PyRight or from Ruff?

astral-sh / ruff-vscode