Ability to remove python's dunder functions from the code completion list

[erictraut]

Moving this issue from pyright to pylance, since it's related to LSP features and not core type checking.

For full request refer to https://github.com/microsoft/pyright/issues/1090

[jakebailey]

As far as I'm aware, we derank dunder methods, so it would surprise me that they would show up mingled with other members. @WillAvudim do you have an example of this happening?

[WillAvudim]

Indeed, I do.

For example,

from dataclasses_json import dataclass_json
from dataclasses import dataclass, field

@dataclass_json
@dataclass
class TrainAndEvalXgboostModel_Request:
  tensor_storage: str

  from_unix: int
  to_unix: int

  label_index: int
  label_less_than_threshold: float

  xgboost_eta: float
  xgboost_max_depth: int

req.<Invoke the code completion window right after this point>

I currently see all members listed once with a blue box, then they are all listed the second time in the same window all over again with the yellow box, each time followed by the list of all dunder functions. I did earlier observed that some dunders came ahead, but cannot reproduce it at the moment. I'll try to be more cautious next time I run into it again, to understand the circumstances when I get that behavior. It would be awesome not to see them ever again at some point, for I have already learned all that list and can simply write them without code completion (never really needed them so far, though, and I'm writing a lot of python code, primary related to our pytorch and xgboost endeavors).

[erictraut]

If you see items listed twice, that probably means you have multiple Python language servers running. I'm guessing that you have Pyright and Jedi running along side each other. VS Code is doing its best to combine the answers from both.

I recommend doing one of the following:

1. Uninstall Pyright and install Pylance. The latter has all of the functionality of the former and then some. If you install Pylance, it will take the place of Jedi.
2. If you want to stick with Pyright instead of Pylance for some reason, then disable Jedi by setting "python.languageServer" to "None".

Let us know if that fixes the problem.

[WillAvudim]

I uninstalled Pyright, reloaded VSCode, installed Pylance, reloaded (both times by completely shutting down VSCode and starting it up anew). I ensured that I've got "python.languageServer": "Pylance", in settings.

At first, all I could see were dunder functions, then my members indeed appeared and they are no longer doubled, so that's indeed fixed!

However I now get red warnings all over the file since it seems to be unable to find the stub files for the modules:

import numpy as np
       ~~~~~
(module) numpy
Stub file not found for "numpy" Pylance (reportMissingTypeStubs)

What's surprising, code completion works correctly. It sees both numpy, and all np.* members. I see that python.analysis.stubPath='./typings' by default and I don't have such a directory at the root of my project. Am I supposed to download those from somewhere?

Here's what Palance language server current prints out:

[Info  - 6:31:18 PM] Pylance language server 2020.9.8 (pyright a0620921) starting
[Info  - 6:31:18 PM] Server root directory: /home/user/.vscode/extensions/ms-python.vscode-pylance-2020.9.8/dist
[Error - 6:31:18 PM] stubPath /storage/mono/typings is not a valid directory.

Switching back to Pyright didn't help! Hmmm, somehow I lost the ability to do "python.analysis.typeCheckingMode": "strict". Setting "python.languageServer": "Jedi", also doesn't seem to help.

[erictraut]

Pylance is built on top of Pyright, so you should see little or no difference in behavior between them. There are a few minor differences that are worth noting:

1. In Pylance, "python.analysis.typeCheckingMode" defaults to "off". In Pyright, it defaults to "basic". You're providing a value for this setting, so the default doesn't matter.
2. In Pylance, "python.analysis.useLibraryCodeForTypes" defaults to true. In Pyright, it defaults to false. This explains why you're getting code completions for numpy when using Pylance.
3. Pylance ships with some additional type stubs for libraries like matplotlib and pandas. These stubs are incomplete and are not useful for strict type checking. They are sufficient for completion suggestions.

It sounds like you are trying to use "strict" type checking. That's difficult to do today given the dearth of available type information for most libraries. I recommend that you start with "basic" type checking. You can then enable additional diagnostic rules on top of basic. The "reportMissingTypeStubs" diagnostic rule requires that all libraries either have inlined types (i.e. a py.typed file is specified as per PEP 561) or type stubs are provided.

[WillAvudim]

I got it working, thank you for your help!

They key was indeed "python.analysis.typeCheckingMode": "basic", and it apparently takes quite a while to scan everything. Initially it couldn't catch non-existing members, e.g. it was not red-lining np.ndarray2 and other non-existing members. But after a while it started highlighting non-existing members and recognizing the existing fields correctly. I'd say it took about ~30 minutes for it to start catching all the types (and I have top of the line hardware with lots of RAM).

(I don't have access to close this issue.)

[erictraut]

It shouldn't take more than a few milliseconds to analyze each file. My team's code base consists of about 3K files, and it finishes that in about 20sec. If you're seeing times in the minutes, there's something wrong. Either you're analyzing way more files than you should be (e.g. you're including a very large virtual environment in your workspace) or you are hitting a piece of code that is triggering some n^2 behavior in the type analyzer. If it's the latter, I'd like to know more about it. If you set "python.analysis.logLevel" to "Trace" and then go to your Output window, you should see a detailed breakdown of the time it takes to parse, bind, and analyze each file. If any of them are over 10K ms, that's an indication that something is very wrong.

[WillAvudim]

You're correct. I've enabled tracing and I see that it normally takes 0-2 ms per file. I observe a few outliers running into ~200ms range, however they are not reproducible. If I restart VSCode, they drop into the regular range of 0-2ms. I am no longer capable of reproducing the initial behavior I observed immediately after the switch at this point. Now it just works.