search

microsoft / pylance-release

Documentation and issues for Pylance
Creative Commons Attribution 4.0 International
1.71k stars 765 forks source link

Clarification on interaction between python.analysis.exclude and indexing of venv #6200

Closed adamh-oai closed 1 month ago

adamh-oai commented 3 months ago

We have a large python monorepo (can share stats if needed) and VSCode + Pylance performance/reliability is a constant struggle.
Running in a separate nodejs process has helped, but now it's just slow instead of OOMing.

One solution we're using is a dependency-aware workspace generator that only includes the project you're working on + transitive deps in python.analysis.include.

We install the relevant python projects into the venv editably (as links from the venv back into the workspace). Someone discovered that if you exclude everything via python.analysis.exclude, Pylance will still seem to index the files via the venv but it seems to use less memory + be faster. I'd like to better understand how the indexing between workspace files included by python.analysis.include interacts with indexing of files found in the venv; I'd expect there's some difference in behavior but am unsure why the performance is different, and what we're giving up with this approach.

top-oai commented 3 months ago

To add a few details, the specific exclusion setting we're experimenting with is python.analysis.exclude": ["**", "*", "**/*"], but since we've installed packages into our venv editably, our source code indeed still gets referenced through the venv.

heejaechang commented 3 months ago

First, here is some doc around how to set up large workspace with pylance - https://github.com/microsoft/pylance-release/wiki/Opening-Large-Workspaces-in-VS-Code

...

Pylance has 2 different kinds of indexing. user file indexing, and third-party library indexing (basically, site-packages)

user file indexing will index files up to python.analysis.userFileIndexingLimit. files that are matched by python.analysis.include/exclude are considered as user files

third-party library indexing will index files in site-packages based on python.analysis.packageIndexDepths. by default, it will index only __init__ at the package folder.

see link for more detail.

...

that said, for your case, since you do editable install, even if you reduce user files using include/exclude, it will be indexed as third party libraries. so you probably need to exclude ones you don't need through packageIndexDepths as well.

by the way, is indexing only thing that is causing the issue? if it is, you can turn off indexing by python.analysis.indexing. we also have this wiki to use pylance as light mode