search

microsoft / pylance-release

Documentation and issues for Pylance
Creative Commons Attribution 4.0 International
1.71k stars 765 forks source link

Globs in extra paths #5364

Open judej opened 9 months ago

judej commented 9 months ago

Discussed in https://github.com/microsoft/pylance-release/discussions/2712

Originally posted by **itsdani** February 21, 2021 I'm trying to setup a python monorepo build with bazel, and I'm struggling to make pylance recognize the library imports. I have managed to make it work by adding e.g. `/external//pypi__numpy` to `python.analysis.extraPaths`, but adding a line for each dependency on every computer we use doesn't seem to be right. My idea/request is that if we could add something like `dependencies/*/` or `dependencies/pypi__*/` to the extra paths, it would solve this problem entirely. The expected behavior would be to add every (matching) directory under `dependencies/` to the extra paths.
luabud commented 9 months ago

we just converted this back to an issue to add it to our roadmap

dpar39 commented 4 months ago

Any ETA on this?

erictraut commented 4 months ago

As has been explained in other threads, I don't think it will work to add globs for extraPaths. The problem is that extraPaths is order dependent. The order affects import resolution behavior. Globs is an order-independent mechanism. If you want to add extra paths to your project, you should specify them one at a time in the order you intend for imports to be resolved.

rchiodo commented 4 months ago

We could make it deterministic by saying globs resolve alphabetically.

heejaechang commented 3 months ago

@itsdani @dpar39 it has been a while but can you guys provide us more detail on the scenario? and we have add extra path code action to help these kinds of situation and wonder why that didn't work?

itsdani commented 3 months ago

We have given up on using Bazel and I work on very different things since I requested it 3 years ago, I'm not even using Pylance anymore. Maybe someone currently experiencing this issue could provide more detail. I can only repeat what I already said in the other thread: https://github.com/microsoft/pylance-release/discussions/2712#discussioncomment-6799676

adzenith commented 3 months ago

Bazel puts each Python dependency in its own directory for sandboxing reasons. Our extraPaths is 601 lines long, and we have to manually update it when our Python dependency tree changes.

Here's a small snippet so you can see what it looks like:

  "external/rules_python~~pip~pip_310_google_re2/site-packages",
  "external/rules_python~~pip~pip_310_google_resumable_media/site-packages",
  "external/rules_python~~pip~pip_310_googleapis_common_protos/site-packages",
  "external/rules_python~~pip~pip_310_graphviz/site-packages",
  "external/rules_python~~pip~pip_310_greenlet/site-packages",
  "external/rules_python~~pip~pip_310_grpcio/site-packages",
  "external/rules_python~~pip~pip_310_grpcio_status/site-packages",
  "external/rules_python~~pip~pip_310_grpcio_tools/site-packages",

It's that for 600 packages.

Instead if we could do something like "external/rules_python~~pip~pip_310_*/site-packages",, we would be all set and wouldn't have to keep the list in sync with Bazel.

Thanks!

heejaechang commented 3 months ago

@adzenith a few questions.

  1. is that list stable? will everyone who install dependency gets the same list? including re-installing the dependencies?
  2. how it works in runtime? will all these paths be included in PYTHONPATH? or Bazel provide some special import logic for this?
  3. is the pattern on how dependency installed customizable? in another word, will everyone (or most of users of Bazel) have dependency installed on external/rules_python.... ?

I am not sure we want to support glob on extraPath, but we might add support for Bazel env directly if it has some common patterns, we can add support for.

adzenith commented 3 months ago
  1. Yes, everyone has this same list. We check in the pyproject.toml file.
  2. Yes, Bazel builds a PYTHONPATH for each binary based on its declared dependencies. So you will have some subset of these available at runtime in Bazel depending on what you're running, and each available library will appear in PYTHONPATH.
  3. The pattern is not customizable. Bazel puts the dependencies where it wants. In our case we have a symlink, external, to the Bazel build output tree where the Python libs are stored.

I don't need a glob specifically; I just assumed that would be the easiest to implement. If Bazel's output tree if directly supported that would also suit my needs just fine. Thanks!