Support for multi-root workspaces without extraPaths/PYTHONPATH

[bharath144]

I work in a project that has multiple components written in Python each in its own Git Repository. Each component gets built into an RPM that is then installed on a target. The components are used by one another.

When working on the project, I create a workspace in vscode containing the folders (git clones) for each of the components. A python module in one folder (git repository) refers to a module from another folder (git repository). I believe this setup falls under the multi-root workspace category.

I see that Pylance claims 'Native support for multi-root workspaces'. I was under the assumption that with Pylancem, I will be able to navigate to a module or an identifier that is in a different folder (git repository) within the same workspace; this wasn't possible with the vanilla Python extension.

Has anyone got this working or have I misunderstood what 'Native support for multi-root workspaces' means?

[jakebailey]

The support for native multi-root workspaces was mainly to indicate that a single language server handles all requests for all workspaces, doesn't need to be restarted on config changes or new workspaces, and so on. Previously with the old language server we would spawn a new LS for every workspace, which was resource intensive and caused issues like microsoft/vscode-python#5132.

I think in this case you're looking for cross-workspace features like navigation, which isn't fully supported at this time. This is similar to #133 (which reports some of those features not working). Internally, each workspace is its own distinct thing which is queried, but has no knowledge about any other workspace directly unless they load each other in some manner (so then things are cached and searchable).

I believe imports between workspaces should work, so long as extraPaths has been configured to make them aware of each other (otherwise they don't know where to find each other).

[bharath144]

I believe imports between workspaces should work, so long as extraPaths has been configured to make them aware of each other (otherwise they don't know where to find each other).

Just to point out, my setup has only one workspace containing multiple folders. The imported modules are from one of the folders within the workspace. I have set up extraPaths to point to the root of the module in each folder with no luck.

[heejaechang]

@bharath144 so, you are not actually using vscode's multi workspace but uses Rylance's extrapaths to have multiple execution environment?

[bharath144]

@bharath144 so, you are not actually using vscode's multi workspace but uses Rylance's extrapaths to have multiple execution environment?

I guess I am. Are you suggesting that I should split each of the folders into their own workspaces?

[heejaechang]

@bharath144 nope, when you said "multi-root workspaces", I assumed you were using vscode's "multi-root workspaces" support. not extraPaths of pylance.

[bharath144]

@bharath144 nope, when you said "multi-root workspaces", I assumed you were using vscode's "multi-root workspaces" support. not extraPaths of pylance.

My setup is similar to what's documented here https://code.visualstudio.com/docs/editor/multi-root-workspaces

[bharath144]

Come to think of it, what's mentioned in #133 is similar to my setup. The reporter of that issue has named his folders 'workspace 1' and 'workspace 2' that are part of a vscode workspace named 'pylance-multiroot'.

BTW, using extraPaths doesn't solve the problem either (assuming I am doing it correctly).

[heejaechang]

@bharath144 so what you linked above is vscode's "multi-root workspace" and currently, Pylance doesn't support navigating symbols from a different workspace. those symbols are considered as completely different symbols.

but you might be able to get similar effects if you are using "executionEnvironments" supported by Pylance (see examples on https://github.com/microsoft/pyright/blob/master/docs/configuration.md#sample-config-file)

but all those execution environments must get all symbols from same locations and must use same venv, platform and etc.

[bharath144]

but you might be able to get similar effects if you are using "executionEnvironments" supported by Pylance (see examples on https://github.com/microsoft/pyright/blob/master/docs/configuration.md#sample-config-file)

Thanks. Let me check that out.

[heejaechang]

@bharath144 "extraPaths" is for "import resolver" for libraries. it will expect your directory to be structured in a library way (ex, < module name >/ init ,py).

execution environment is like "multiroot workspace", it lets you define multiple user code folders with its own environment.

[ErwanDL]

@bharath144 I'm not sure my setup in #133 is exactly the same as what you described. I'm using an actual multi-root workspace in this issue, which means that workspace1 and workspace2 are imported as workspaces and not just folders.

The explorer should look like this if you are in a multi-root workspace :

Whereas it will simply look like this if you have multiple folders that are not imported as individual workspaces themselves :

[bharath144]

I'm not sure my setup in #133 is exactly the same as what you described. I'm using an actual multi-root workspace in this issue, which means that workspace1 and workspace2 are imported as workspaces and not just folders.

Ah, I see. OK, this is super confusing right now.

Also, I haven't had time to check pyright (suggested by @heejaechang).

[schperplata]

The same issue here. A copy-paste of description of my setup:

My project is built from a sources on a few locations. Each of this locations has it's own workspace and settings. My project add this locations to workspace via .env file and by standard "Add folder to workspace" action. While Jedi has no problem with linking to sources on other location (for example, go to definition in other source folder), MSL often fails and crashes until reload.

Without setting Pylance settings:

• python.analysis.extraPaths
• python.autoComplete.extraPaths

... Pylance is unable to resolve imports from other locations, added via .env file or folders added to workspace. In addition, even if added, they must be absolute paths (seems like environment variables were not properly resolved in my case). This issue was discussed in other VS Code python repos a few times now - there are:

• folders added to workspace
• .env files with PYTHONPATH
• pylint and its settings
• pylance settings

Ideally, everything should work from .env file which supports environment variables, so workspaces can be versioned and shared across many users. Currently, pylance is forcing me to duplicate paths in all fields as described above, which is hard to maintain and in general very confusing and error prone.

Is there a plan to simplify and generalize these scenarios with multi-folder workspaces/projects?

[jakebailey]

How to handle extra search paths has been a long standing discussion; there are so many ways to define the same thing that all need to somehow work together, and everyone has a different view of how these should work. Now that we have a language server that supports multi-root workspaces, that's yet another place we now need to ask about how search paths work (should they know about each other? Right now, we say no, not dissimilar to other LSs and langauges).

As it stands now, we don't support PYTHONPATH because we don't pass environment variables down from the Python extension; this was done in the old LS but may need to be done for Pylance as well (if we are definitely calling out to Python with them). Environment variables just have serious implications in that once set, they never change. If the Python extension starts us with a specific environment variable, it's that variable forever until a full LS restart (which is what we do now if an env var changes), unless we too start duplicating that code, accepting .env as yet another configuration source, watch it for changes, etc.

Things added via python.analysis.extraPaths should not need to be absolute, so please file a bug if that's not the case. The intention is that it's identical to python.autoComplete.extraPaths in behavior as if used with the old language server (this setting has a slightly different meaning in the extension's Jedi support, which is why we didn't want to continue reusing it).

[schperplata]

Of course, I understand things are complex. What I was trying to say is just that python & VSC configuration has been a pain from the very beginning - the lack of common approach between VSC python extension, VSC workspace folders, pylint args, autocomplete args, env files, ... is the reason that VSC never really worked for my case, where a bit more complex project is in place. I've been switching between Jedi, language server and others, trying to find a combination where everything would work, and I've only manage to get it usefull by using .env file and standard PYTHONPATH. And then extensions was upgraded and things slowly started to fall apart. The other common approach is l virtual env, which is not possible in my case.

However, don't get me wrong - I am still very happy with VS Code & python. Pylance, except these extraPaths, seems like an amazing thing and I will switch as soon as possible. It is just not really ready to be used on larger scale, where a lot of projects are in use and paths would have to be multiplied, as they are currently only set by .env file.

I know you will find the best way, thank you :)

[jakebailey]

We had been wanting to come up with some consistent messaging around search paths and a system to combine them for a while (that can be shared between us and vscode-python and its pylint integration), but we've just been working on getting Pylance out and ready for use by the public and the changeover impacted that effort.

I'll look into getting the core extension to get PYTHONPATH set as it would with the old LS.

You should be able to use .env and python.analysis.extraPaths together, though. Sure, it's duplicated, but it should work. Since we read the latter and not the former, there's no conflict, so we'd appreciate an issue or a test project we can use as an example during testing.

[erictraut]

@schperplata, do you need to use multi-root workspaces? Or are you open to using a single workspace?

Pylance is built on Pyright. I originally designed Pyright to work with a large, complex code base that my team is responsible for maintaining. We don't use multi-root workspaces for our code base. Instead, we use the "execution environment" support that's built into Pyright (and also supported by Pylance). For details about how to configure execution environments, check out this documentation. I'm curious whether that might work for you.

[schperplata]

@jakebailey Great. When I find a short free slot, I can do the example project. @erictraut Yes, I am using multi-root workspaces. As long as I can see sources from all these paths and run their relative pytests, it is fine by me. Don't know how to achieve this with single workspace.

Workspace: It is basically a few folders on some paths with py sources, and a single VSC workspace where these folders are added. Until now, .env file is adding these paths to PYTHONPATH so imports were possible and VSC pylint/autocomplete is able to find sources. I will check out Pyright. However, I wish to keep as little files and dependencies as possible, so hopefully no new files are needed with only .extraPaths.

[jakebailey]

Revisiting this:

Until now, .env file is adding these paths to PYTHONPATH so imports were possible and VSC

Pylance has supported PYTHONPATH for a bit now; are you still experiencing issues?

[schperplata]

No. Also, using py projects as packages and using virtal env solved most of these issues. Thank you.

[judej]

Closing old issue. If this is still a problem, please reopen with the information requested. thanks