PYTHONPATH not being used in Windows Subsystem for Linux Ubuntu

[PAGWatson]

Problem Description

The PYTHONPATH in my terminal does not seem to be used by spyder (and I've started spyder from the command line). In the console, sys.path just contains

['/home/<username>/miniconda3/envs/<env name>/lib/python310.zip',
 '/home/<username>/miniconda3/envs/<env name>/lib/python3.10',
 '/home/<username>/miniconda3/envs/<env name>/lib/python3.10/lib-dynload',
 '',
 '/home/<username>/miniconda3/envs/<env name>/lib/python3.10/site-packages']

os.environ['PYTHONPATH'] also does not exist, though other custom environment variables are present in os.environ, seeming to suggest that something has acted to exclude importing the PYTHONPATH variable. If I use ipython in the terminal in the same environment, sys.path and os.environ['PYTHONPATH'] contain all the directories in PYTHONPATH.

This is in a fresh spyder install in a miniconda environment. I'm opening Spyder from a MobaXterm terminal in a Windows 10 Subsystem for Linux Ubuntu installation.

I'm coming back to spyder after a while, so perhaps there are some extra steps I need to perform that I don't know - I don't remember needing to do anything special to use PYTHONPATH in spyder before, and I can't find documentation on it.

Versions

• Spyder version: 5.5.0
• Python version: 3.10.13
• Qt version: 5.15.8
• PyQt version: 5.15.9
• Operating System name/version: Ubuntu 20.04 in Windows 10 Subsystem for Linux

Dependencies

Mandatory:

atomicwrites >=1.2.0 : 1.4.1 (OK) chardet >=2.0.0 : 5.2.0 (OK) cloudpickle >=0.5.0 : 3.0.0 (OK) cookiecutter >=1.6.0 : 2.5.0 (OK) diff_match_patch >=20181111 : 20230430 (OK) intervaltree >=3.0.2 : 3.1.0 (OK) IPython >=8.13.0,<9.0.0,!=8.17.1 : 8.18.1 (OK) jedi >=0.17.2,<0.20.0 : 0.19.1 (OK) jellyfish >=0.7 : 1.0.3 (OK) jsonschema >=3.2.0 : 4.20.0 (OK) keyring >=17.0.0 : 24.3.0 (OK) nbconvert >=4.0 : 7.12.0 (OK) numpydoc >=0.6.0 : 1.6.0 (OK) parso >=0.7.0,<0.9.0 : 0.8.3 (OK) pexpect >=4.4.0 : 4.8.0 (OK) pickleshare >=0.4 : 0.7.5 (OK) psutil >=5.3 : 5.9.7 (OK) pygments >=2.0 : 2.17.2 (OK) pylint >=2.5.0,<3.1 : 3.0.3 (OK) pylint_venv >=3.0.2 : 3.0.3 (OK) pyls_spyder >=0.4.0 : 0.4.0 (OK) pylsp >=1.9.0,<1.10.0 : 1.9.0 (OK) pylsp_black >=1.2.0,<3.0.0 : 1.3.0 (OK) qdarkstyle >=3.2.0,<3.3.0 : 3.2 (OK) qstylizer >=0.2.2 : 0.2.2 (OK) qtawesome >=1.2.1 : 1.3.0 (OK) qtconsole >=5.5.0,<5.6.0 : 5.5.1 (OK) qtpy >=2.1.0 : 2.4.1 (OK) rtree >=0.9.7 : 1.1.0 (OK) setuptools >=49.6.0 : 68.2.2 (OK) sphinx >=0.6.6 : 7.2.6 (OK) spyder_kernels >=2.5.0,<2.6.0 : 2.5.0 (OK) textdistance >=4.2.0 : 4.5.0 (OK) three_merge >=0.1.1 : 0.1.1 (OK) watchdog >=0.10.3 : 3.0.0 (OK) xdg >=0.26 : 0.28 (OK) zmq >=22.1.0 : 25.1.2 (OK)

Optional:

cython >=0.21 : None (NOK) matplotlib >=3.0.0 : 3.8.2 (OK) numpy >=1.7 : 1.26.2 (OK) pandas >=1.1.1 : None (NOK) scipy >=0.17.0 : 1.11.4 (OK) sympy >=0.7.3 : None (NOK)

[vaire-simon]

Same problem. It's important when you're managing a bunch of environment data for multiple tools in a Linux system. I want to connect the user to a specific set of python modules that can be simply imported from a script that they run. I don't want to install them into Python because the version may change or it might affect other users. I don't want to use conda virtual environments because I need to manage the environment of many tools - not just Spyder - and this is best done with environment variables and environment modules.

[smax1]

One more vote. The use case for PYTHONPATH is when a large project configuration is done via environment variables. Python is just one component of such project (e.g. an EDA project). The scripts and tools normally honor PYTHONPATH and work as expected. Spyder 4.1.5 does honor PYTHONPATH, so integration with the rest of the system is flawless and automatic. Spyder 5.5 completely ignores PYTHONPATH and offers no option to honor it. While I can imagine that small-scale, purely python project may benefit from ignoring PYTHONPATH, this is a major hinderance for large-scale enterprise one.

[mrclary]

@PAGWatson, @vaire-simon, @smax1, Thank you for reporting this issue. I agree that PYTHONPATH is an important tool and Spyder intends to honor this for user's.

A little explanation. PYTHONPATH is removed from Spyder's internal runtime environment at startup in order to protect it from possibly unsafe search paths introduced by the user. This should only impact Spyder's application operation. The paths in a user's system PYTHONPATH should be seen in the PYTHONPATH Manager (Tools -> PYTHONPATH Manager) under the heading System paths. These paths should be passed on to the IPython Console and language server and should appear in sys.path in the user's console (not Spyder's internal console).

However, since you are not seeing this, there is certainly a bug or oversight here. I suspect that it is related to Spyder running on Windows subsystem for Linux. Spyder obtains the system PYTHONPATH on Windows machines by looking at the Windows Environment Variables. On Linux and macOS it obtains it from the shell startup files (~/.bashrc, ~/.bash_profile, etc.). What I suspect is happening here is that Spyder, being launched from the subsystem for Linux, is looking in these files rather than the Windows Environment Variables.

Can you confirm the following?

• PYTHONPATH is defined in Windows Environment Variables
• PYTHONPATH is not defined in ~/.bashrc or ~/.bash_profile, as seen by the subsystem for Linux.
• If you export PYTHONPATH=... in one of these files, as seen by the subsystem for Linux, do the paths show up in PYTHONPATH Manager?

[smax1]

I can confirm that on my Linux (CentOS) PYTHONPATH is defined and exported in environment variables and when I start spyder from that environment, the value of PYTHONPATH does NOT show up in PYTHONPATH Manager. Same for Cygwin64 on Windows 10.

[mrclary]

I can confirm that on my Linux (CentOS) PYTHONPATH is defined and exported in environment variables and when I start spyder from that environment, the value of PYTHONPATH does NOT show up in PYTHONPATH Manager. Same for Cygwin64 on Windows 10.

Python path is defined in Windows Environment Variables, exported to the CentOS environment, but not defined in any shell startup files, correct?

Could you add an export statement to a shell startup file and see if that changes anything?

[smax1]

Python path is defined in Windows Environment Variables, exported to the CentOS environment, but not defined in any shell startup files, correct?

No, standalone CentOS, nothing to do with Windows. Never tried under WSL

export PYTHONPATH is indeed in .bashrc

Same for Cygwin64 on Windows: export PYTHONPATH is in .bashrc

[Flamefire]

I see this on a Linux system:

$ export PYTHONPATH=$PYTHONPATH:/foo
$ spyder 

And then in the console:

In[1]: import os; os.environ['PYTHONPATH']
Out[1]: '/home/alex/pylint_plugins'

So to me it looks like only the PYTHONPATH set in the .bashrc is used not the one from where spyder is started

So basically confirming the suspicion of @mrclary which can be reduced to:

What [..] is happening here is that Spyder [...] is looking in these files rather than the [...] Environment Variables.

[PAGWatson]

Thanks for looking into this. In my Windows subsystem case, PYTHONPATH is being defined in my .bashrc. So it doesn't seem to fit the theory I'm afraid.

[Flamefire]

@mrclary What I found is that it is $SPY_PYTHONPATH that ends up in the console and that is determined from spyder_pythonpath in the config which in turns is the result of https://github.com/spyder-ide/spyder/blob/5f4c5cd326ff3b2bbf7b96b95d9ad787aa2c0bb5/spyder/plugins/pythonpath/utils.py#L33

However there is some filtering: All non-existant paths are removed (which is why my /foo didn't show up) and all lib[...]site-packages or lib[...]dist-packages are also removed.

The latter will be a problem for our use case where we do the equivalent of

$ pip install foo --prefix=/opt/foo
$ export PYTHONPATH=/opt/foo/lib/python3.8/site-packages:$PYTHONPATH

@PAGWatson Is it something similar for you? What is the content of your PYTHONPATH?

[vaire-simon]

Hi there - I was using RHEL8 Linux at the time and set PYTHONPATH in the .bashrc.

I wanted to use PYTHONPATH to make our modules available to users and to add custom Spyder widgets.

It was a while back, so I've worked around the problem since then - specifically I used pip install -e - so that I could continue developing modules and use them.

[mrclary]

Thanks for looking into this. In my Windows subsystem case, PYTHONPATH is being defined in my .bashrc. So it doesn't seem to fit the theory I'm afraid.

Is PYTHONPATH defined in your Windows Environment Variables?

[mrclary]

I see this on a Linux system:

$ export PYTHONPATH=$PYTHONPATH:/foo
$ spyder 

And then in the console:

In[1]: import os; os.environ['PYTHONPATH']
Out[1]: '/home/alex/pylint_plugins'

So to me it looks like only the PYTHONPATH set in the .bashrc is used not the one from where spyder is started

So basically confirming the suspicion of @mrclary which can be reduced to:

What [..] is happening here is that Spyder [...] is looking in these files rather than the [...] Environment Variables.

This is correct. To be clear, in this particular example, /foo is not excluded because it does not exist. While Spyder does filter non-existent directories from PYTHONPATH before sending it on to the IPython Console (but perhaps it shouldn't?), in this case /foo does not appear because it is not in ~/.bashrc or ~/.bash_profile.

Technically, Spyder does not look in these files, rather it uses a subprocess to start an interactive login shell with a clean environment and extracts the environment variables from there. Hence only those environment variables defined in a shell startup file will be present in the IPython Console's os.environ.

[mrclary]

Python path is defined in Windows Environment Variables, exported to the CentOS environment, but not defined in any shell startup files, correct?

No, standalone CentOS, nothing to do with Windows. Never tried under WSL

export PYTHONPATH is indeed in .bashrc

Same for Cygwin64 on Windows: export PYTHONPATH is in .bashrc

@smax1 Okay, I misunderstood. So for a Linux (or any posix) system with PYTHONPATH defined in a shell startup file, the paths in PYTHONPATH should show up under the heading System paths in the PYTHONPATH Manager as well as os.environ and sys.path in the IPython Console, unless the paths do not exist or they match the pattern of a Python environment library, as mentioned by @Flamefire (https://github.com/spyder-ide/spyder/issues/21633#issuecomment-2434586281).

If your PYTHONPATH meets those criteria and still does not show up in PYTHONPATH Manager, sys.path or os.environ, then we'll need to investigate in further detail what may be happening in your situation.

[ccordoba12]

unless the paths do not exist or they match the pattern of a Python environment library

This is very important. We do that to avoid people mixing libraries from different envs or installations through our Pythonpath manager. And I don't think we're going to change that because it prevents a lot of headaches for users that don't know that that's basically not possible.

[ccordoba12]

However, we're talking about overriding our cleaning of PYTHONPATH at startup and when creating kernels through an env var (see #22681).

[smax1]

Technically, Spyder does not look in these files, rather it uses a subprocess to start an interactive login shell with a clean environment and extracts the environment variables from there. Hence only those environment variables defined in a shell startup file will be present in the IPython Console's os.environ.

Not a good idea. It is user's responsibility to setup PYTHONPATH correctly though .bashrc or otherwise. This penalizes responsible users and does not bring any value to irresponsible ones.

At least add an option for users to configure Spyder to simply take PYTHONPATH from the environment.

[mrclary]

However, we're talking about overriding our cleaning of PYTHONPATH at startup and when creating kernels through an env var (see #22681).

I should clarify regarding this issue as well. The proposed option to not clean PYTHONPATH will only affect Spyder runtime and Spyder-kernel runtime and would have no impact on user's code. At least that's how I am envisioning its implementation.

The treatment of PYTHONPATH for user's code (i.e. IPython Console sys.path and os.environ) is a separate issue. I can see that for @Flamefire's situation in #22681, we must have some discussion regarding lib/.../[site|dist]-package type paths. But let's isolate that conversation to #22681 so that we don't hijack this issue.

[Flamefire]

While Spyder does filter non-existent directories from PYTHONPATH before sending it on to the IPython Console (but perhaps it shouldn't?), in this case /foo does not appear because it is not in ~/.bashrc or ~/.bash_profile. Technically, Spyder does not look in these files, rather it uses a subprocess to start an interactive login shell with a clean environment and extracts the environment variables from there. Hence only those environment variables defined in a shell startup file will be present in the IPython Console's os.environ.

Actually that is only done when spyder is not started from a terminal. If it is $PYTHONPATH is used from the environment of the shell (as expected).
I guess the system environment variables (e.g. from Windows) are defined in both cases, aren't they? I'm not sure if/why the environment will be different in the newly started shell compared to the environment spyder is currently using and if that should be handled differently.

It is user's responsibility to setup PYTHONPATH correctly though .bashrc or otherwise. This penalizes responsible users and does not bring any value to irresponsible ones.

At least add an option for users to configure Spyder to simply take PYTHONPATH from the environment.

The proposed option to not clean PYTHONPATH will only affect Spyder runtime and Spyder-kernel runtime and would have no impact on user's code.

So we have 2 related issues on where PYTHONPATH can be honored: Spyder runtime/kernels and the user environment/console. They both have in common, that the current clean/filtering is done to "protect" novice users from making common mistakes.
And for both we want/need a I-know-what-I-am-doing switch to turn those handrails off.
In the other issue I proposed to just issue a warning such that those novice users know why an issue may appear and what to do when the behavior is not intended. I'd say the same reasoning applies to e.g. the filtering of site-packages-like paths. We need that to put pip install --prefix-paths into the environment, although a workaround exists.

Having said that: Using PYTHONPATH is actually problematic for other reasons too: When creating a virtualenv on top the packages from the virtualenv will not be preferred over those in PYTHONPATH. This is why we reduce our usage of it in favor of a sitecustomize.py, but that needs to be in PYTHONPATH but at least not in a site-packages folder.

[mrclary]

Technically, Spyder does not look in these files, rather it uses a subprocess to start an interactive login shell with a clean environment and extracts the environment variables from there. Hence only those environment variables defined in a shell startup file will be present in the IPython Console's os.environ.

It is user's responsibility to setup PYTHONPATH correctly though .bashrc or otherwise.

I agree.

This penalizes responsible users and does not bring any value to irresponsible ones.

How so?

At least add an option for users to configure Spyder to simply take PYTHONPATH from the environment.

Spyder does this already for the case where it is launched from a shell. Nevertheless, this is not necessary. The PYTHONPATH in the current Spyder runtime environment is the result of two inputs: the definition in a shell startup and any modifications made by the user after starting the shell but before launching Spyder. If Spyder is launched from a shell, then both are reflected in the PYTHONPATH Manager. However, the designed method for modifying the PYTHONPATH from the shell startup definition is the PYTHONPATH Manager, which allows users to inject, remove, or even just disable paths.

[mrclary]

Technically, Spyder does not look in these files, rather it uses a subprocess to start an interactive login shell with a clean environment and extracts the environment variables from there. Hence only those environment variables defined in a shell startup file will be present in the IPython Console's os.environ.

Actually that is only done when spyder is not started from a terminal. If it is $PYTHONPATH is used from the environment of the shell (as expected).

True. That was a recent change to fix a specific esoteric bug; but that could change in the future.

[ccordoba12]

Not a good idea. It is user's responsibility to setup PYTHONPATH correctly though .bashrc or otherwise. This penalizes responsible users and does not bring any value to irresponsible ones. At least add an option for users to configure Spyder to simply take PYTHONPATH from the environment.

@smax1, I don't understand what you mean by this. If you set PYTHONPATH in your .bashrc, then it should be picked up by Spyder 6 using the process described by @mrclary. Is that not the case for you?

[mrclary]

For the sake of the original post, @PAGWatson, I'm not sure how the WSL interacts with Spyder. Can you please answer the following questions?

• Is PYTHONPATH defined in your Windows Environment Variables?
• Is PYTHONPATH defined in any shell startup files in WSL?
• What version of Spyder are you running?
• Can you share the result of $ echo $PYTHONPATH executed in the shell environment from which you launch Spyder?

[smax1]

Not a good idea. It is user's responsibility to setup PYTHONPATH correctly though .bashrc or otherwise. This penalizes responsible users and does not bring any value to irresponsible ones. At least add an option for users to configure Spyder to simply take PYTHONPATH from the environment.

@smax1, I don't understand what you mean by this. If you set PYTHONPATH in your .bashrc, then it should be picked up by Spyder 6 using the process described by @mrclary. Is that not the case for you?

In large EDA projects (Electronic Design Automation) at many/most semiconductor companies each project has an environment setup file. The file is version-controlled. The file meticulously configures the project environment by setting the bash environment to point to the right versions of all the tool dependencies. Often times, this is done via environment modules: https://en.wikipedia.org/wiki/Environment_Modules_(software)

This is an important use case for most people working in EDA. It is sad that Spyder is no longer easy to use in such setting. Not even an option to take PYTHONPATH straight in. This is how responsible users are penalized.

I do appreciate your work on Spyder, but it is getting more and more difficult to use.

[Flamefire]

The file meticulously configures the project environment by setting the bash environment to point to the right versions of all the tool dependencies. Often times, this is done via environment modules:

Just to add to this: This is the same for us working on HPC, we also use environment modules.

The PYTHONPATH in the current Spyder runtime environment is the result of two inputs: the definition in a shell startup and any modifications made by the user after starting the shell but before launching Spyder. If Spyder is launched from a shell, then both are reflected in the PYTHONPATH Manager.

In short: if Spyder is launched from a shell, the environment set up by the user should be reflected in the PYTHONPATH manager. Whether this is done by the .bashrc or environment modules shouldn't matter. Do you see that too @PAGWatson @smax1 ?

That was a recent change to fix a specific esoteric bug; but that could change in the future.

I'm surprised by that because this "recent change" is almost all* required to resolve the current issue. If that is changed we are back to this inability to actively customize the environment

*almost all because of the related issue that PYTHONPATH is ignored for the Spyder internals for which it could be required nonetheless, e.g. when using environment modules for its dependencies.

[PAGWatson]

For the sake of the original post, @PAGWatson, I'm not sure how the WSL interacts with Spyder. Can you please answer the following questions?

Is PYTHONPATH defined in your Windows Environment Variables?

I don't think so. It's just set in my .bashrc in my WSL. PYTHONPATH is not displayed if I run set in Windows Command Prompt.

Is PYTHONPATH defined in any shell startup files in WSL?

Yes, in .bashrc, using the "export" command.

What version of Spyder are you running?

The OP was with 5.5.0, and I've not checked what happens with later versions - I've not used spyder that much since.

Can you share the result of $ echo $PYTHONPATH executed in the shell environment from which you launch Spyder?

It's rather long as it contains probably ~20-30 directories, but here are the first couple and the last to show the pattern, with <...> indicating text strings I thought I'd take out (with no special characters apart from _):

/home/<username>/Research_python_code/Python_git_repos/<repo_name>:/home/<username>/Research_python_code/Python_git_repos/<repo_name2>:.../home/<username>/Research_python_code/Python_gen:

[mrclary]

I'm surprised by that because this "recent change" is almost all* required to resolve the current issue. If that is changed we are back to this inability to actively customize the environment

Yes, it does work in your favor at the moment. I say it may change because the that particular fix bifurcates the behavior of Spyder based on its launch mechanism, which I don't think is a good idea. Spyder should behave the same way, regardless of launch mechanism.

@Flamefire and @smax1, I am committed to resolving the issue regarding your situations, e.g. accommodating disparate package locations and compatibility with environment modules. However, I think that this is separate from (though related to) the current issue. I recommend continuing our conversation at #22681.

[mrclary]

@PAGWatson, thank you for answering those questions; this is helpful. I'm sorry, just a few more questions:

• How did you install Spyder? Was it a standalone Windows application? or did you install it via conda in WSL?

• If you start an interactive Python prompt from a WSL shell, what system does Python think it's running on?

  $ python
  
  >>> import os; os.name

I suspect that the latter will return 'nt', in which case Spyder is trying to get PYTHONPATH from your Windows Environment Variables rather than the shell startup files in WSL. If this is true, then you could add PYTHONPATH to your Windows Environment Variables (either the user or the system) and see if paths shows up in PYTHONPATH Manager in Spyder.

[PAGWatson]

How did you install Spyder? Was it a standalone Windows application? or did you install it via conda in WSL?

conda install -c conda-forge spyder in WSL.

If you start an interactive Python prompt from a WSL shell, what system does Python think it's running on?

It returns 'posix'.

[mrclary]

How did you install Spyder? Was it a standalone Windows application? or did you install it via conda in WSL?

conda install -c conda-forge spyder in WSL.

If you start an interactive Python prompt from a WSL shell, what system does Python think it's running on?

It returns 'posix'.

Excellent, thank you. So, in this circumstance, if Spyder is launched from the WSL shell prompt, then it should be getting PYTHONPATH from the shell startup files. The only thing that I can recommend right now is to try updating to Spyder 5.5.6 or Spyder 6.0.1. I would recommend creating a separate conda environment in WSL.

$ conda create -n spy-env python=3.11 spyder=6.0.1
$ conda run -n spy-env spyder

[PAGWatson]

Unfortunately my computer just died, so it may be a while before I could test this...