Spyder project directory in src-folder Layout

[OyiboRivers]

Issue Report Checklist

• [X] Searched the issues page for similar reports

Problem Description

Hey Spyder Team,

First of all I want to say thank you for your time and effort. I appreciate the work that you do on the Spyder IDE.

There is a small issue which is related to feature suggestion Issue #21319 “Per project PYTHONPATH settings“.

I'm using Spyder as my IDE, and I prefer to work with a "src-folder" layout for my Python packages. In this layout, the source code resides within a sub-directory named "src." However, Spyder currently does not allow for the automatic recognition of the parent project folder as the project directory, causing import issues within the package.

Here is a project tree as example:

MyPackage ├── docs ├── pyproject.toml ├── src │   └── MyPackage (<= I usually select this as Spyder project directory) │   ├── init.py │   ├── module.py │   └── utils.py └── tests

Spyder uses the chosen path automatically as “current working directory“. That is necessary to import between modules of the package in development mode. You can import like this:

# module.py
from MyPackage.utils import some_function

I would actually like to be able to set the parent project folder as the project directory in Spyder. When using the "src-folder" layout, that is not possible automatically as import statements within the package would not work correctly.

MyPackage (<= I would like to choose this as Spyder project directory) ├── docs ├── pyproject.toml ├── src │   └── MyPackage │   ├── init.py │   ├── module.py │   └── utils.py └── tests

Unfortunately Spyder does not automatically recognize the package “src-folder“ layout. Import statements within “src“ would fail.

# module.py
from MyPackage.utils import some_function  # this would fail => ModuleNotFoundError
from src.MyPackage.utils import some_function  # this would work

I have found 3 workarounds: 1) Use “src“ folder as Spyder project folder (This is what I prefer) 2) Append “src-folder“ to “sys.path“ via code. This would be required for development only. Before compilation the code should be removed from the package. 3) Go to Spyder IDE “Tools-PYTHONPATH manager“ and add the “src-folder“ to “User paths“ manually. When you finish the project you could remove the path from “User paths“.

A “Per project PYTHONPATH“ would solve the issue as you could add “User paths“ per project, too. Is it possible to set the parent project folder as Spyder project directory in “src-folder“ layout and set the correct paths automatically?

Versions

• Spyder version: 5.4.3 (conda)
• Python version: 3.10.13 64-bit
• Qt version: 5.15.2
• PyQt5 version: 5.15.10
• Operating System: Linux 5.15.0-88-generic

[dalthviz]

Hi @OyiboRivers thank you for the feedback and sorry for the late response! That makes sense to me but what do you think @spyder-ide/core-developers ?

[mrclary]

I think this is a good idea. Perhaps we would just need to add logic to the Projects plugin where it sends the path to PYTHONPATH Manager: if <project path>/src exists, send <project path>/src (and <project path>?), otherwise send <project path>.

[CAM-Gerlach]

I have found 3 workarounds:

Just to note, since you presumably already have a Python distribution package (with pyproject.toml, etc), the "proper" approach to avoid these kinds of workarounds or relying on Spyder's magic is to just install your package into your working environment in editable mode, with pip install -e .. That way, it will work consistently inside or outside of Spyder, and also for anyone else using it, at least installing from a repo clone (or in most cases a packaged sdist/wheel; there are certain edge cases due to file includes and the like that pip install -e . won't catch but a packaged sdist/wheel would, but still way better than relying on PYTHONPATH hacking or magic tricks).

Perhaps we would just need to add logic to the Projects plugin where it sends the path to PYTHONPATH Manager: if <project path>/src exists, send <project path>/src (and <project path>?), otherwise send <project path>.

Right, and this would presumably happen transparently like the existing behavior. One concern though is that this makes Spyder's behavior a lot more "magical" and less consistent with other ways the user might try to import or execute the project outside of it, resulting in more user confusion as to why their project can be imported/executed in Spyder but not outside of it.

The one other issue is it defeats some of the explicit benefit of the src layout in actively avoiding the local checkout being imported instead of the actually installed version (pip install -e ., which we plan to make easier to do in Spyder 7), which can hide packaging issues that will only rear their heads once the package is installed on other machines—though may be an acceptable tradeoff in making the src layout more convenient for interactive use while also ensuring the installed package is properly tested externally, when running unit tests, on CI, etc (so long as that is actually done).

[mrclary]

One concern though is that this makes Spyder's behavior a lot more "magical" and less consistent with other ways the user might try to import or execute the project outside of it, resulting in more user confusion as to why their project can be imported/executed in Spyder but not outside of it.

Agreed. Another option, in addition to editable install, is allowing users to specify PYTHONPATHs on a per-project basis. This is a feature that I've wanted for some time. I think it could work by leveraging the existing PYTHONPATH Manager, but save the user paths persistently with the project meta-data.