Editable install doesn't work with pylint

[cimes-isi]

I've created a demo project to reproduce this issue: https://github.com/cimes-isi/py-build-cmake-editable-pylint

In short, pylint produces errors when trying to import from a Python project built with py-build-cmake and installed as editable. The issue does not occur for normal installs.

I see the following differences with pip freeze. A setuptools-based project adds:

-e git+ssh://git@github.com/cimes-isi/py-build-cmake-editable-pylint.git@c0f9e2ca54368e646724ed7268488708d33d1a69#egg=foo&subdirectory=foo-setuptools
foo-setuptools==0.1.0

but a py-build-cmake project only adds:

-e git+ssh://git@github.com/cimes-isi/py-build-cmake-editable-pylint.git@c0f9e2ca54368e646724ed7268488708d33d1a69#egg=foo&subdirectory=foo-py-build-cmake

It seems like maybe the editable install is incomplete?

Thanks!

[tttapa]

Thanks for the detailed report!

The problem with editable installations for C/C++ extensions is that your code gets spread out over two locations: the editable Python source files live in your project directory, and the compiled extensions live somewhere else (they are usually not installed into the source folders of the project).

The way py-build-cmake handles this is as follows:

• The Python source files simply remain in your project folder, they are not installed to the site-packages folder when you do an editable install.
• The compiled extension modules and other files installed by CMake are installed to the site-packages folder. This is fine, because you don't expect them to be “editable”, if you change your C/C++ source files, you'll have to trigger a rebuild before the changes take effect.
• A wrapper __init__.py script is installed to the site-packages folder. Its job is to add the project folder to Python's search path (specifically, to __spec__.submodule_search_locations), so the editable Python source files are located correctly. When that is done, it locates and executes the actual __init__.py script in the project folder.
• Additionally, a .pth file is installed to site-packages, pointing to the project folder as well. This is not necessary when importing the package, but it helps IDEs and other tools locate the source files.

The main limitation of this approach is that it doesn't work with tools that expect all files and modules to be in a single directory. Many tools have their own rules for setting search paths and locating modules, which usually differ slightly from what the actual Python interpreter does. Furthermore, many tools do not search paths in __spec__.submodule_search_locations, or never even execute the code that edits this list.

What happens when you import a submodule sub of your (editable) package foo is as follows:

• Python searches in its search path and finds site-packages/foo as a first match.
• Python executes site-packages/foo/__init__.py
• The project dev folder is added to the submodule search path, which tells Python to search for submodules foo.sub in both the project folder and in the existing search path (site-packages/foo)
• The project folder's __init__.py script is located and executed, and its global variables are loaded into the current wrapper's global scope, which makes it seem as if the project folder's __init__.py was the one that was actually found first.
• After __init__.py is done, Python searches for the foo.sub module (in both folders) and executes and imports it.

Other tools don't follow these same steps. For example, they could do something like this:

• Find the first foo folder in the search path.
• Look for the requested import foo.sub in that folder.
• If it doesn't exist, give up. In other words, if there are two folders foo in the search path, the second one is never considered.

For tools with this logic, I don't think there is a way to support editable installations with compiled extension modules.
I'm not familiar with the search mechanism that pylint uses internally, and a quick glance at the command line options reveals no way to set the path (except perhaps using the init hooks?). I'm afraid that I don't have enough free time to look into this at the moment.

Some tools allow you to manually specify search paths, for example, here's what I use for pyright:

https://github.com/kul-optec/QPALM/blob/f2aeb0c3e3ebc4fcd1ebf02fda340d7ec96a2d25/pyproject.toml#L81-L87

An editable install using setuptools is quite different: the only thing that gets installed to the site-packages folder is a .egg-link file that points to your project folder. This expects your project folder to contain a valid egg project, including all compiled extension modules and generated metadata files.

It seems like maybe the editable install is incomplete?

I think the difference you're seeing is simply caused by different mechanisms used by setuptools and py-build-cmake: setuptools supported editable installs way before there was any official standard for it, whereas py-build-cmake uses the mechanisms defined by PEP 517 and PEP 660 (which is just a little over one year old at this point).

In theory, the install should be complete, in the sense that it contains all files necessary to correctly set up the search paths to locate the editable files, as well as all files that are not in the project folder, but which are usually part of an install (e.g. entry points, stubs, compiled extension modules).

You can verify exactly which files py-build-cmake installs using pip's --no-clean option:

$ pip install -e . -v --no-clean
...
Stored in directory: /tmp/pip-ephem-wheel-cache-somethingsomething
...

You can then open the .whl file in that folder as a ZIP and inspect all files.

[cimes-isi]

Thank you for the extraordinarily detailed explanation! I think I understand the approach. I started with a report here b/c the editable install works with setuptools, which for better or worse is the baseline. If you think that py-build-cmake is compliant with the stated PEPs and not in violation of others, it seems reasonable to report an issue with pylint. I could submit the issue myself, but frankly I think you'd be able to make a much stronger case to them, esp. about which PEP(s) they might be incompatible with.

FYI it does seem that pylint respects PYTHONPATH at least, e.g., PYTHONPATH=src pylint test.py works as expected in the test project.

[tttapa]

I've added two new editable installation modes in 0.1.5.
Documentation: https://tttapa.github.io/py-build-cmake/Editable-install.html

Symlink mode should be the one that causes the least friction with tools like pylint. The reason that it is not the default is that Windows does not support symbolic links for ordinary users without developer mode enabled.