xcube plugins not auto-recognized when using `pyproject.toml`

[forman]

Describe the bug

xcube plugin packages are not automatically recognized as xcube plugins

1. although the package name is prefixed using xcube_;
2. if installed via pip -e .;
3. if using a pyproject.toml rather than setup.py.

To Reproduce

See plugin xcube-cmems that is forced to explicitly declare the entry point xcube_plugins in its pyproject.toml.

Expected behavior

Packages named xcube_<name> should automatically be recognized as xcube plugins.

[konstntokas]

The function _collect_plugins(list(iter_ent... finds the entry points. Therefore if the entry point is given explicitly in the pyproject.toml, it will find the xcube_cmems plugins.

The function _collect_plugins(discover_plu... iterates through the installed packages in the site-package folder of the environment. If xcube_cmems is installed with pip install -e ., the xcube_cmems folder is not put in the site-package folder of the environment. Therefore xcube_cmems can not be found. If xcube_cmems is installed with pip install . , the xcube_cmems folder is not put in the site-package folder of the environment, and the plugin is recognized.

[forman]

That is actually known. The issue is that now that we use the pyproject.toml the detection by name no longer functions. Before using pyproject.toml we used setup.py, which worked.

[konstntokas]

This is an issue with the pyproject.tomlfile in xcube-cmems. Discussion continues at xcube-dev/xcube-cmems#39

[konstntokas]

What does setuptools during editable installs

Using setuptools within pyproject.toml creates __editable__.<package>-<version>.pth in the site-packagesfolder, which gives information on the path to the package. (See https://setuptools.pypa.io/en/latest/userguide/development_mode.html). This is done in two different ways depending on the project structure (See https://discuss.python.org/t/help-testing-pep-660-support-in-setuptools/16904/44?page=3).

• If src/ layout is used, the __editable__.<package>-<version>.pth is static and contains the path to the packages, which is added to sys.pathand therefore the package can be found by pkgutil.iter_modules(). Therefore the plugin is recognized.
• If a flat hierarchy is used, like we do in all of the projects, __editable__.<package>-<version>.pth is dynamic and the python script __editable___<package>_<version>_finder.py is added. When starting a python script, __editable___<package>_<version>_finder.py adds a path hook to sys.path (https://setuptools.pypa.io/en/latest/userguide/development_mode.html), which adds the path to the package during import, but this can not be resolved by pkgutil.iter_modules(). Therefore the plugin is not recognized.

Solution

Resolve the package name from the __editable__.<package_name>-<version>.finder.__path_hook__ file in discover_plugin_modules, and add them to the variable entry_points.

[forman]

@konstntokas, thanks for the investigation!