Signal extension YAML file path not compatible with src-layout in editable install

hakonanes commented 1 month ago

Describe the bug

Hi all,

The derived path to the signal extension file hyperspy_extension.yaml is not compatible with an extension package using a src-layout in an editable install (pip install -e .).

The reason is that the name and not the value of the entry point is used when creating the path. The following

https://github.com/hyperspy/hyperspy/blob/e20d095930c77c1fdb892afd68ca2c456ec76a79/hyperspy/extensions.py#L65

should use _external_extension.value instead.

This way, a project that wants to tell HyperSpy where to find the hyperspy_extension.yaml file this way

[project.entry-points."hyperspy.extensions"]
kikuchipy = "src/kikuchipy"

can do so (name = "value").

An example src-layout:

|__ hyperspy_sample_extension_src_layout
    |__ pyproject.toml
    |__ src
        |__ hspy_ext_src_layout
            |__ __init__.py
            |__ hyperspy_extension.yaml
            |__ signal.py
            |__ model.py
            |__ component.py

To fix this, I intend to make changes here:

extensions.py on RELEASE_next_patch
Update https://hyperspy.org/hyperspy-doc/current/dev_guide/writing_extensions.html#registering-extensions with details for extensions using pyproject.toml, and what name = "value" actually means
Include this example repository somewhere in the HyperSpy organization?

To Reproduce

Fetch the example repository with src-layout and pyproject.toml

$ git clone git@github.com:hakonanes/hyperspy_sample_extension_src_layout.git
...

Create environment, print contents of pyproject.toml, and install dependencies and package (in editable mode)

$ cd hyperspy_sample_extension_src_layout
$ conda create -n hs-ext python=3.12 -y
...
$ conda activate hs-ext
$ cat pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "hspy_ext_src_layout"
license = {file = "LICENSE"}
version = "0.1.0"
requires-python = ">= 3.10"
dependencies = [
    "hyperspy >= 2",
    "numpy",
    "pytest",
    "traits",
]

[project.entry-points."hyperspy.extensions"]
hspy_ext_src_layout = "src/hspy_ext_src_layout"

$ pip install -e .
...

Attempt to print known signal types and notice how HyperSpy looks for the incorrect path to hyperspy_extension.yaml

$ python -c "import hyperspy.api as hs; hs.print_known_signal_types()"
(hs-ext) hakon:hyperspy_sample_extension_src_layout [main] $ python -c "import hyperspy.api as hs; hs.print_known_signal_types()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/hs-ext/lib/python3.12/site-packages/hyperspy/api.py", line 23, in <module>
    from hyperspy.defaults_parser import preferences
  File "/opt/homebrew/Caskroom/miniforge/base/envs/hs-ext/lib/python3.12/site-packages/hyperspy/defaults_parser.py", line 28, in <module>
    from hyperspy.ui_registry import add_gui_method
  File "/opt/homebrew/Caskroom/miniforge/base/envs/hs-ext/lib/python3.12/site-packages/hyperspy/ui_registry.py", line 34, in <module>
    from hyperspy.extensions import ALL_EXTENSIONS, _external_extensions
  File "/opt/homebrew/Caskroom/miniforge/base/envs/hs-ext/lib/python3.12/site-packages/hyperspy/extensions.py", line 71, in <module>
    with open(str(_path)) as stream:
         ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/hakon/kode/personal/hyperspy_sample_extension_src_layout/hspy_ext_src_layout/hyperspy_extension.yaml'

Expected behavior

$ python -c "import hyperspy.api as hs; hs.print_known_signal_types()"
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
+-------------+---------+------------+---------------------+
| signal_type | aliases | class name |       package       |
+-------------+---------+------------+---------------------+
|   MySignal  |         |  MySignal  | hspy_ext_src_layout |
+-------------+---------+------------+---------------------+

Python environment:

HyperSpy version: 2.1.1
Python version: 3.12

Additional context

If not installing in editable mode, the path is correct (because the expected path is correct due to everything within src/ is found in site-packages):

$ cd hyperspy_sample_extension_src_layout
$ pip install .
$ python -c "import hyperspy.api as hs; hs.print_known_signal_types()"
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
WARNING | Hyperspy | Numba is not installed, falling back to non-accelerated implementation. (hyperspy.decorators:256)
+-------------+---------+------------+---------------------+
| signal_type | aliases | class name |       package       |
+-------------+---------+------------+---------------------+
|   MySignal  |         |  MySignal  | hspy_ext_src_layout |
+-------------+---------+------------+---------------------+

ericpre commented 1 month ago

Include this example repository somewhere in the HyperSpy organization?

Maybe more simple to add branches for different configuration / layout and list these in the readme?

hakonanes commented 1 month ago

Yes, that is a better solution.

If #3446 goes through, I can make a PR to https://github.com/hyperspy/hyperspy_sample_extension which after review can be pushed as a branch.

Edit: With a separate PR to update the README pointing to this branch.

hyperspy / hyperspy