search

pypa / flit

Simplified packaging of Python modules
https://flit.pypa.io/
BSD 3-Clause "New" or "Revised" License
2.15k stars 131 forks source link

Editable build could be broken when user miss docstring for description #534

Open ileixe opened 2 years ago

ileixe commented 2 years ago

I had a weird behavior which one of the two packages were working fine for editable installation, but the other one was failed to install emitting No matching distribution found

After digging a bit, I found I had a mistake not to write docstring with dynamic description.

From the code (https://github.com/pypa/flit/blob/main/flit_core/flit_core/buildapi.py#L34), when user missed the required docstring, flit returns the package which will be checked as build dependency. I'm using pip for frontend and pip seems to consider the build dependencies to be installed (https://github.com/pypa/pip/blob/main/src/pip/_internal/distributions/sdist.py#L117) with ignore_installed=True flag as build dependencies should (or may?) be installed from the clean environment.

It results in distribution discovery failure when user use dev branch without publishing yet (e.g. 0.0.1dev).

I'm not sure whether it's front-end issue or flit side, but it seems to be more related to flit side as I thought it's OK to set ignore_installed=true for build dependencies.

As I took a lot of time to figure it out so I made this issue to hope someone who will have same mistake like me to get a viable hint, and please let me know if flit is not the place to solve the issue.

Thanks.

takluyver commented 2 years ago

Sorry, I'm a bit confused. Could you describe exactly what you're trying and what error you see?

I've just tried removing the docstring from a package and then doing pip install --editable . with the latest pip, and while it's rather verbose, it does include the relevant error message from Flit:

$ pip install --editable .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/takluyver/Code/flit/flit_core/flit_core/tests/samples/pep621
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      Traceback (most recent call last):
        File "/home/takluyver/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/takluyver/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/takluyver/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 191, in prepare_metadata_for_build_editable
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-7drhdtkz/overlay/lib/python3.10/site-packages/flit_core/buildapi.py", line 49, in prepare_metadata_for_build_wheel
          metadata = make_metadata(module, ini_info)
        File "/tmp/pip-build-env-7drhdtkz/overlay/lib/python3.10/site-packages/flit_core/common.py", line 408, in make_metadata
          md_dict.update(get_info_from_module(module, ini_info.dynamic_metadata))
        File "/tmp/pip-build-env-7drhdtkz/overlay/lib/python3.10/site-packages/flit_core/common.py", line 211, in get_info_from_module
          raise NoDocstringError(
      flit_core.common.NoDocstringError: Flit cannot package module without docstring, or empty docstring. Please add a docstring to your module (/home/takluyver/Code/flit/flit_core/flit_core/tests/samples/pep621/module1a.py).
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
ileixe commented 2 years ago

Sorry I was not clear enough. I myself was confused to understand my description after a few days. :joy:

Here's how to reproduce

(.venv) issue534$ tree
.
├── bar
│   ├── bar
│   │   ├── __init__.py
│   └── pyproject.toml
└── foo
    ├── foo
    │   ├── __init__.py
    └── pyproject.toml

Actually there were two packages involved. I confirmed that just one package worked fine emitting your log

Package foo has valid doc string, and installed correctly.

(.venv) issue534$ cat foo/pyproject.toml
[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "foo"
version = "0.1.0.dev0"
dynamic = ["description"]
(.venv) issue534$ cat foo/foo/__init__.py
"""foo"""

But the problem is the other package bar. It depends on foo package and no docstring there. (This was my mistake)

(.venv) issue534$ cat bar/pyproject.toml
[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "bar"
version = "0.1.0.dev0"
dynamic = ["description"]

dependencies = ["foo == 0.1.*"]
(.venv) issue534$ cat bar/bar/__init__.py
(.venv) issue534$

If I try to install bar with editable flag after foo installed, it failed like below

(.venv) bar$ pip list
Package    Version    Editable project location
---------- ---------- -----------------------------------------------
foo        0.1.0.dev0 /home/ys/Practice/languages/python/issue534/foo
pip        22.0.4
setuptools 58.1.0

(.venv) bar$ pip install -e .
Obtaining file:///home/ys/Practice/languages/python/issue534/bar
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      ERROR: Could not find a version that satisfies the requirement foo==0.1.* (from versions: none)
      ERROR: No matching distribution found for foo==0.1.*
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
(.venv) bar$

If I add the missing docstring to bar, I managed to install the package then.

(.venv) bar$ cat bar/__init__.py
"""bar"""
(.venv) bar$ pip install -e .
Obtaining file:///home/ys/Practice/languages/python/issue534/bar
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: foo==0.1.* in /home/ys/Practice/languages/python/issue534/.venv/lib/python3.10/site-packages (from bar==0.1.0.dev0) (0.1.0.dev0)
Building wheels for collected packages: bar
  Building editable for bar (pyproject.toml) ... done
  Created wheel for bar: filename=bar-0.1.0.dev0-py2.py3-none-any.whl size=937 sha256=3571d18954142a35b0760ff330019a5d384b4590b5067a9a1a41074745c0fee4
  Stored in directory: /tmp/pip-ephem-wheel-cache-k7zor9xj/wheels/86/88/6e/96d01046ed08574cdea413fa27de12cb2dfe4780faea4805bf
Successfully built bar
Installing collected packages: bar
Successfully installed bar-0.1.0.dev0
pradyunsg commented 2 years ago

Can you run the failing pip install run with -vvv and post the output of that?

ileixe commented 1 year ago

Sorry for too late response. I did not recognize @pradyunsg

Here's verbose log for failure

(.venv) bar$ pip install -e .
Obtaining file:///home/ys/Source/Practice/languages/python/issue534/bar
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [4 lines of output]
      ERROR: Could not find a version that satisfies the requirement foo==0.1.* (from versions: none)
      ERROR: No matching distribution found for foo==0.1.*
      WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
      You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
(.venv) bar$
(.venv) bar$ pip install -e . -vvv
Using pip 22.0.4 from /home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-d3ak3__e
Created temporary directory: /tmp/pip-req-tracker-cmvalpf9
Initialized build tracking at /tmp/pip-req-tracker-cmvalpf9
Created build tracker: /tmp/pip-req-tracker-cmvalpf9
Entered build tracker: /tmp/pip-req-tracker-cmvalpf9
Created temporary directory: /tmp/pip-install-kkffeywm
Obtaining file:///home/ys/Source/Practice/languages/python/issue534/bar
  Added file:///home/ys/Source/Practice/languages/python/issue534/bar to build tracker '/tmp/pip-req-tracker-cmvalpf9'
  Created temporary directory: /tmp/pip-build-env-q5r_5__7
  Created temporary directory: /tmp/pip-standalone-pip-mxs4yfbf
  Running command pip subprocess to install build dependencies
  Using pip 22.0.4 from /tmp/pip-standalone-pip-mxs4yfbf/__env_pip__.zip/pip (python 3.9)
  Collecting flit_core<4,>=3.2
    Using cached flit_core-3.8.0-py3-none-any.whl (62 kB)
  Installing collected packages: flit_core
  Successfully installed flit_core-3.8.0
  WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
  You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
  Installing build dependencies ... done
  Running command Checking if build backend supports build_editable
  Checking if build backend supports build_editable ... done
  Running command Getting requirements to build editable
  Getting requirements to build editable ... done
  Created temporary directory: /tmp/pip-standalone-pip-9l28clcb
  Running command pip subprocess to install backend dependencies
  Using pip 22.0.4 from /tmp/pip-standalone-pip-9l28clcb/__env_pip__.zip/pip (python 3.9)
  ERROR: Could not find a version that satisfies the requirement foo==0.1.* (from versions: none)
  ERROR: No matching distribution found for foo==0.1.*
  WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
  You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/ys/Source/Practice/languages/python/issue534/.venv/bin/python /tmp/pip-standalone-pip-9l28clcb/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-q5r_5__7/normal --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'foo == 0.1.*'
  cwd: [inherit]
  Installing backend dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Exception information:
Traceback (most recent call last):
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 339, in run
    requirement_set = resolver.resolve(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 75, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 506, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 188, in _make_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 313, in __init__
    super().__init__(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 323, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 609, in prepare_editable_requirement
    dist = _get_prepared_distribution(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 45, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 108, in _install_build_reqs
    self.req.build_env.install_requirements(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/build_env.py", line 202, in install_requirements
    self._install_requirements(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/build_env.py", line 260, in _install_requirements
    call_subprocess(
  File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: pip subprocess to install backend dependencies exited with 1
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
Removed file:///home/ys/Source/Practice/languages/python/issue534/bar from build tracker '/tmp/pip-req-tracker-cmvalpf9'
Removed build tracker: '/tmp/pip-req-tracker-cmvalpf9'
(.venv) bar$ vi bar/__init__.py

After adding docstring,

(.venv) bar$ pip install -e . -vvv
Using pip 22.0.4 from /home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-axr297gn
Created temporary directory: /tmp/pip-req-tracker-xkr7xz8v
Initialized build tracking at /tmp/pip-req-tracker-xkr7xz8v
Created build tracker: /tmp/pip-req-tracker-xkr7xz8v
Entered build tracker: /tmp/pip-req-tracker-xkr7xz8v
Created temporary directory: /tmp/pip-install-buk43w7s
Obtaining file:///home/ys/Source/Practice/languages/python/issue534/bar
  Added file:///home/ys/Source/Practice/languages/python/issue534/bar to build tracker '/tmp/pip-req-tracker-xkr7xz8v'
  Created temporary directory: /tmp/pip-build-env-chnpzrnq
  Created temporary directory: /tmp/pip-standalone-pip-hwmh6ahi
  Running command pip subprocess to install build dependencies
  Using pip 22.0.4 from /tmp/pip-standalone-pip-hwmh6ahi/__env_pip__.zip/pip (python 3.9)
  Collecting flit_core<4,>=3.2
    Using cached flit_core-3.8.0-py3-none-any.whl (62 kB)
  Installing collected packages: flit_core
  Successfully installed flit_core-3.8.0
  WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
  You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
  Installing build dependencies ... done
  Running command Checking if build backend supports build_editable
  Checking if build backend supports build_editable ... done
  Running command Getting requirements to build editable
  Getting requirements to build editable ... done
  Created temporary directory: /tmp/pip-modern-metadata-6rneano9
  Running command Preparing editable metadata (pyproject.toml)
  Preparing editable metadata (pyproject.toml) ... done
  Source in /home/ys/Source/Practice/languages/python/issue534/bar has version 0.1.0.dev0, which satisfies requirement bar==0.1.0.dev0 from file:///home/ys/Source/Practice/languages/python/issue534/bar
  Removed bar==0.1.0.dev0 from file:///home/ys/Source/Practice/languages/python/issue534/bar from build tracker '/tmp/pip-req-tracker-xkr7xz8v'
Requirement already satisfied: foo==0.1.* in /home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages (from bar==0.1.0.dev0) (0.1.0.dev0)
Created temporary directory: /tmp/pip-unpack-zpc7ihtt
Building wheels for collected packages: bar
  Created temporary directory: /tmp/pip-wheel-dp4idaus
  Destination directory: /tmp/pip-wheel-dp4idaus
  Running command Building editable for bar (pyproject.toml)
  Building editable for bar (pyproject.toml) ... done
  Created wheel for bar: filename=bar-0.1.0.dev0-py2.py3-none-any.whl size=945 sha256=95edbb8c5ea7f41367c850e2cb38817f1cc8fbee0182156add07a78dc64fd2fd
  Stored in directory: /tmp/pip-ephem-wheel-cache-axr297gn/wheels/aa/e2/a6/6e6128a1a5ff0f61c0869a6a92f6594ef38861ed4e5dcfa201
Successfully built bar
Installing collected packages: bar

Successfully installed bar-0.1.0.dev0
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
Removed build tracker: '/tmp/pip-req-tracker-xkr7xz8v'
takluyver commented 1 year ago

I think I understand, and this is more or less expected, albeit not ideal.

Flit tries to get the version number & docstring by static analysis first - i.e. parsing the code but not executing it. If that doesn't work, it falls back to importing your module, i.e. executing its top-level code. If it's trying that, any dependencies of your package become build dependencies, because they may need to be installed to import it. Pip builds your package in an isolated environment by default, so although foo is already installed in your target environment, it's not installed in the build environment, and pip can't find it to install.

I think using flit install --symlink or --pth-file instead of pip install -e . will likely fail with a clearer error message.