`pip wheel -r ...` does not pass `config_settings`

belm0 commented 11 months ago

Description

pip wheel correctly passes config_settings when given a requirement specifier directly, but not when given a requirements file.

Works:

pip wheel --config-settings "setup-args=-Dblas=blas" scipy==1.10.1

Doesn't work:

echo 'scipy==1.10.1' > r.txt
pip wheel --config-settings "setup-args=-Dblas=blas" -r r.txt

Expected behavior

config_settings are passed when using pip wheel with a requirements file

pip version

23.2.1 (also head and older versions)

Python version

3.8

OS

Linux

How to Reproduce

in an environment with gfortran intentionally missing, try to build scipy with pip wheel -r ... and setting setup-args
in the error output (due to missing gfortran), confirm whether setup-args propagated to the meson command line

`-r` case (broken)

pip install --upgrade pip wheel
echo 'scipy==1.10.1' > r.txt
pip wheel --config-settings "setup-args=-Dblas=blas" -r r.txt

direct specifier case

pip install --upgrade pip wheel
pip wheel --config-settings "setup-args=-Dblas=blas" scipy==1.10.1

Output

`-r` case (broken)

-Dblas is not passed to meson build

+ meson setup --prefix=/usr /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9 /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-oc75antm/build --native-file=/tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2

direct specifier case

+ meson setup --prefix=/usr /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7 /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-l963lj1w/build --native-file=/tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 -Dblas=blas

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

qci-amos commented 11 months ago

I just discovered this problem too. In my case, I'm using --config-settings editable_mode=compat in order for pylance to find my editable installations.

qci-amos commented 11 months ago

And it's not trivial to just convert my reqs into command line args. For example, I'd have to work out a new mechanism for environment variable interpolation and/or deal with changes in req sort order.

belm0 commented 11 months ago

This seems to be a regression, because it was working for me last year.

I wonder if the culprit is pyproject-hooks and/or the pyproject.toml transition.

belm0 commented 11 months ago

I've confirmed the discrepancy of -r vs. direct specifier this far:

in both cases, ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel() is called with config_settings=None
prepare_metadata_for_build_wheel() ignores config_settings anyway, and defers to self.config_holder.config_settings
for -r case, self.config_holder.config_settings is None
for direct specifier, self.config_holder.config_settings reflects the command line config_settings (e.g. {'setup-args': '-Dblas=blas'})

aside: it's odd that ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel() ignores the config_settings arg

https://github.com/pypa/pip/blob/496b268c1b9ce3466c08eb4819e5460a943d1793/src/pip/_internal/utils/misc.py#L757-L768

... tracing higher up, in bad case install_req_from_line() is called with config_settings=None

belm0 commented 11 months ago

here are backtraces for bad (passing config_settings=None) and good cases:

bad (-r)

Traceback (most recent call last):
  File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
    reqs = self.get_requirements(args, options, finder, session)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 436, in get_requirements
    req_to_add = install_req_from_parsed_requirement(
  File "/.../pip/src/pip/_internal/req/constructors.py", line 500, in install_req_from_parsed_requirement
    req = install_req_from_line(
  File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
    assert False, f'{config_settings=}'
AssertionError: config_settings=None

good

Traceback (most recent call last):
  File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
    reqs = self.get_requirements(args, options, finder, session)
  File "/.../pip/src/pip/_internal/cli/req_command.py", line 411, in get_requirements
    req_to_add = install_req_from_line(
  File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
    assert False, f'{config_settings=}'
AssertionError: config_settings={'setup-args': '-Dblas=blas'}

belm0 commented 11 months ago

in CLI get_requirements():

install_req_from_parsed_requirement() case:

config_settings=parsed_req.options.get("config_settings") if parsed_req.options else None

install_req_from_line() case:

config_settings=getattr(options, "config_settings", None)

So the code is clearly only expecting config_settings from the requirements entries, and ignoring the command line.

I suppose the settings should be merged, with command line taking precedence...

https://github.com/pypa/pip/blob/496b268c1b9ce3466c08eb4819e5460a943d1793/src/pip/_internal/cli/req_command.py#L431-L445

belm0 commented 11 months ago

I suppose the settings should be merged, with command line taking precedence...

Apparently, when #11634 added per-requirement config_settings, the PR initially tried to merge with the command line settings, but that was removed before the final version.

It appears intentional: https://github.com/pypa/pip/issues/11915#issuecomment-1500557768 and reinforced: https://github.com/pypa/pip/pull/11941

My use case for setting config_settings on the command line, and expecting it to propagate to dependency builds, is valid:

using pip wheel -r ... to build all dependencies of an application to target a less-popular Python implementation (Pyston)
need to control the blas library selected by scipy via config_settings

I'll investigate per-requirement config_settings for my use case. But at the least, it may be good for pip to issue a warning that the command-line config settings are being ignored. @sbidoul @pfmoore

Configuration settings provided via --config-settings command line options (or the equivalent environment variables or configuration file entries) are passed to the build of requirements explicitly provided as pip command line arguments. They are not passed to the build of dependencies, or to the build of requirements provided in requirement files.

belm0 commented 11 months ago

I'll investigate per-requirement config_settings for my use case.

It's not practical, because pip-compile doesn't appear to propagate settings from .in to .txt.

$ echo 'scipy==1.10.1 --config-settings="setup-args=-Dblas=blas"' > r2.in
$ pip-compile -r r2.in
numpy==1.24.4
    # via scipy
scipy==1.10.1
    # via -r r2.in

filed https://github.com/jazzband/pip-tools/issues/2000

uranusjr commented 10 months ago

I feel that config settings should not be merged (perferrably with a warning), but if config settings are only provided from one source that should be respected instead of being “cancelled”.

sbidoul commented 7 months ago

So as noted in https://github.com/pypa/pip/issues/12310#issuecomment-1758479394 the behaviour is intentional and was changed to resolve an inconsistency with passing config settings to the build of dependencies.

That does not mean it cannot be changed again. I think this would require a detailed specification of the desired behaviour, though. For instance I'm not sure what build backends should do with config settings they don't understand - is that defined in a standard? How should per-requirements config settings behave when there are "global" config settings? Should per-requirements config settings propagate to their dependencies or not?

qci-amos commented 7 months ago

I'll note that my use case is, as far as I can tell, due to some kind of disagreement between pylance and pip and I feel kinda "caught in the middle". Ultimately, I believe I shouldn't need to pass any cli arg for vscode to be able to find my editable installations, but I'm willing do so if pip/pylance say that's what I need to do.

pfmoore commented 7 months ago

@qci-amos Yes, it's a disagreement between two tools, if you want to look at it that way. The tools in question are setuptools and pylance. The problem is that setuptools implements editable installs by default in a way that pylance can't detect. The compat mode uses a different implementation that is recognised by pylance.

I'd suggest bringing this up with setuptools. They may have a way of setting a global (or per-project) configuration which chooses a different default implementation. Or they may have another way of working around this - I'd be surprised if you are the only person affected by this.

I know setuptools and the typing community have had discussions about this issue - the technical problem behind it is currently unresolved (it needs people to work on, and agree, a standard way of publishing the data that type checkers need from an editable install) as far as I know, but I don't know if it's something they are actively working on, or if there are acceptable workarounds for now (if there are, they would presumably help you as well).

qci-amos commented 7 months ago

Ok, thank you. Digging around in here I found many potentially related issues so I just picked one to cross-link.

I'd be surprised if you are the only person affected by this.

Yea, I agree! I and my users are not doing anything complicated as far as the build system itself is concerned. All I'm trying to do is install multiple repos in editable mode and not get orange squiggles all over my vscode tabs! My guess is that many devs out there assume it's their bug somehow but give up trying to figure it out and decide instead to just live with the squiggles.

pfmoore commented 7 months ago

Digging around in here I found many potentially related issues so I just picked one to cross-link.

The specific one you want is https://github.com/pypa/setuptools/issues/3518

qci-amos commented 7 months ago

Thank you! One suggestion made there that might be relevant here is to define an environment variable as an alternative to a cli arg. That might be easier to get working with -r and I could see this being useful beyond only the PEP-660 use case.

pfmoore commented 7 months ago

All of pip's command line options can be specified via environment variables - see the documentation

qci-amos commented 7 months ago

Ah, ok, it looks like I can get this to work:

export PIP_CONFIG_SETTINGS="editable_mode=compat"
pip install -e .

and this is a win for me because usually I don't install from a requirements file (and the cli arg is difficult for me in my day-to-day ad hoc work). However, this doesn't seem to see the env var:

export PIP_CONFIG_SETTINGS="editable_mode=compat"
echo "-e ." > reqs.txt
pip install -r reqs.txt

sbidoul commented 7 months ago

However, this doesn't seem to see the env var

Yes, that is because pip processes PIP_CONFIG_SETTINGS identically as the --config-settings CLI option. And these are applied to requirements passed as arguments only and do not currently propagate to requirements passed inside -r requirement files.

Note that with https://github.com/pypa/pip/pull/12494, you should be able to add --config-settings to -e lines in requirement files. Would that help in your use case?

qci-amos commented 7 months ago

Yes, @sbidoul I think that would work for me. Thank you!

qci-amos commented 5 months ago

For what it's worth, the env var solution I'd been using before, described above https://github.com/pypa/pip/issues/12310#issuecomment-1915385493, seems to have stopped working with pip 24, but I haven't had time to debug this.

pypa / pip