Open belm0 opened 11 months ago
I just discovered this problem too. In my case, I'm using --config-settings editable_mode=compat
in order for pylance to find my editable installations.
And it's not trivial to just convert my reqs into command line args. For example, I'd have to work out a new mechanism for environment variable interpolation and/or deal with changes in req sort order.
This seems to be a regression, because it was working for me last year.
I wonder if the culprit is pyproject-hooks and/or the pyproject.toml
transition.
I've confirmed the discrepancy of -r
vs. direct specifier this far:
ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel()
is called with config_settings=None
prepare_metadata_for_build_wheel()
ignores config_settings
anyway, and defers to self.config_holder.config_settings
-r
case, self.config_holder.config_settings
is None
self.config_holder.config_settings
reflects the command line config_settings (e.g. {'setup-args': '-Dblas=blas'}
)aside: it's odd that ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel()
ignores the config_settings
arg
... tracing higher up, in bad case install_req_from_line()
is called with config_settings=None
here are backtraces for bad (passing config_settings=None
) and good cases:
bad (-r
)
Traceback (most recent call last):
File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
reqs = self.get_requirements(args, options, finder, session)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 436, in get_requirements
req_to_add = install_req_from_parsed_requirement(
File "/.../pip/src/pip/_internal/req/constructors.py", line 500, in install_req_from_parsed_requirement
req = install_req_from_line(
File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
assert False, f'{config_settings=}'
AssertionError: config_settings=None
good
Traceback (most recent call last):
File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
reqs = self.get_requirements(args, options, finder, session)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 411, in get_requirements
req_to_add = install_req_from_line(
File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
assert False, f'{config_settings=}'
AssertionError: config_settings={'setup-args': '-Dblas=blas'}
in CLI get_requirements()
:
install_req_from_parsed_requirement()
case:
config_settings=parsed_req.options.get("config_settings") if parsed_req.options else None
install_req_from_line()
case:
config_settings=getattr(options, "config_settings", None)
So the code is clearly only expecting config_settings from the requirements entries, and ignoring the command line.
I suppose the settings should be merged, with command line taking precedence...
I suppose the settings should be merged, with command line taking precedence...
Apparently, when #11634 added per-requirement config_settings, the PR initially tried to merge with the command line settings, but that was removed before the final version.
It appears intentional: https://github.com/pypa/pip/issues/11915#issuecomment-1500557768 and reinforced: https://github.com/pypa/pip/pull/11941
My use case for setting config_settings
on the command line, and expecting it to propagate to dependency builds, is valid:
pip wheel -r ...
to build all dependencies of an application to target a less-popular Python implementation (Pyston)scipy
via config_settings
I'll investigate per-requirement config_settings for my use case. But at the least, it may be good for pip
to issue a warning that the command-line config settings are being ignored. @sbidoul @pfmoore
Configuration settings provided via
--config-settings
command line options (or the equivalent environment variables or configuration file entries) are passed to the build of requirements explicitly provided as pip command line arguments. They are not passed to the build of dependencies, or to the build of requirements provided in requirement files.
I'll investigate per-requirement config_settings for my use case.
It's not practical, because pip-compile
doesn't appear to propagate settings from .in
to .txt
.
$ echo 'scipy==1.10.1 --config-settings="setup-args=-Dblas=blas"' > r2.in
$ pip-compile -r r2.in
numpy==1.24.4
# via scipy
scipy==1.10.1
# via -r r2.in
I feel that config settings should not be merged (perferrably with a warning), but if config settings are only provided from one source that should be respected instead of being “cancelled”.
So as noted in https://github.com/pypa/pip/issues/12310#issuecomment-1758479394 the behaviour is intentional and was changed to resolve an inconsistency with passing config settings to the build of dependencies.
That does not mean it cannot be changed again. I think this would require a detailed specification of the desired behaviour, though. For instance I'm not sure what build backends should do with config settings they don't understand - is that defined in a standard? How should per-requirements config settings behave when there are "global" config settings? Should per-requirements config settings propagate to their dependencies or not?
I'll note that my use case is, as far as I can tell, due to some kind of disagreement between pylance
and pip
and I feel kinda "caught in the middle". Ultimately, I believe I shouldn't need to pass any cli arg for vscode to be able to find my editable installations, but I'm willing do so if pip/pylance say that's what I need to do.
@qci-amos Yes, it's a disagreement between two tools, if you want to look at it that way. The tools in question are setuptools
and pylance
. The problem is that setuptools implements editable installs by default in a way that pylance can't detect. The compat
mode uses a different implementation that is recognised by pylance.
I'd suggest bringing this up with setuptools. They may have a way of setting a global (or per-project) configuration which chooses a different default implementation. Or they may have another way of working around this - I'd be surprised if you are the only person affected by this.
I know setuptools and the typing community have had discussions about this issue - the technical problem behind it is currently unresolved (it needs people to work on, and agree, a standard way of publishing the data that type checkers need from an editable install) as far as I know, but I don't know if it's something they are actively working on, or if there are acceptable workarounds for now (if there are, they would presumably help you as well).
Ok, thank you. Digging around in here I found many potentially related issues so I just picked one to cross-link.
I'd be surprised if you are the only person affected by this.
Yea, I agree! I and my users are not doing anything complicated as far as the build system itself is concerned. All I'm trying to do is install multiple repos in editable mode and not get orange squiggles all over my vscode tabs! My guess is that many devs out there assume it's their bug somehow but give up trying to figure it out and decide instead to just live with the squiggles.
Digging around in here I found many potentially related issues so I just picked one to cross-link.
The specific one you want is https://github.com/pypa/setuptools/issues/3518
Thank you! One suggestion made there that might be relevant here is to define an environment variable as an alternative to a cli arg. That might be easier to get working with -r
and I could see this being useful beyond only the PEP-660 use case.
All of pip's command line options can be specified via environment variables - see the documentation
Ah, ok, it looks like I can get this to work:
export PIP_CONFIG_SETTINGS="editable_mode=compat"
pip install -e .
and this is a win for me because usually I don't install from a requirements file (and the cli arg is difficult for me in my day-to-day ad hoc work). However, this doesn't seem to see the env var:
export PIP_CONFIG_SETTINGS="editable_mode=compat"
echo "-e ." > reqs.txt
pip install -r reqs.txt
However, this doesn't seem to see the env var
Yes, that is because pip processes PIP_CONFIG_SETTINGS identically as the --config-settings
CLI option.
And these are applied to requirements passed as arguments only and do not currently propagate to requirements passed inside -r
requirement files.
Note that with https://github.com/pypa/pip/pull/12494, you should be able to add --config-settings
to -e
lines in requirement files. Would that help in your use case?
Yes, @sbidoul I think that would work for me. Thank you!
For what it's worth, the env var solution I'd been using before, described above https://github.com/pypa/pip/issues/12310#issuecomment-1915385493, seems to have stopped working with pip 24, but I haven't had time to debug this.
Description
pip wheel
correctly passesconfig_settings
when given a requirement specifier directly, but not when given a requirements file.Works:
Doesn't work:
Expected behavior
config_settings
are passed when usingpip wheel
with a requirements filepip version
23.2.1 (also head and older versions)
Python version
3.8
OS
Linux
How to Reproduce
gfortran
intentionally missing, try to build scipy withpip wheel -r ...
and settingsetup-args
gfortran
), confirm whethersetup-args
propagated to themeson
command line-r
case (broken)direct specifier case
Output
-r
case (broken)-Dblas
is not passed to meson build+ meson setup --prefix=/usr /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9 /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-oc75antm/build --native-file=/tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
direct specifier case
+ meson setup --prefix=/usr /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7 /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-l963lj1w/build --native-file=/tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 -Dblas=blas
Code of Conduct