Various package-index filtering flags do not affect the environment markers

[pradyunsg]

This is a common pattern that I'm seeing in pip's issue tracker; so filing an issue to (a) trigger a discussion of how to improve documentation / output / errors to better deal with the mismatch in user expectations vs behaviour here and (b) consolidate those issues.

Basically, the fundamental problem here is that pip has various options that affect the wheels that pip will consider when triaging things (--platform <platform>, --python-version <python_version>, --implementation <implementation>, --abi <abi>) but those do not affect the marker evaluation environment for dependencies.

This results in a subtle failure mode: pip install [options] package will use a wheel for package that has a (hypothetical) Python version 9.22 but when evaluating environment markers (https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers), that would still use the environment for the current Python interpreter.

It is not possible to compute the environment markers based on the values passed in by the user via the CLI.

[pradyunsg]

https://github.com/pypa/pip/issues/10050#issuecomment-1113147833 has the relevant piece from that issue.

[pradyunsg]

I think the right move here is doing what pex has done, and taking a complete environment specification including environment markers; and not inferring anything from the interpreter when that's done. It'll be a slightly disruptive change, like any other change, but we'll be in a better spot in terms of this.

Thoughts on this?

[pfmoore]

Agreed, this sounds better. In fact, I'd like to go even further and completely isolate environment discovery - not just marker values, but sysconfig locations, and anything else that we introspect from the Python interpreter. That would allow us to manage any sort of environment, even "artificial" ones constructed from information supplied independently of the running interpreter.

I'm thinking of this in support of --python, or --target/"install scheme" types of option.

I'm happy to limit the change to markers as an incremental step, if that's easier to achieve, but I'd like to have our long-term goal be to have full isolation of "interpreter introspection".

[pradyunsg]

I really don't want to allow the user to specify arbitrary paths with schemes (this was extensively discussed during installer's API/CLI design discussions) because I'm wary of what hacky layouts people might expect to be "supported" if we allow that.

I am OK with exposing basically everything else, since misuse/abuse of them seems less likely/problematic to me.

[pfmoore]

I wasn't suggesting that paths get exposed directly as user options. Simply that internally, we collect everything that is introspected from the interpreter in one place. Then, flags for things like --platform can modify that internal structure. There won't be flags to alter the paths, but existing means of configuring paths, like --target or --user, can be modified to simply change the "interpreter config" structure, rather than having effects across the code base. It's not a big deal, though.

I absolutely agree, we don't want user options to individually specify paths. That way lies chaos...

[pradyunsg]

Oh, that wasn't clear. 😅

We're in complete agreement here. :)

[sbidoul]

I'm copying @uranusjr's UX proposal from https://github.com/pypa/pip/issues/9981#issuecomment-843046920 here:

pip install --marker="os_name='nt'" --marker="platform_release='...'"

I assume that whenever one --marker is provided, none of the other markers usually provided by the platform will be made available to the resolver. I suppose it would also make sense to rely only on explicit markers whenever any of --platform, --implementation or --python-version is used. Installation or build would fail if any specifier uses a marker that is not explicitly provided by the user.

[ichard26]

If it's feasible (I haven't checked), it may be worth it to emit a warning when a marker is encountered when a package index filtering option is given.