General support for target installation paths

[dazza-codes]

py_version=$(python --version | grep -o -E '[0-9]+[.][0-9]+')
package_dir=$(mktemp -d -t tmp_python_XXXXXX)
package_dst=${package_dir}/python/lib/python${py_version}/site-packages
mkdir -p "$package_dst"

python -m pip install -t "${package_dst}" six

# the following cannot be done with pip 20.x
python -m pip show -t "${package_dst}" six
python -m pip uninstall -t "${package_dst}" -y six
python -m pip check -t "${package_dst}"

# the list command has an option named '--path' instead of '-t' or '--target';
# a unified set of options for this API could be created for target paths?
python -m pip list --path "$package_dst"

The value of full support for a target installation path is that it only contains added dependencies and it would be useful to be able to pip show, pip uninstall and pip check the packages in the target path without touching anything else. Even a clean-virtualenv contains a lot of extra packaging cruft, making it difficult to extract only added dependencies; one use-case for this is to build AWS lambda layers with only the required packages.

[uranusjr]

There are many design considerations needed for this. How does pip handle packages in sys.path? If it reads them, what happens if there are packages of different versions in sys.path and the target directory? What is the precedence? install -t currently skips all these by implying --ignore-installed, which is already not optimal in many cases, and wouldn’t work at all if we expand it to other subcommands. I’m not saying it’s impossible, but that you’ll need a much more fleshed-out proposal for this, and be willing to take lead on the (likely very long) discussion and implementation work to the end.

[dazza-codes]

I'm not in a position to execute on this feature request, sorry. Nevertheless, it stands as is.

Another possibility to address the use-case for creating zip-files from requirements could be some form of package archive functionality that takes a requirements.txt file and only archives the dependency tree involved. The idea is to use a virtualenv for all package operations and only export an archive from the virtualenv that only contains the dependency tree actually required (e.g. unless pip is in the requirements.txt it would not be exported from the virtualenv into an archive file). In effect, it could use some form of tracking all the content required for a default clean-venv to use some file set operations to exclude that content from the package archives defined by the dependency tree in the requirements.txt file. The pip show --files seems to have the file tracking functionality required to iterate on all the packages in the required dependency tree so they could be archived. Hypothetically, this could create an archive package of dependencies that excludes development dependencies.

For purposes of issue-management, I'm happy to get feedback on whether this comment is a duplicate and/or needs to be extracted to a new issue/feature-request.

[pfmoore]

I'd personally like to improve pip's support for install locations (so things like --target aren't quite such "second class citizens") but I don't expect it to give full generality like you're suggesting. I'm also only really interested in "single target" situations, so layering target on top of sys.path or similar is very much outside of what I'm interested in (as @uranusjr says, covering that is when things start to get really complicated). And it's a lot harder to implement than you seem to be assuming, so it's not something that's going to happen soon (particularly given the number of people requesting additional features when compared to the number of people with the time or interest in implementing them...)