Add `--ignore-dep` argument to `pip download` / improve recursive dependency resolution and download logic when `--platform` is specified?

What's the problem this feature will solve?

I would like to use pip download to download a package and all of it's dependencies to install them all later on an armv7l machine without internet access (but with ssh/scp access). On the x86_64 machine that is connected to the internet, I can run:

pip download --only-binary=:all: <pkg>

But that downloads also .whl files compiled for x86_64, which I can't send to the target armv7l machine. If I specify --platform=linux-armv7l, I am forced to either resolve the dependencies by myself by using --no-deps and iterating them one by one, or I need to use --only-binary=:all:. This logic is a bit peculiar. Why not download none-any.whl if available, and use source tarballs otherwise? Also, the word download doesn't suggest that it may try to compile / cross compile the package to the target --platform (related: https://github.com/pypa/pip/issues/12465), so downloading the source tarball and that's it is the natural thing to do IMO.

Anyway, as suggested by the CLI errors, I tried to run:

pip download --platform=linux-armv7l --only-binary=:all: <pkg>

But I got the error:

Collecting linien-server==2.0.4
  Using cached linien_server-2.0.4-py3-none-any.whl.metadata (1.1 kB)
Collecting cma<4.0,>=3.0.3 (from linien-server==2.0.4)
  Using cached cma-3.4.0-py3-none-any.whl.metadata (8.0 kB)
INFO: pip is looking at multiple versions of linien-server to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement fire>=0.6.0 (from linien-server) (from versions: none)
ERROR: No matching distribution found for fire>=0.6.0

Because the pure python dependency fire doesn't distribute a wheel. This situation forced me to use --no-deps instead of --only-binary=:all:, and resolve the dependency tree by myself - something that took me ~hour.

In addition to the above issue with --only-binary=:all: and --platform=linux-armv7l, I have another feature request related to the dependency resolution:

numpy and scipy, which are not pure Python transitive dependencies of my target <pkg>, are already installed on the target machine as a Debian packages (also because they don't provide wheels for this target architecture (Numpy issue). This is very helpful for me, but I currently don't have a way to tell pip download to ignore these dependencies. It would have been great if I could run something like this:

pip download --ignore-dep=scipy --ignore-dep=numpy <pkg>

And pip would automatically remove scipy and numpy from the recursive dependencies recursion.

Describe the solution you'd like

The --ignore-dep argument is described well enough above I hope.

The summary of the 1st solution proposal would be: Allow to specify --platform=... and don't require --only-binary=:all: in order to allow the dependency tree to be resolved. Prefer binary distribution if available, and reside to source tarballs if --only-binary is not specified.

Alternative Solutions

Use pip-compile from pip-tools and iterate the requirements by myself. Here's what worked for me:

python -m pip download \
  --platform=linux-armv7l \
  --no-deps \
  --requirement <(python -m piptools \
    compile \
    --output-file - 2>/dev/null | sed \
      -e 's/^\(numpy==1.26.4\|six==1.16.0\|scipy==1.14.0\)/#\0 is available on the target host by the Debian distribution/g' \
  )

Additional context

Haven't found very much related issues.

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

@doronbehar the problem of needing --only-binary=:all: is relatively well documented in these issues, but I think the maintainers did an exceptional job of describing it in the related issue I opened a couple years ago: https://github.com/pypa/pip/issues/11310. In short, there's no way to guarantee foreign interpreter compatibility with sdists.

The first issue you ran into is addressed by @pfmoore's suggestion in that issue (which was thankfully subsequently added to the pip download documentation). Simply download the sdist (for fire in this case), build the wheel using build -w or something similar, and then point to the directory containing that py3-none-any wheel using --find-links in the pip download call. These cases are us being punished by lazy developers, and if we're really nice we offer to publish a wheel for them or set up a CI/CD job to do so, but I do think the provided solution is acceptable for these cases.

However, I'm here because I'm running into a very similar issue with packages that cannot be compiled for a foreign interpreter. In my case it's gssapi and krb5 for pyspnego: they have to be compiled against the kerberos libs on the target machine. Your proposed --ignore-dep is an ideal solution for me in this case: I've identified the issue, and I can can handle the consequences of not having those included, but I really want pip download to be able to get my other ~30 requirements that are all available for my foreign interpreter rather than giving up completely and getting them all by hand.

Lastly, your pip-tools script alternative solution was very helpful, thank you! It got me doing what I needed today, but still hoping for a native solution as you stated.

pypa / pip