Add parsable output option to `pip download` (JSON?)

[cooperlees]

How do people feel about adding easily parsable output to pip download output so that one can easily learn all the dependencies for PyPI packages from other programs.

e.g. From @philipjameson's proto type: I download django and pytz-convert and the output would look like: pip download -d tmp django pytz-convert --json --log-stderr | jq https://gist.github.com/philipjameson/9d63842cc4865e659d5df21aeca7451e

Thoughts?

[pradyunsg]

Relevant here: https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies

[cooperlees]

@pradyunsg - Thanks for the link, I've read this post before and understand the limitations. But pip download is as close as we get, today. I currently use pip wheel to build module wheels in my build environment and it's pretty accurate. When it's not, I do the PR to help make it so (i.e. adding the missing dependencies to install_requires).

Why I think this is useful in the short term, is that this allows a way to have external processes get a somewhat accurate view of the dependencies of a package. As I mirror PyPI locally via bandersnatch, I do not impose a burden on the main Infra and it's local and fast. I have an internal system @ Facebook that builds via pip wheel, and, at the moment I always have to rebuild (unnecessarily) all dependencies for the package, even if we already have the wheel up on our internal mirror.

Happy to chat in person if you're currently at PyCon - hit me up on Twitter or Linked In (both @cooperlees).

[pradyunsg]

Happy to chat in person if you're currently at PyCon

Nope. :/

at the moment I always have to rebuild (unnecessarily) all dependencies for the package

Does pip wheel --no-deps flask fit your workflow?

[benoit-pierre]

You could use pip wheel -f wheels_dir -w wheels_dir to keep/update a cache of the wheels you need.

[cooperlees]

Exactly. The --no-deps is what I want to do. But first I need to use download (or something else I don't know about) to learn (as best I can) all the dependencies to check if they exist in my internal mirror. Then I will pip wheel --no-deps PKG1 PKG2 ....

E.g.

pip download flask
...
Successfully downloaded flask click Werkzeug Jinja2 itsdangerous MarkupSafe

Adding --json would allow me to consume those package names and versions nicely and then check my metadata (in my case, in a MySQL DB) and then only build what I need, lets say that is:

pip wheel --no-deps flask Werkzeug==0.6.9 itsdangerous==1.4.2

Then I only get the wheels I need to build, and it is really nice for 'fun' dependency heavy packages like numpy, scipy and all their many packages that consume them.

[pradyunsg]

--report in 22.2 covers this: https://pip.pypa.io/en/stable/reference/installation-report/