Enable pip-list to print packages in dependency order.

[kiranik]

What's the problem this feature will solve? User should be able to determine the order to install python packages so that the entire state can be reproduced with out issues from transitive dependencies?

Describe the solution you'd like pip list prints packages in alphabetical order. Instead the user may need to get the package list ordered corresponding to its dependencies

This is useful when we need to restore a virtual environment state with just the package list. If we use package list from default 'pip list', it will install different versions of package due to transitive dependencies.

Alternative Solutions pipdeptree displaying the installed python packages in form of a dependency tree. User can manually select the install order from this. But it needs time and effort and is often error prone.

Additional context printing packages in dependency order can fail sometime, in case of cyclic dependencies. In this case pip can print a commented message saying that it failed to find an install order.

[McSinyx]

Hi, this could be duplication of GH-8077. If so, you might want to join the discussion there.

[kiranik]

What I suggested was a feature which can print python package list in dependency order. For example, we can add an option like below to pip freeze.

pip freeze --dep-order

This will print python packages in the order of dependency, so that we can install each package with out any transitive dependency issues. For example: The above command can print lines like shown below(In dependency order).

notebook==6.0.3
nbconvert==5.6.1
ipykernel==5.2.1
nbformat==5.0.6
jupyter-client==6.1.3
jupyter-core==4.5.0
jsonschema==3.2.0
ipython==7.14.0
bleach==3.1.5
traitlets==4.3.3
terminado==0.8.3
python-dateutil==2.8.1
pyrsistent==0.16.0
prompt-toolkit==3.0.5
pexpect==4.8.0
packaging==20.3
jinja2==2.11.2
jedi==0.17.0
importlib-metadata==1.6.0
zipp==3.1.0
webencodings==0.5.1
wcwidth==0.1.9
tornado==6.0.4
testpath==0.4.4
six==1.14.0
send2trash==1.5.0
pyzmq==19.0.1
pyparsing==2.4.7
pygments==2.6.1
ptyprocess==0.6.0
prometheus-client==0.7.1
pickleshare==0.7.5
parso==0.7.0
pandocfilters==1.4.2
mistune==0.8.4
markupsafe==1.1.1
ipython-genutils==0.2.0
entrypoints==0.3
defusedxml==0.6.0
decorator==4.4.2
backcall==0.1.0
attrs==19.3.0
appnope==0.1.0

Here, Notebook package comes before all of it's dependent packages so that a particular version of dependent package can be installed.

If we used pip freeze for checking packages, we can see output like this(In alphabetical order):

appnope==0.1.0
attrs==19.3.0
backcall==0.1.0
bleach==3.1.5
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
importlib-metadata==1.6.0
ipykernel==5.2.1
ipython==7.14.0
ipython-genutils==0.2.0
jedi==0.17.0
Jinja2==2.11.2
jsonschema==3.2.0
jupyter-client==6.1.3
jupyter-core==4.5.0
MarkupSafe==1.1.1
mistune==0.8.4
nbconvert==5.6.1
nbformat==5.0.6
notebook==6.0.2
packaging==20.3
pandocfilters==1.4.2
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==3.0.5
ptyprocess==0.6.0
Pygments==2.6.1
pyparsing==2.4.7
pyrsistent==0.16.0
python-dateutil==2.8.1
pyzmq==19.0.1
Send2Trash==1.5.0
six==1.14.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.4
traitlets==4.3.3
wcwidth==0.1.9
webencodings==0.5.1
zipp==3.1.0

If we restore a virtual environment with the above set of packages, using these two approaches, then we can see that the list of package getting installed is different for both the approaches. Dependency ordered list restores the virtual environment completely using the versions from original virtual environment. But the alpahabetically ordered package list restore virtual environment with different set of package versions.

For example: In the above case, If we restore using dependency order, then we get exact same version of packages. But with alphabetically ordered package list, we will get a different package list as shown below.

appnope==0.1.0
attrs==19.3.0
backcall==0.1.0
bleach==3.1.5
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
importlib-metadata==1.6.0
ipykernel==5.2.1
ipython==7.14.0
ipython-genutils==0.2.0
jedi==0.17.0
Jinja2==2.11.2
jsonschema==3.2.0
jupyter-client==6.1.3
jupyter-core==4.6.3
MarkupSafe==1.1.1
mistune==0.8.4
nbconvert==5.6.1
nbformat==5.0.6
notebook==6.0.3
packaging==20.3
pandocfilters==1.4.2
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==3.0.5
ptyprocess==0.6.0
Pygments==2.6.1
pyparsing==2.4.7
pyrsistent==0.16.0
python-dateutil==2.8.1
pyzmq==19.0.1
Send2Trash==1.5.0
six==1.14.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.4
traitlets==4.3.3
wcwidth==0.1.9
webencodings==0.5.1
zipp==3.1.0

Check the version of jupyter-core(V4.6.3) in this environment whereas it's version was 4.5.0 in the original environment.

This type of issues occurs when the user uses a package list which is ordered alpahbetically and contains some dependencies. It can be fixed by installing packages in dependencies order.

[McSinyx]

What should happens to circular dependencies then? I guess you'd need to wait for a maintainer to answer that because I don't know the new resolver enough (I thing the behavior if the new one should be the same regardless of the order of packages you feed to it). In case of the older (non backtracking) resolver, I am not sure if it will get much work on. AFAIK the legacy resolver just pick the first compatible version, hence the unproducible list you commented above. I am not entirely sure if the new resolver will fix your case but I guess it doesn't hurt to give it a try: see GH-8099 for more details.

[kiranik]

Hi, In the case of circular dependencies, we can print the circular dependency info as well as error details. This will help user to decide the versions of one or more packages(In case if more packages are involved in loop) in order to remove circular dependency.

[uranusjr]

I want to add that Python packaging allows circular dependencies. They should not be treated as errors. pip should not suggest removing them either.

[kiranik]

Hi, In case of circular dependencies, we should print the circular dependency information. Otherwise a proper dependency order can be printed.

[McSinyx]

@kiranik, I've just tried out the dependencily ordered list from a fresh venv and seems your problem has more to do with the incompetence of the legacy resolver than the output of pip list:

$ pip install --unstable-feature=resolver -r ordered
Collecting notebook==6.0.3
  Downloading notebook-6.0.3.tar.gz (14.0 MB)
     |████████████████████████████████| 14.0 MB 235 kB/s 
ERROR: Could not find a version that satisfies the requirement jupyter-core==4.5.0
ERROR: Could not find a version that satisfies the requirement jupyter_core>=4.6.1 (from notebook)
ERROR: No matching distribution found for jupyter-core, jupyter-core