New feature suggested in "pip list": option to differentiate packages installed by pip and conda.

[vx1920]

I am using Anaconda, some packages installed by conda, but some other avalable only from "pip install". I have problem with packages updates, since Anaconda packages should be updated with conda and pip packages should be updated respectively with pip (normally).

Updating of conda packages can be done properly using "conda update --all", but "pip list outdated" reports all outdated packages, including just updated conda ones. Problem is that some packages can be installed in both ways and ones from pip are oftenly newer, since Anaconda doing prpagation with some delay. Normally it is better to use slightly older conda-installed packages with conda, but pip-only packages have to be updated and I want to see who did install package originally. There is optional column "Installer" in "pip list format=columns -verbose" with pip/conda values inside, but experiments shows this information is useless for me. It may just indicate that package uses metadata in "*dist-info" folder format and it is true for many conda-installed packages.

Solution: I need additional column with additional information. My new suggested output format can be enabled using new option "format=columnsx". Pay attention to new column "Owner" with "Anaconda"/"ThePip" values. You can also see that column "Installer" is useless, while new "Owner" is more useful...

``` Act: d:\InstSoft\Python\Conda3x64\Scripts\activate.bat Run: d:\InstSoft\Python\Conda3x64\Python.exe Arg: -m pip list --format=columnsx Package Version Installer Owner ------------------------------------ ------------------- --------- -------- adal 1.2.4 pip ThePip alabaster 0.7.12 pip Anaconda anaconda-client 1.7.2 Anaconda anaconda-navigator 1.9.12 Anaconda anaconda-project 0.8.3 conda Anaconda ansiwrap 0.8.4 pip ThePip appdirs 1.4.3 Anaconda applicationinsights 0.11.9 pip ThePip argh 0.26.2 Anaconda asn1crypto 1.3.0 pip Anaconda astroid 2.4.1 pip Anaconda astropy 4.0.1.post1 Anaconda async-generator 1.10 pip ThePip atomicwrites 1.4.0 conda Anaconda attrs 19.3.0 conda Anaconda augment-nd 0.1.1 pip ThePip autopep8 1.4.4 pip Anaconda azure-common 1.1.25 pip ThePip azure-core 1.6.0 pip ThePip azure-graphrbac 0.61.1 pip ThePip azure-identity 1.3.1 pip ThePip azure-mgmt-authorization 0.60.0 pip ThePip azure-mgmt-containerregistry 2.8.0 pip ThePip azure-mgmt-keyvault 2.2.0 pip ThePip azure-mgmt-network 10.2.0 pip ThePip azure-mgmt-resource 10.0.0 pip ThePip azure-mgmt-storage 10.0.0 pip ThePip azureml 0.2.7 pip ThePip azureml-automl-core 1.6.0 pip ThePip azureml-automl-runtime 1.6.0.post1 pip ThePip azureml-contrib-notebook 1.6.0 pip ThePip azureml-core 1.6.0.post1 pip ThePip azureml-dataprep 1.7.0 pip ThePip azureml-dataprep-native 14.2.0 pip ThePip azureml-defaults 1.6.0 pip ThePip azureml-explain-model 1.6.0 pip ThePip azureml-interpret 1.6.0 pip ThePip azureml-model-management-sdk 1.0.1b6.post1 pip ThePip azureml-pipeline 1.6.0 pip ThePip azureml-pipeline-core 1.6.0 pip ThePip azureml-pipeline-steps 1.6.0 pip ThePip azureml-sdk 1.6.0 pip ThePip azureml-telemetry 1.6.0 pip ThePip azureml-train 1.6.0 pip ThePip azureml-train-automl 1.6.0 pip ThePip azureml-train-automl-client 1.6.0.post1 pip ThePip azureml-train-automl-runtime 1.6.0 pip ThePip azureml-train-core 1.6.0.post1 pip ThePip azureml-train-restclients-hyperdrive 1.6.0 pip ThePip azureml-widgets 1.6.0 pip ThePip Babel 2.8.0 conda Anaconda backcall 0.1.0 pip Anaconda backports.functools-lru-cache 1.6.1 conda Anaconda backports.shutil-get-terminal-size 1.0.0 pip Anaconda backports.tempfile 1.0 pip Anaconda backports.weakref 1.0.post1 Anaconda bcrypt 3.1.7 conda Anaconda beautifulsoup4 4.9.1 pip Anaconda bitarray 1.2.1 pip Anaconda bkcharts 0.2 Anaconda black 19.10b0 conda Anaconda bleach 3.1.4 conda Anaconda blinker 1.4 Anaconda bokeh 2.0.2 pip Anaconda boto 2.49.0 Anaconda boto3 1.9.66 Anaconda botocore 1.12.189 conda Anaconda Bottleneck 1.3.2 pip Anaconda cachetools 3.1.1 conda Anaconda certifi 2020.4.5.1 Anaconda cffi 1.14.0 pip Anaconda chardet 3.0.4 conda Anaconda click 7.1.2 conda Anaconda cloudpickle 1.4.1 conda Anaconda clyent 1.2.2 Anaconda colorama 0.4.3 conda Anaconda comtypes 1.1.7 Anaconda conda 4.8.3 Anaconda conda-build 3.18.11 Anaconda conda-package-handling 1.7.0 conda Anaconda conda-verify 3.4.2 conda Anaconda configparser 5.0.0 conda Anaconda contextlib2 0.6.0.post1 conda Anaconda coverage 5.0 Anaconda cryptography 2.9.2 pip Anaconda cycler 0.10.0 Anaconda Cython 0.29.17 pip Anaconda cytoolz 0.10.1 Anaconda dask 2.17.2 conda Anaconda decorator 4.4.2 conda Anaconda defusedxml 0.6.0 Anaconda diff-match-patch 20181111 conda Anaconda dill 0.3.1.1 conda Anaconda distributed 2.17.0 pip Anaconda distro 1.5.0 pip ThePip docker 4.2.1 pip ThePip docutils 0.16 pip Anaconda dotnetcore2 2.1.14 pip ThePip entrypoints 0.3 Anaconda ephem 3.7.7.1 pip ThePip et-xmlfile 1.0.1 Anaconda fastcache 1.1.0 pip Anaconda ffmpeg 1.4 pip ThePip filelock 3.0.12 pip Anaconda fire 0.3.1 pip ThePip flake8 3.7.9 conda Anaconda Flask 1.1.2 pip Anaconda fsspec 0.7.4 conda Anaconda fusepy 3.0.1 pip ThePip future 0.18.2 conda Anaconda gensim 3.8.0 pip Anaconda ```

This new option is most useful with "pip list --outdated". In the example below it shows: all outdated items are from conda and pip update is not needed.

``` Act: d:\InstSoft\Python\Conda3x64\condabin\conda.bat activate Run: d:\InstSoft\Python\Conda3x64\Python.exe Arg: -m pip list --outdated --format=columnsx Package Version Latest Type Installer Owner ------------------------ ----------------- ---------- ----- --------- -------- appdirs 1.4.3 1.4.4 wheel Anaconda astroid 2.4.1 2.4.2 wheel pip Anaconda autopep8 1.4.4 1.5.3 sdist pip Anaconda bitarray 1.2.1 1.2.2 sdist pip Anaconda bleach 3.1.4 3.1.5 wheel conda Anaconda boto3 1.9.66 1.13.24 wheel Anaconda botocore 1.12.189 1.16.24 wheel conda Anaconda cachetools 3.1.1 4.1.0 wheel conda Anaconda certifi 2020.4.5.1 2020.4.5.2 wheel Anaconda coverage 5.0 5.1 wheel Anaconda Cython 0.29.17 0.29.19 wheel pip Anaconda dask 2.17.2 2.18.0 wheel conda Anaconda distributed 2.17.0 2.18.0 wheel pip Anaconda flake8 3.7.9 3.8.2 wheel conda Anaconda gensim 3.8.0 3.8.3 wheel pip Anaconda gevent 1.4.0 20.6.0 wheel Anaconda google-api-core 1.17.0 1.19.1 wheel pip Anaconda google-auth 1.14.1 1.16.1 wheel conda Anaconda google-cloud-storage 1.28.0 1.28.1 wheel conda Anaconda google-resumable-media 0.5.0 0.5.1 wheel conda Anaconda googleapis-common-protos 1.51.0 1.52.0 wheel pip Anaconda greenlet 0.4.15 0.4.16 wheel pip Anaconda hypothesis 5.11.0 5.16.0 wheel conda Anaconda importlib-metadata 1.6.0 1.6.1 wheel pip Anaconda ipykernel 5.1.4 5.3.0 wheel pip Anaconda ipython 7.13.0 7.15.0 wheel pip Anaconda jedi 0.15.2 0.17.0 wheel pip Anaconda jmespath 0.9.4 0.10.0 wheel pip Anaconda json5 0.9.4 0.9.5 wheel conda Anaconda jupyterlab 1.2.6 2.1.4 wheel Anaconda jupyterlab-server 1.1.4 1.1.5 wheel conda Anaconda keyring 21.1.1 21.2.1 wheel pip Anaconda lazy-object-proxy 1.4.3 1.5.0 wheel conda Anaconda lightgbm 2.3.0 2.3.1 wheel pip Anaconda lxml 4.5.0 4.5.1 wheel pip Anaconda matplotlib 3.1.3 3.2.1 wheel pip Anaconda nltk 3.4.5 3.5 sdist pip Anaconda numpy 1.18.1 1.18.5 wheel Anaconda numpydoc 0.9.2 1.0.0 wheel conda Anaconda packaging 20.3 20.4 wheel conda Anaconda pandas 1.0.3 1.0.4 wheel pip Anaconda parso 0.5.2 0.7.0 wheel conda Anaconda path 13.1.0 14.0.1 wheel pip Anaconda pathspec 0.7.0 0.8.0 wheel conda Anaconda pip 20.0.2 20.1.1 wheel Anaconda portalocker 1.5.2 1.7.0 wheel pip Anaconda prometheus-client 0.7.1 0.8.0 wheel conda Anaconda protobuf 3.11.4 3.12.2 wheel Anaconda pyarrow 0.11.1 0.17.1 wheel Anaconda pyasn1-modules 0.2.7 0.2.8 wheel Anaconda pycodestyle 2.5.0 2.6.0 wheel Anaconda pydocstyle 4.0.1 5.0.2 wheel conda Anaconda pyflakes 2.1.1 2.2.0 wheel pip Anaconda pylint 2.5.2 2.5.3 wheel pip Anaconda PyNaCl 1.3.0 1.4.0 wheel Anaconda pyodbc 4.0.0-unsupported 4.0.30 wheel pip Anaconda pytest 5.4.2 5.4.3 wheel Anaconda pytest-cov 2.8.1 2.9.0 wheel conda Anaconda pytest-doctestplus 0.5.0 0.7.0 wheel conda Anaconda python-language-server 0.31.10 0.33.1 wheel conda Anaconda pyzmq 18.1.1 19.0.1 wheel Anaconda QtAwesome 0.7.0 0.7.2 wheel conda Anaconda regex 2020.5.14 2020.6.8 wheel pip Anaconda ruamel-yaml 0.15.87 0.16.10 wheel Anaconda s3transfer 0.1.13 0.3.3 wheel Anaconda scikit-image 0.16.2 0.17.2 wheel pip Anaconda scikit-learn 0.22.1 0.23.1 wheel pip Anaconda sortedcollections 1.1.2 1.2.1 wheel pip Anaconda sortedcontainers 2.1.0 2.2.2 wheel Anaconda sphinxcontrib-websupport 1.2.1 1.2.2 wheel conda Anaconda sympy 1.5.1 1.6 wheel pip Anaconda tenacity 6.0.0 6.2.0 wheel pip Anaconda tifffile 0.15.1 2020.6.3 wheel Anaconda toml 0.10.0 0.10.1 wheel pip Anaconda tqdm 4.46.0 4.46.1 wheel conda Anaconda typing-extensions 3.7.4.1 3.7.4.2 wheel pip Anaconda ujson 1.35 3.0.0 wheel Anaconda urllib3 1.25.8 1.25.9 wheel pip Anaconda wcwidth 0.1.9 0.2.4 wheel conda Anaconda wrapt 1.11.2 1.12.1 sdist pip Anaconda XlsxWriter 1.2.8 1.2.9 wheel conda Anaconda yapf 0.28.0 0.30.0 wheel conda Anaconda zope.interface 4.7.1 5.1.0 wheel pip Anaconda ```

I have Win-X64, conda version : 4.8.3, conda-build version : 3.18.11, python version : 3.7.7.final.0. I did modify my local pip 20.0.2, everything works good for me, see above. We can discuss these changes more if it will be considered as new useful feature for "pip list". Old behavior completely unchanged, new feature available only with new option.

[uranusjr]

pip already shows the installer information in the verbose output (pip list -v). It should satisfy your need if Conda provides that information (and you’ll need to ask Conda to provide the info if it doesn’t already do).

[vx1920]

I already wrote that "pip list -v" provide additional column "Installer" with irrelevant information. If you don't believe - open first Details marker in my original post and review first two lines. In my system I have package "adal" installed by pip, while next package "alabaster" installed by conda. Your "pip list -v" marks both of them as "pip". With my new column I can see "alabaster" marked as "Anaconda", while "adal" marked "ThePip". I decide to combine both columns (old "Installer" and my new "Owner") to illustrate that "Installer" column contains something useless.

[uranusjr]

It’s a conda bug if packages it installs do not report the installer correctly. I don’t think pip should be responsible for fixing things other tools do wrong.

[vx1920]

I am not sure that it is conda bug, but I am sure that conda developers will tell you and me that there is no any bug in conda and conda itself can provide ability to update conda-only packages ("conda update --all"). Also "installer=pip" may legitimately indicate that conda uses "setuptools" (it is general tool used both by pip and conda) to install some particular package. Some other conda tools may set marker as "conda". Conda has tracking items installed by conda and pip should do the same. If I use "pip list --outdated" - it should inform about outdated packages installed by pip and don't care about items installed by conda. My option solves this problem somehow by adding extra marker column. Mybe it is better to have special option "pip list --outdatedpip", but with my option I could do some system cleanup by uninstalling pip packages and installing same name conda ones. Maybe somebody else you can consider my option as useful. Any case: I have it now, I use now and I solve some of my problems.

[pfmoore]

I concur with @uranusjr - pip reports the value in the INSTALLER metadata file. If conda populates that fire correctly, pip will report as you wish. If conda don't do so, they are not respecting the existing interoperability standard, and it's not down to pip to work around that problem.

[redbar0n]

I've encountered the problem where pip list will report the newer version of a package installed by conda (and vice versa, see: https://github.com/conda/conda/issues/9924). When I removed the newer package from conda then pip list will change to suddenly show the older version of that package. Presumably the one that was installed with pip in the first place.

I don't know where in the interaction between pip and conda that the problem lies. But in any case it seems inconsistent of the output of pip list to rely on activities conducted within conda.

[pfmoore]

@redbar0n As has been noted above, the fix here is for conda to follow the existing standards for recording installed packages. If they can't do that (they have their own processes, which may or may not work well alongside the packaging standards) then pip and conda will never be able to interoperate completely seamlessly.

As an aside, conda developers are more than welcome to participate in packaging standards discussions, in the same way that, for example, Linux distro packagers do. So far, we've had very little involvement from them, so it's unfortunate but not exactly unexpected that awkward interactions like this crop up occasionally.