Other `pipx list` output formats

[itsayellow]

How would this feature be useful?

Hopefully this will synthesize (and possibly supersede) Issues #390 and #109. It addresses the reason for PRs #572 and #392.

This Issue is to track exactly what users are looking for with different pipx list output formats.

"Import / Export" set of pipx packages case

In #109, it appears that one primary motivator is the ability to export a file that specifies a set of pipx-installable packages that pipx can then install (possibly on another machine) in the future.

How many users desire this or would find it useful?

It would probably best be done by making a file in json format.

Within this there are some further points for those desiring this functionality:

Do people consider it desired (or necessary?) to freeze all versions of all dependencies for this to be useful? If so then this might be very difficult, based on my investigation creating test code for this.

Simply re-creating a pipx environment, and pinning the version of the main package and any injected packages, and any pipx options is very doable. I have a branch of doing this basically working. In this case, the output would be equivalent to what you would end up with on one machine executing pipx reinstall-all. You can be sure the main package and any injected packages will stay with their specified versions, but you cannot be sure that the versions of any dependencies that still satisfy all package requirements will be the same.

Archive / Documentation case

In #390 it appears that one motivation is simply a concise version of pipx list for "keeping track" or for archival or documentation purposes.

How many users desire this or would find it useful?

Is this type of list output (that is merely more concise) still desired if the "Import / Export" case is available?

What are the goals of this format? What are the use cases specifically? How would one know if the output of this was "successful"?

I must confess that I need this one explained to me, it doesn't naturally occur to me what is desired. If all that's needed is documentation of what is currently installed, then the current pipx list seems to cover that. If what is desired is a structured output that would allow one to recreate a pipx environment, then the "Import / Export" json scenario seems to satisfy the need.

@comkieffer @gvoysey @sanketdg @zachvalenta

Describe the solution you'd like

Describe alternatives you've considered

[zachvalenta]

For #390, my idea is:

# freeze list of pkg
pipx list > $DOTFILES_DIR/pkg-pipx.txt

# parses list of pkg and installs each
cat pkg-pipx.txt | pipx_install.sh

Right now this second step (piping pkg list file to install script) is fine but would just be easier with more structured output format. If output format that could be directly be read by pipx itself to install, all the better.

Also, thanks for putting all this together :)

[itsayellow]

For #390, my idea is:

# freeze list of pkg
pipx list > $DOTFILES_DIR/pkg-pipx.txt

# parses list of pkg and installs each
cat pkg-pipx.txt | pipx_install.sh

Right now this second step (piping pkg list file to install script) is fine but would just be easier with more structured output format. If output format that could be directly be read by pipx itself to install, all the better.

@zachvalenta do I understand you correctly that if you had a structured json output, you wouldn't need another kind of pipx list output format? (Your personal opinion)

[zachvalenta]

@zachvalenta do I understand you correctly that if you had a structured json output, you wouldn't need another kind of pipx list output format? (Your personal opinion)

Yes indeed. I'm looking for something more structured than the status quo but agnostic about format.

[gvoysey]

Thanks for pulling together a bunch of threads!

Simply re-creating a pipx environment, and pinning the version of the main package and any injected packages, and any pipx options is very doable.

This is all i want, anyway (and i filed #109 ).

An ancillary use case that I am now looking forward to is bumping python interpreter versions for my pipx-managed venvs on the same machine. I have many 3.6.8 venvs, and wouldn't mind kicking them all up to 3.8.x or higher without manually redoing all of it.

[gvoysey]

@itsayellow It occurs to me that if you do not already know about it, https://pypi.org/project/pipdeptree/ might be extremely useful for inspecting pipx-created environments and generating a list of top-level packages that were injected into them.

For a busy example: i have a very full ipython environment managed by pipx right now. If i pipx inject ipython pipdeptree and then manually run .local/pipx/venvs/ipython/bin/pipdeptree, i get a tree whose roots are the things i injected:

cattrs==1.2.0
  - attrs [required: >=20.1.0, installed: 20.3.0]
dataset==1.4.5
  - alembic [required: >=0.6.2, installed: 1.5.5]
    - Mako [required: Any, installed: 1.1.4]
      - MarkupSafe [required: >=0.9.2, installed: 1.1.1]
    - python-dateutil [required: Any, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - python-editor [required: >=0.3, installed: 1.0.4]
    - SQLAlchemy [required: >=1.3.0, installed: 1.3.23]
  - banal [required: >=1.0.1, installed: 1.0.6]
  - sqlalchemy [required: >=1.3.2, installed: 1.3.23]
desert==2020.11.18
  - attrs [required: Any, installed: 20.3.0]
  - marshmallow [required: >=3.0, installed: 3.10.0]
  - typing-inspect [required: Any, installed: 0.6.0]
    - mypy-extensions [required: >=0.3.0, installed: 0.4.3]
    - typing-extensions [required: >=3.7.4, installed: 3.7.4.3]
glom==20.11.0
  - attrs [required: Any, installed: 20.3.0]
  - boltons [required: >=19.3.0, installed: 20.2.1]
  - face [required: >=20.1.0, installed: 20.1.1]
    - boltons [required: >=20.0.0, installed: 20.2.1]
httpx==0.16.1
  - certifi [required: Any, installed: 2020.12.5]
  - httpcore [required: ==0.12.*, installed: 0.12.3]
    - h11 [required: ==0.*, installed: 0.12.0]
    - sniffio [required: ==1.*, installed: 1.2.0]
  - rfc3986 [required: >=1.3,<2, installed: 1.4.0]
  - sniffio [required: Any, installed: 1.2.0]
hyperlink==21.0.0
  - idna [required: >=2.5, installed: 3.1]
imageio==2.9.0
  - numpy [required: Any, installed: 1.20.1]
  - pillow [required: Any, installed: 8.1.0]
ipython==7.20.0
  - backcall [required: Any, installed: 0.2.0]
  - decorator [required: Any, installed: 4.4.2]
  - jedi [required: >=0.16, installed: 0.18.0]
    - parso [required: >=0.8.0,<0.9.0, installed: 0.8.1]
  - pexpect [required: >4.3, installed: 4.8.0]
    - ptyprocess [required: >=0.5, installed: 0.7.0]
  - pickleshare [required: Any, installed: 0.7.5]
  - prompt-toolkit [required: >=2.0.0,<3.1.0,!=3.0.1,!=3.0.0, installed: 3.0.16]
    - wcwidth [required: Any, installed: 0.2.5]
  - pygments [required: Any, installed: 2.8.0]
  - setuptools [required: >=18.5, installed: 53.0.0]
  - traitlets [required: >=4.2, installed: 5.0.5]
    - ipython-genutils [required: Any, installed: 0.2.0]
lxml==4.6.2
Pint==0.16.1
  - packaging [required: Any, installed: 20.9]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
pipdeptree==2.0.0
  - pip [required: >=6.0.0, installed: 21.0.1]
plotnine==0.7.1
  - descartes [required: >=1.1.0, installed: 1.1.0]
    - matplotlib [required: Any, installed: 3.3.4]
      - cycler [required: >=0.10, installed: 0.10.0]
        - six [required: Any, installed: 1.15.0]
      - kiwisolver [required: >=1.0.1, installed: 1.3.1]
      - numpy [required: >=1.15, installed: 1.20.1]
      - pillow [required: >=6.2.0, installed: 8.1.0]
      - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
      - python-dateutil [required: >=2.1, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
  - matplotlib [required: >=3.1.1, installed: 3.3.4]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.20.1]
    - pillow [required: >=6.2.0, installed: 8.1.0]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - mizani [required: >=0.7.1, installed: 0.7.2]
    - matplotlib [required: >=3.1.1, installed: 3.3.4]
      - cycler [required: >=0.10, installed: 0.10.0]
        - six [required: Any, installed: 1.15.0]
      - kiwisolver [required: >=1.0.1, installed: 1.3.1]
      - numpy [required: >=1.15, installed: 1.20.1]
      - pillow [required: >=6.2.0, installed: 8.1.0]
      - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
      - python-dateutil [required: >=2.1, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
    - numpy [required: Any, installed: 1.20.1]
    - palettable [required: Any, installed: 3.3.0]
    - pandas [required: >=1.1.0, installed: 1.2.2]
      - numpy [required: >=1.16.5, installed: 1.20.1]
      - python-dateutil [required: >=2.7.3, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - pytz [required: >=2017.3, installed: 2021.1]
  - numpy [required: >=1.16.0, installed: 1.20.1]
  - pandas [required: >=1.1.0, installed: 1.2.2]
    - numpy [required: >=1.16.5, installed: 1.20.1]
    - python-dateutil [required: >=2.7.3, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.3, installed: 2021.1]
  - patsy [required: >=0.5.1, installed: 0.5.1]
    - numpy [required: >=1.4, installed: 1.20.1]
    - six [required: Any, installed: 1.15.0]
  - scipy [required: >=1.2.0, installed: 1.6.1]
    - numpy [required: >=1.16.5, installed: 1.20.1]
  - statsmodels [required: >=0.11.1, installed: 0.12.2]
    - numpy [required: >=1.15, installed: 1.20.1]
    - pandas [required: >=0.21, installed: 1.2.2]
      - numpy [required: >=1.16.5, installed: 1.20.1]
      - python-dateutil [required: >=2.7.3, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - pytz [required: >=2017.3, installed: 2021.1]
    - patsy [required: >=0.5, installed: 0.5.1]
      - numpy [required: >=1.4, installed: 1.20.1]
      - six [required: Any, installed: 1.15.0]
    - scipy [required: >=1.1, installed: 1.6.1]
      - numpy [required: >=1.16.5, installed: 1.20.1]
pugsql==0.2.3
  - sqlalchemy [required: >=1.3,<2.0, installed: 1.3.23]
pydantic==1.7.3
PyYAML==5.4.1
scikit-learn==0.24.1
  - joblib [required: >=0.11, installed: 1.0.1]
  - numpy [required: >=1.13.3, installed: 1.20.1]
  - scipy [required: >=0.19.1, installed: 1.6.1]
    - numpy [required: >=1.16.5, installed: 1.20.1]
  - threadpoolctl [required: >=2.0.0, installed: 2.1.0]
seaborn==0.11.1
  - matplotlib [required: >=2.2, installed: 3.3.4]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.20.1]
    - pillow [required: >=6.2.0, installed: 8.1.0]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - numpy [required: >=1.15, installed: 1.20.1]
  - pandas [required: >=0.23, installed: 1.2.2]
    - numpy [required: >=1.16.5, installed: 1.20.1]
    - python-dateutil [required: >=2.7.3, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.3, installed: 2021.1]
  - scipy [required: >=1.0, installed: 1.6.1]
    - numpy [required: >=1.16.5, installed: 1.20.1]
tabulate==0.8.9
typer==0.3.2
  - click [required: >=7.1.1,<7.2.0, installed: 7.1.2]
wheel==0.36.2
xlrd==2.0.1

and running pipdeptree -j spits out json that's more programmatically handy to attack with something like glom.

[itsayellow]

Thanks @gvoysey , I am aware of pipdeptree (and use it myself if I accidentally install something into my system python packages!) Luckily in pipx we have a bit of an easier time of it because of our isolated venvs, and our internal metadata. Our internal metadata keeps track of what we intentionally installed, and each venv can be pip listed to find any dependencies added.

[gvoysey]

@itsayellow oh. well, that's just /cheating/! :grin:

[itsayellow]

The problem seems to be in trying to implement something to install from such a json list. Doing a basic job is fairly straightforward, but covering all the edge cases gets a little hairy.

For example, how do you handle things in your json file that do not exist on the target install system? Like paths to specific local packages, or paths to specific python interpreters.

This might just mean that implementing a json format for list output is doable, but installing from such an output might be DIY.

[zachvalenta]

This might just mean that implementing a json format for list output is doable, but installing from such an output might be DIY.

This makes sense to me as a fair middle ground btw added convenience for users and not forcing pipx to deal with a hairy problem.

[gvoysey]

i noticed when using pipx reinstall-all that packages that i installed "stringly" to include extras failed and were tidly logged at the end. (e.g., "reinstalled ipython, black, failed to reinstall python-language-server[all]", because you have to pipx install 'python-language-server[all]').

Flow like that for injected libraries in a pipx install seems like a good compromise to me.

[itsayellow]

i noticed when using pipx reinstall-all that packages that i installed "stringly" to include extras failed and were tidly logged at the end. (e.g., "reinstalled ipython, black, failed to reinstall python-language-server[all]", because you have to pipx install 'python-language-server[all]').

Flow like that for injected libraries in a pipx install seems like a good compromise to me.

I don't quite parse what you mean by '"stringly" including extras', but I think I get the basic point, that we could just go through the list and try to install everything as described, and just fail on each particular package that doesn't work out, but otherwise keep installing.

I suppose specifying an invalid package or python path would just fail to install that one as if one were trying to install something improperly from the command line.

[itsayellow]

Here's a UX question. How would a command that outputs json be structured?

Possibilities that come to mind:

1. The user must pipe the output from pipx list to a file, e.g. pipx list --json > my_packages.pipx
2. Use pipx list but have an option to specify an output file, e.g. pipx list --json-file=my_packages.pipx
3. A separate command, e.g. pipx export-spec my_packages.pipx

I'm leaning toward something like 2 or 3, or in general specifying an output file. It seems to me that you'd really never want to stream the json text to the console anyway. And for a command that always wants an output file, it seems weird to me to make the user have to redirect by hand.

I suppose the reason to output the json in a stdout stream would be to pipe it to some other command. But would that be useful? Would that be useful if pipx had its own native command to import such json?

[zachvalenta]

The best reasons I can think of for streaming to stdout would be if list:

• took any alternative format (list --txt, list --yaml
• if list was able to show top-level keys only (maybe you have a lot installed and just want to get a high-level view; the Ansible CLI is a good example that adds a lot of extra output you might want to trim).

But specifying a file output makes sense to me as well.

[gwerbin]

I am much more in favor of streaming to stdout. The user can always redirect to a file if they want to. And you could also pipe the output to jq or grep or whatever else you might need.

pipx list --json \
| jq '.[].env_name' \
| grep 'mypy|pyre-check' \
| xargs -n1 pipx reinstall

Somewhat of a contrived example, but I can't see any reason to force the user to output to a file.

[gvoysey]

I would pretty naturally spell that as "a subcommand with an optional argument". so pipx list [-j,--json] does as @gwerbin proposes, and pipx list [-j,json] [outfile] stuffs it into a file of your choice.

[itsayellow]

I'm starting to think the clearest next step for a PR is to implement (as suggested above)

pipx list --json

Which just sends the json output to stdout (and only the json output to stdout).

I have more experience with macOS and linux--I assume redirecting to a file is just as easy on the various Windows shells?

[gwerbin]

I have more experience with macOS and linux--I assume redirecting to a file is just as easy on the various Windows shells?

The Windows command prompt and Powershell both support > and >> syntax for output redirection.

[ashwinvis]

JSON is fine, but another modern alternative could be TOML which would be a good balance between machine readable and human readable, without pitfalls of YAML. If one loads a JSON file as is and dump it as TOML, you get:

pipx_metadata_version = "0.2"
python_version = "Python 3.8.5"
venv_args = []

[injected_packages]

[main_package]
apps = ["xon.sh", "xonsh", "xonsh-cat"]
apps_of_dependencies = []
include_apps = true
include_dependencies = false
package = "xonsh"
package_or_url = "xonsh"
package_version = "0.9.27"
pip_args = []
suffix = ""
[[main_package.app_paths]]
__Path__ = "/home/user/.local/pipx/venvs/xonsh/bin/xon.sh"
__type__ = "Path"

[[main_package.app_paths]]
__Path__ = "/home/user/.local/pipx/venvs/xonsh/bin/xonsh"
__type__ = "Path"

[[main_package.app_paths]]
__Path__ = "/home/user/local/pipx/venvs/xonsh/bin/xonsh-cat"
__type__ = "Path"

[main_package.app_paths_of_dependencies]

I made the following sample TOML after manually editing pipx_metadata_version.json for a xonsh virtual environment.

[metadata]
version = 0.2

[package.xonsh]
spec = "xonsh"  # Note: instead of package_or_url
version = "0.9.27"
apps = ["xon.sh", "xonsh", "xonsh-cat"]
apps_of_dependencies = []
include_apps = true
include_dependencies = false
package = "xonsh"
pip_args = []
python_version = "Python 3.8.5"  # Note: moved down
suffix = ""
venv_args = []  # Note: moved down

[[package.xonsh.app_paths]]
"xon.sh" = "/home/user/.local/pipx/venvs/xonsh/bin/xon.sh"
"xonsh" = "/home/user/.local/pipx/venvs/xonsh/bin/xonsh"
"xonsh-cat" = "/home/user/.local/pipx/venvs/xonsh/bin/xonsh-cat"

[[package.xonsh.app_paths_of_dependencies]]

# continue with more packages ...

[gwerbin]

There's no TOML equivalent of JQ yet, and JSON generally has more support across programming languages. Therefore TOML would singificantly limit the usefulness of this option as something you can pipe into other scripts/tools for downstream processing.

Also JSON comes with the Python standard library, which keeps the dependency count down (although I expect that TOML will end up in the stdlib eventually because it's part of PEP 517/518).

That said, maybe Pipx could support multiple output formats?

pipx list --format=list  # default if --format is omitted
pipx list --format=json
pipx list --format=toml
pipx list --format=yaml

There are well-supported TOML and YAML libraries on PyPI to do the output formatting.

[ashwinvis]

I think it is best if this would be combined with an option like pipx reinstall-all --file pipx_packages.toml so that we do not rely on platform dependent shell commands to reinstall pipx packages..

Some pros and cons of different options

	pros	cons
list	human friendly	non-standardised, can break with whitespace changes, cannot represent hierarchial data
json	portable, machine-readable, in standard library	verbose, not human friendly
toml	machine-readable, support for basic types, allows comments, human friendly	new
yaml	machine-readable, support for basic types, allows comments, human friendly	can break with whitespace changes, has known security vulnerablities

[itsayellow]

What is the advantage of having a format that is both human friendly and machine readable?

I'm aware we could get toml to work, but I'm just not sure why we need it.

[gvoysey]

The argument for TOML usually arises in the context of config files, which this isn't.

I'm fine with just spitting out JSON, fwiw.

[ashwinvis]

config files, which this isn't

It is very similar to a requirements.txt, environment.yml, Config.toml, pyproject.toml etc - call it what you will :)

What is the advantage of having a format that is both human friendly and machine readable?

I understand the arguments made for JSON. Here are some reasons for the file to be human readable. In the event when:

• the package does not install (for. eg. change in git URL, due to upgrade of user's Python and wheels being unavailable etc)
• tracking the file in a dotfiles repository and handling merge conflicts

it would make it easier for the user to go in and edit the file. That's my take on why TOML would be a better fit.

P.S.: I forgot, you can comment out a package too.

[itsayellow]

The json I'm producing right now is pretty-printed. So editing a particular path by hand is actually quite reasonable.

Also, I don't think I'm eager to have this format be created by hand from scratch, like a config file. I'd like to keep it pipx-generated so that we don't have to support human quirks that might pop up when we parse it.

[zachvalenta]

Also, I don't think I'm eager to have this format be created by hand from scratch, like a config file.

For the folks who want a language associated w/ config files like TOML/YAML, you can generate those e.g. Poetry generates a pyproject.toml. IMO JSON is the best option out of the gate, just assuming that's what they're thinking of when suggesting TOML/YAML.

[itsayellow]

Not sure if the linked PR will trigger anyone's notifications so I'll post this: Check out PR #660 and tell me what you think. I made a short blurb describing the format.

[gvoysey]

:+1: from me; i think this is the output side of #109 and it looks pretty tidy!

[itsayellow]

I'm going to close this now because I believe #660 has implemented json list output format, and that's probably the only other format besides text that we want to implement right now.

Thanks everybody for the good discussion.