Reading compressed inventory from stdin results in decoding error

[pawamoy]

Brief description

Trying to display an inventory in the terminal, by curling it and piping it to sphobjinv convert:

curl https://pip.pypa.io/en/latest/objects.inv -o- | sphobjinv convert plain - -

Expected behavior sphobjinv should read stdin, try to decode it, but catch exceptions and consider it compressed.

Actual behavior

sphobjinv doesn't catch the UnicodeDecodeError.

To reproduce

% curl https://pip.pypa.io/en/latest/objects.inv -o- | sphobjinv convert plain - -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 
100  9485  100  9485    0     0  14280      0 --:--:-- --:--:-- --:--:-- 14306
Traceback (most recent call last):
  File "/home/pawamoy/.local/bin/sphobjinv", line 8, in <module>
    sys.exit(main())
  File "/home/pawamoy/.local/pipx/venvs/sphobjinv/lib/python3.10/site-packages/sphobjinv/cli/core.py", line 88, in main
    inv = inv_stdin(params)
  File "/home/pawamoy/.local/pipx/venvs/sphobjinv/lib/python3.10/site-packages/sphobjinv/cli/load.py", line 245, in inv_stdin
    data = sys.stdin.read()
  File "/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.10.8/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 119: invalid continuation byte

System information

• Device: Computer?
• OS: ArchLabs

Python environment

Python 3.11

Libraries

% pipx runpip sphobjinv freeze 
attrs==22.2.0
certifi==2022.12.7
jsonschema==4.17.3
pyrsistent==0.19.3
sphobjinv==2.3.1

Additional context

Passing a temporary file descriptor to the command does not work either:

% sphobjinv convert plain <(curl -s https://pip.pypa.io/en/latest/objects.inv -o-) -

Error while parsing input file path:
FileNotFoundError: Indicated path is not a valid file

[bskinn]

I looked into this a bit when I implemented the stdin stream capability, and something about running a compressed inventory through a pipe appears to mangle it---no matter what I tried, I couldn't get it to recognize correctly.

I'm guessing it has something to do with character encoding, as AFAIK the nix piping system expects to be passing plaintext, not bytestreams. There could* be an env variable out there you can set to change the encoding behavior (a "raw" pipe mode?), in which case it might be workable.

I didn't want to put the work in to figure that out until I was sure it wasn't YAGNI. Which--now, it's not. 😬

If you have any thoughts about things to try, please let me know!

[bskinn]

Although this makes me think I'm wrong about this, in which case I'm not sure what went wrong.

Maybe at the *nix/Python interface....

[pawamoy]

Oh, I see, thanks for the prompt reply!

It seems hard to get right indeed, so I'll just tell you that you can close this, because the workaround is so easy: just actually download the inv file locally, then run sphobjinv.

I think you also have tickets open for accepting URLs, that would solve the issue as well, and provide an even better UX (no pipe needed, more cross-platform, etc) :+1:

[bskinn]

Hm. The feature is already there for pulling inventories directly from URLs.

Did you try:

% sphobjinv convert plain -u https://pip.pypa.io/en/latest/objects.inv -

?

[pawamoy]

Ah, my bad, I glanced at the issues and the usage (-h) but didn't understand this was possible. Well that's exactly what I needed haha, thanks!

[bskinn]

Aha, I need to add to the base help message (sphobjinv -h) to indicate that there's additional help for each subcommand (sphobjinv convert -h and sphobjinv suggest -h).

[pawamoy]

It's already the case :smile:

% sphobjinv -h
usage: sphobjinv [-h] [-v] {convert,suggest} ...

Format conversion for and introspection of intersphinx 'objects.inv' files.

options:
  -h, --help         show this help message and exit
  -v, --version      Print package version & other info

Subcommands:
  {convert,suggest}  Execution mode. Type 'sphobjinv [mode] -h' for more information on available options. Mode names can be abbreviated to their first two letters.
    convert (co)     Convert intersphinx inventory to zlib-compressed, plaintext, or JSON formats.
    suggest (su)     Fuzzy-search intersphinx inventory for desired object(s).

I had just missed the -u option in the help for the convert subcommand :slightly_smiling_face:

[bskinn]

Oh hey! So it is. I scanned my own help message too quickly... 😅

[bskinn]

Though -- since you missed it, probably means I should promote that message about subcommand help to someplace more prominent in the overall help message.

[pawamoy]

I generally print something like this myself:

usage: griffe [GLOBAL_OPTS...] COMMAND [COMMAND_OPTS...]

Signatures for entire Python programs.

Global options:
  -h, --help  Show this help message and exit. Commands also accept the -h/--help option.

Commands:
  COMMAND
    dump      Load package-signatures and dump them as JSON.
    check     Check for API breakages or possible improvements.

Not sure if it's more visible :shrug:

[bskinn]

Cool - thanks for the suggestion!

[bskinn]

Closing since the original issue was resolved, and the help message change spun into #282.