Verbose output without final result when outputting to a file

[daveisfera]

What's the problem this feature will solve?

Having verbose output is nice so progress can be viewed, but when writing the result to a file, it's still output to the console

Describe the solution you'd like

Verbose output can be used without dumping all of the result to the console, when writing to a file

Alternative Solutions

Flag to turn on progress output

[webknjaz]

It's a decades-old POSIX convention to output the usable/structured/parseable output to stdout but direct the decorative information like logging to stderr, in CLI programs. This makes such apps chainable through piping. If you want to merge them merged, use stream redirection on the shell level.

[daveisfera]

I definitely get that and I'm using --output-file to write to a file at the end so there's not a period where there's an empty file. But aside from that, the verbose output goes to stdout so you can't redirect to a file with --verbose, because you'll end up with an invalid file that has all of the logging at the top, so is the verbose output going to be changed to stderr because that would be a reasonable solution as well.

[webknjaz]

I'm not sure I understand, then. Could you include a detailed illustrative example of what's happening vs. what you expect?

[daveisfera]

Assuming a super simple input of the following:

django~=3.2.0

I run with the following command:

pip-compile requirements.txt --verbose --output-file requirements.lock > stdout.txt 2> stderr.txt

The resulting lock file should be (and is) written to requirements.lock. Using the UNIX/POSIX convention you're mentioning, stdout.txt should have the same contents as requirements.lock and stderr.txt should have the extra output about the rounds of resolution that are enabled by --verbose, but all of the output is written to stdout.txt

Basically, the concept is that stdout is for the "actual output" and stderr is for extra output/logging/information, so the redirection you mentioned can be used as if it was coming from an output file. I think that's a great solution as well (because I could just redirect the "actual output" to /dev/null and get the desired result I was trying to describe originally)

[chrysle]

The resulting lock file should be (and is) written to requirements.lock.

Right.

Using the UNIX/POSIX convention you're mentioning, stdout.txt should have the same contents as requirements.lock and stderr.txt should have the extra output about the rounds of resolution that are enabled by --verbose, but all of the output is written to stdout.txt

I can't reproduce this. For me, all the output is written to stderr.txt.

$ pip-compile requirements.txt --verbose --output-file requirements.in > stdout.txt 2> stderr.txt
$ cat stdout.txt 
$ cat stderr.txt 
Using pip-tools configuration defaults found in 'pyproject.toml'.
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
Using indexes:
  https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  anyio==3.7.1 (from -r requirements.txt (line 7))
  certifi==2023.5.7 (from -r requirements.txt (line 9))
  exceptiongroup==1.1.2 (from -r requirements.txt (line 13))
  h11==0.14.0 (from -r requirements.txt (line 15))
  httpcore==0.17.3 (from -r requirements.txt (line 17))
  httpx==0.24.1 (from -r requirements.txt (line 19))
  idna==3.4 (from -r requirements.txt (line 21))
  sniffio==1.3.0 (from -r requirements.txt (line 25))

Finding the best candidates:
  found candidate anyio==3.7.1 (constraint was ==3.7.1)
  found candidate certifi==2023.5.7 (constraint was ==2023.5.7)
  found candidate exceptiongroup==1.1.2 (constraint was ==1.1.2)
  found candidate h11==0.14.0 (constraint was ==0.14.0)
  found candidate httpcore==0.17.3 (constraint was ==0.17.3)
  found candidate httpx==0.24.1 (constraint was ==0.24.1)
  found candidate idna==3.4 (constraint was ==3.4)
  found candidate sniffio==1.3.0 (constraint was ==1.3.0)

Finding secondary dependencies:
  httpx==0.24.1             requires certifi, httpcore<0.18.0,>=0.15.0, idna, sniffio
  anyio==3.7.1              requires exceptiongroup; python_version < "3.11", idna>=2.8, sniffio>=1.1
  sniffio==1.3.0            requires -
  httpcore==0.17.3          requires anyio<5.0,>=3.0, certifi, h11<0.15,>=0.13, sniffio==1.*
  idna==3.4                 requires -
  exceptiongroup==1.1.2     requires -
  certifi==2023.5.7         requires -
  h11==0.14.0               requires -

New dependencies found in this round:
  adding ('anyio', '<5.0,>=3.0', [])
  adding ('certifi', '', [])
  adding ('exceptiongroup', '', [])
  adding ('h11', '<0.15,>=0.13', [])
  adding ('httpcore', '<0.18.0,>=0.15.0', [])
  adding ('idna', '>=2.8', [])
  adding ('sniffio', '==1.*,>=1.1', [])
Removed dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  anyio<5.0,==3.7.1,>=3.0 (from -r requirements.txt (line 7))
  certifi==2023.5.7 (from -r requirements.txt (line 9))
  exceptiongroup==1.1.2 (from -r requirements.txt (line 13))
  h11<0.15,==0.14.0,>=0.13 (from -r requirements.txt (line 15))
  httpcore<0.18.0,==0.17.3,>=0.15.0 (from -r requirements.txt (line 17))
  httpx==0.24.1 (from -r requirements.txt (line 19))
  idna==3.4,>=2.8 (from -r requirements.txt (line 21))
  sniffio==1.*,==1.3.0,>=1.1 (from -r requirements.txt (line 25))

Finding the best candidates:
  found candidate anyio==3.7.1 (constraint was >=3.0,==3.7.1,<5.0)
  found candidate certifi==2023.5.7 (constraint was ==2023.5.7)
  found candidate exceptiongroup==1.1.2 (constraint was ==1.1.2)
  found candidate h11==0.14.0 (constraint was >=0.13,==0.14.0,<0.15)
  found candidate httpcore==0.17.3 (constraint was >=0.15.0,==0.17.3,<0.18.0)
  found candidate httpx==0.24.1 (constraint was ==0.24.1)
  found candidate idna==3.4 (constraint was >=2.8,==3.4)
  found candidate sniffio==1.3.0 (constraint was ==1.*,>=1.1,==1.3.0)

Finding secondary dependencies:
  h11==0.14.0               requires -
  httpx==0.24.1             requires certifi, httpcore<0.18.0,>=0.15.0, idna, sniffio
  certifi==2023.5.7         requires -
  idna==3.4                 requires -
  anyio==3.7.1              requires exceptiongroup; python_version < "3.11", idna>=2.8, sniffio>=1.1
  exceptiongroup==1.1.2     requires -
  sniffio==1.3.0            requires -
  httpcore==0.17.3          requires anyio<5.0,>=3.0, certifi, h11<0.15,>=0.13, sniffio==1.*
------------------------------------------------------------
Result of round 2: stable, done

#
# This file is autogenerated by pip-compile with Python 3.8
# by the following command:
#
#    pip-compile --output-file=requirements.in requirements.txt
#
anyio==3.7.1
    # via
    #   -r requirements.txt
    #   httpcore
certifi==2023.5.7
    # via
    #   -r requirements.txt
    #   httpcore
    #   httpx
exceptiongroup==1.1.2
    # via
    #   -r requirements.txt
    #   anyio
h11==0.14.0
    # via
    #   -r requirements.txt
    #   httpcore
httpcore==0.17.3
    # via
    #   -r requirements.txt
    #   httpx
httpx==0.24.1
    # via -r requirements.txt
idna==3.4
    # via
    #   -r requirements.txt
    #   anyio
    #   httpx
sniffio==1.3.0
    # via
    #   -r requirements.txt
    #   anyio
    #   httpcore
    #   httpx

On first glance, casting this log call to a print call would work:

https://github.com/jazzband/pip-tools/blob/4c7bc26541ae11b18f8b9260104eede32cbc114a/piptools/writer.py#L267

[daveisfera]

Sorry, that was a typo. All of the output does go to stderr.txt 🤦

I agree that it sees that having the final result be a print so it goes to stdout would be the simplest solution

[globau]

It should just be a matter of added err=False to both log.info() (and log.log() a few lines earlier), however with --verbose the output from pip also needs to be redirected to stderr which is a little more exciting to resolve.