Allow output to file with different format to output to console #2388

Open ngnpope opened 1 year ago

Currently, when running in CI I have to do the following, e.g. in a (simplified) .gitlab-ci.yml:

lint:
  script:
    - ...
    - ruff .
    - ruff --format=gitlab . > quality.json
  artifacts:
    reports:
      codequality: quality.json

I'd like to keep the text output in the job output, but also make use of the capabilities GitLab provides to display violations.

Now, ruff is super fast ⚡, so running twice isn't a pain, but it seems suboptimal. Could we enable one of the following?

ruff --format=text --output=gitlab:/path/to/quality.json . ← dedicated --output option
ruff --format=text --format=gitlab:/path/to/quality.json . ← allow multiple --format with filename
ruff --format=text,gitlab:/path/to/quality.json . ← allow multiple values in --format

I'd rather avoid CLI options with such a complex syntax, since it complicates the CLI and also doesn't work with the shell autocompletion generation by clap_complete.

How about instead introducing a new --stderr-format option that accepts the same values as the current --format option and would additionally output the specified format on stderr? ... then you could replace your two commands with ruff --stderr-format=gitlab . 2> quality.json (2> is shell syntax for stderr redirection).

I guess we could do that, but that would only work as long as ruff never chose to output something else to stderr...

It seems a little irregular. Normally stdout is used for the normal output from a program and stderr is use to output errors or diagnostic information about the execution of that program.

Fair point ... and it also wouldn't support outputting to more than two formats. I think it's probably best to just implement a new convert subcommand that can consume the json output and convert it to any of the other supported output formats ... although for that to work nice with shell pipes we'd need a stream-friendly JSON format (where each message is on one line).

So --format=jsonl then? (See https://jsonlines.org/)

I guess that allows for using tee to redirect to multiple processors then?

It would work, but does just seem much more complex than a command line option... 🤔

The advantages I see are that 1) it wouldn't complicate the ruff check CLI for users who don't need this feature, 2) shell completion would work fine and 3) it's not unthinkable that we end up introducing more format-specific options (for --format text we already have several), having one ruff command call always only produce one output avoids the ambiguity of which output option belongs to which output.

I agree that how to use this would be less obvious for users unfamiliar with shell piping/redirection, however I think that could be accommodated via an example in the documentation ... and I wouldn't consider this to be something users commonly use but rather something you set up once for a CI, so I think that should be alright.

Also think about this situation Is it possible to have: 1) always have stdout with some default format, ex. grouped/text 2) if --outputfile=/tmp/file than apply --format to file, not to stdout 3) disable stdout with -q, like it should be

I prefer not to run ruff more than one time and not to have any |jq processors

Having a dedicated output (or output-file) option has precedence in

ESLint (output-file | -o)
dotnet format --report
Jest (--outputFile) in combination with --json

Ah good call. I'd want to look at what ESLint does in various cases. If you use --fix, does it write the fixed code to the output file? (Does it write the fixed code to stdout, if you write via stdin? I never checked that). What should go to stderr vs. stdout? Etc.

We now support writing to a file (#4950), but not to both standard output and a file. I'm going to close this, and we can open a fresh issue if that additional behavior is desired.

@charliermarsh Could you re-open this? I'm not sure why it makes sense to create a new issue when it would be exactly the same description... This issue was clearly stated as allowing the usual text output to standard output while also allowing a separate output to file for a specific report format.

The use case in question is for runs in CI where we want the usual report to the job output but would also like to generate a report file, e.g. for GitLab, so that artifact can be used to integrate the results with their UI. (This could/would also apply for services other than GitLab.) Currently this is impossible to achieve without running ruff twice.

I should follow up that, while the PR you linked to (#4950) does support writing to an output file with a flag, it doesn't do anything other than remove the need for shell redirection.

CC: @dhruvmanila

I would also be very glad to have this feature included in Ruff (https://github.com/astral-sh/ruff/discussions/6467).

@charliermarsh Good day! Is any new thoughts about CI usage of ruff? We stiil have to run it twice in a row

Agreed this would be a nice feature.
Pylint now supports multiple output formats as follows:
pylint --output-format=colorized,parseable:pylint.log

That's not to say their syntax is the best but the ability target multiple outputs would be great for CI systems.

astral-sh / ruff

Allow output to file with different format to output to console #2388