Run note-level validation rules by default

microsoft / sarif-sdk

.NET code and supporting files for working with the 'Static Analysis Results Interchange Format' (SARIF, see https://github.com/oasis-tcs/sarif-spec)

Other

193 stars 92 forks source link

Run note-level validation rules by default #2018

Open ghost opened 4 years ago

ghost commented 4 years ago

The validator does not run note-level rules by default, so it's too easy to forget to run them. Make them run by default. Options:

Provide a --quiet options to turn off note-level rules.
Provide a detailed filter arg that allows users to choose arbitrary failure levels. Error|Warning|Note would be the default.

eddynaka commented 3 years ago

Hi @lgolding ,

I was taking a look at this...

if I use the following command: sarif validate sarif_with_note_errors.sarif -o temp.sarif, it is showing all the note issues in the temp.sarif file. Is that the same command that you were talking about?

What I used: the latest sarif.multitool from nuget.org (version 2.3.8)

ghost commented 3 years ago

That is the command. Could you please attach the log file?

eddynaka commented 3 years ago

Hi @lgolding , my mistake. I saw results, but those aren't Note.

eddynaka commented 3 years ago

Testing a little more, I saw that:

if we use --verbose true: it will print all results (none, note, warning, error)
if we use --quiet true: it won't show in the console
if we use --verbose true --quiet true, it will print all results, but won't show in the console

Based on that, we could: enable verbose by default.

ghost commented 3 years ago

Then the question is how you turn the note-level results off. The proposal in the issue description (written at a moment when I forgot we already have a --quiet option) was to turn off note-level results if --quiet is specified.

Now I am starting to think that it is confusing to have --quiet and --verbose control two different things:

How much console output do you see?
Which results do you see?

Maybe we need separate options. I leave it to you and Michael to decide the correct behavior.