search

GoogleChrome / lighthouse-ci

Automate running Lighthouse for every commit, viewing the changes, and preventing regressions
Apache License 2.0
6.4k stars 641 forks source link

Need More Documentation of Assertion Flags for Min Score #829

Open ps185367 opened 2 years ago

ps185367 commented 2 years ago

I'm trying to move away from using a configuration file to using just flags but having trouble converting specific assertions with min scores. Here is the example of what I'm trying to do but it doesn't seem to be working as expected.

lhci assert --assertions.categories:accessibility="['error', { minScore: 0.95 }]"

All of the documented examples show how to turn off an assertion but not how to pass on things like minScore. Thanks.

ps185367 commented 2 years ago

After more testing it seems the problem is with categories:accessibility in particular as using categories:performance in the above command works as expected. I've noticed misspelling assertions causes an error message but I receive no such error message for categories:accessibility. For reference, viewing the collect file confirms a score of 94 was given for accessibility so it should be failing.

devin-purple commented 4 months ago

Does not work for me either. Cannot get lhci to fail on accessibility with minScore.

Edit: does work for me, oddly I had to remove a budget.json file I was using in a github action.

{
  "ci": {
    "assert": {
      "assertions": {
        "categories:accessibility": [
          "error",
          {
            "minScore": 0.9
          }
        ]
      }
    },

....

That results in this:

image