total-byte-weight started failing but only on the CLI

nstringham commented 9 months ago

FAQ

[X] Yes, my issue is not about variability or throttling.
[X] Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://othello-rust.web.app/

What happened?

recently the lhci autorun command began giving the following error

       ✘  total-byte-weight failure for minScore assertion
       Avoids enormous network payloads
       https://developer.chrome.com/docs/lighthouse/performance/total-byte-weight/

        expected: >=0.9
           found: 0
      all values: 0, 0, 0

config:

ci:
  collect:
    autodiscover-url-blocklist: /privacy-policy.html
  upload:
    target: "temporary-public-storage"
  assert:
    preset: "lighthouse:recommended"
    assertions:
      csp-xss: "off"
      bf-cache: "warn"

What did you expect?

I would expect the CLI to pass because I do not get an error about "Avoid enormous network payloads" anywhere else. including in the HTML report generated by the CLI

What have you tried?

using the lighthouse inside of chrome and using the lighthouse inside of page speed insights

How were you running Lighthouse?

CLI

Lighthouse Version

0.13.0

Chrome Version

120.0.6099.129

Node Version

v18.12.1

OS

Windows 11 23H2

Relevant log output

https://github.com/nstringham/othello-web-app/actions/runs/7315817257/job/19929865498

davidlj95 commented 9 months ago

+1 Another job run sample:

https://github.com/davidlj95/chrislb/actions/runs/7305097678/job/19908261745?pr=264

If running locally, I can see in the total byte weight audit is around 320Kbs which shouldn't raise the audit AFAIK

Local output:

lighthouseci.zip

nstringham commented 8 months ago

Any update on this?

gilhanan commented 8 months ago

I'm also expereince this issue. If someone want to repro it easily: npx @lhci/cli autorun --collect.url=https://gh-lhci-azure-demo-app---staging.jollyrock-437e64c9.northeurope.azurecontainerapps.io --assert.preset=lighthouse:no-pwa


  ×  total-byte-weight failure for minScore assertion
       Avoids enormous network payloads
       https://developer.chrome.com/docs/lighthouse/performance/total-byte-weight/

        expected: >=0.9
           found: 0
      all values: 0, 0, 0

davidlj95 commented 4 months ago

After digging again into this, noticed that score is null in the JSON report

    "total-byte-weight": {
      "id": "total-byte-weight",
      "title": "Avoids enormous network payloads",
      "description": "Large network payloads cost users real money and are highly correlated with long load times. [Learn how to reduce payload sizes](https://developer.chrome.com/docs/lighthouse/performance/total-byte-weight/).",
      "score": null,
      "scoreDisplayMode": "informative",
      "numericValue": 356395,
      "numericUnit": "byte",
      "displayValue": "Total size was 348 KiB",
      "details": {
        "type": "table",
        "headings": [
          {
            "key": "url",
            "valueType": "url",
            "label": "URL"
          },
          {
            "key": "totalBytes",
            "valueType": "bytes",
            "label": "Transfer Size"
          }
        ],
        "items": [
          {
            "url": "http://localhost:4000/chunk-XALDH5GC.js",
            "totalBytes": 55713
          }
        ],
        "sortedBy": [
          "totalBytes"
        ]
      },
      "guidanceLevel": 1
    },

(removed items to paste a shorter JSON)

davidlj95 commented 4 months ago

Checkout for a workaround here:

https://github.com/GoogleChrome/lighthouse-ci/issues/994#issuecomment-2096481472

GoogleChrome / lighthouse-ci