Mismatching results from Lighthouse report attached to the output manifest

[drazenbuljovcic]

Hi, I have a case of the numbers of the lighthouse score in the report are not equal as we see them in the PR.

Type Bug

The result variable below which inside the console.log is retrieved by ${{ steps.lighthouse_audit.outputs.manifest }}[0].summary. lighthouse_audit is the identifier for the above step that runs the audit!

https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1624269254964-65478.report.html

Part of lighthouse.yaml..

...
      - name: Audit URLs using Lighthouse
        id: lighthouse_audit
        uses: treosh/lighthouse-ci-action@v7
        with:
          configPath: './lighthouserc.json'
          uploadArtifacts: true
          temporaryPublicStorage: true
      - name: Format lighthouse score
        id: format_lighthouse_score
        uses: actions/github-script@v3
        with:
          github-token: ${{secrets.GITHUB_TOKEN}}
          script: |
            const result = ${{ steps.lighthouse_audit.outputs.manifest }}[0].summary
            console.log({ result })
            const links = ${{ steps.lighthouse_audit.outputs.links }}
            const formatResult = (res) => Math.round((res * 100))
            Object.keys(result).forEach(key => result[key] = formatResult(result[key]))
            const score = res => res >= 90 ? '🟢 ' : res >= 50 ? '🟠 ' : '🔴 '
            const comment = [
                `⚡️ [Lighthouse report](${Object.values(links)[0]}) for the changes in this PR:`,
                '| Category | Score |',
                '| --- | --- |',
                `| ${score(result.performance)} Performance | ${result.performance} |`,
                `| ${score(result.accessibility)} Accessibility | ${result.accessibility} |`,
                `| ${score(result['best-practices'])} Best practices | ${result['best-practices']} |`,
                `| ${score(result.seo)} SEO | ${result.seo} |`,
                `| ${score(result.pwa)} PWA | ${result.pwa} |`,
            ].join('\n');
            core.setOutput("comment", comment);
 ...

Expected The results returned to the script are exactly the same as the generated report!

[awentzel]

Any explanation on why the results are mismatched?

[johnnyreilly]

I also see this quite regularly. Consider the following screenshot which is driven by the outputs of the GitHub Action:

Performance is listed as 65.

However, if we look at the actual report:

Performance is listed as 68. Not a vast difference, but crucially these are not the same numbers.

[Yu040419]

Same here with v9.

[mshick]

I ran into this issue. It looks like the last item in your manifest is the score that gets reported, but most formatting scripts are grabbing the first item in the array.

You can fix relatively easy by updating your formatting logic from something like this:

const result = ${{ steps.lighthouse_audit.outputs.manifest }}[0].summary

to this:

const manifest = ${{ steps.lighthouse_audit.outputs.manifest }}
const result = manifest[manifest.length - 1].summary

[mttfiorio]

I have the same problem. I think that generally printing the median is better when looking at multiple runs

const outputs = ${{ steps.lighthouse_audit.outputs.manifest }};
outputs.sort((a, b) => a.summary.performance - b.summary.performance);
const middleIndex = Math.floor(outputs.length / 2);
const result = outputs[middleIndex].summary

Sadly you will have the same mismatch with this code