Assertion is not done against `isRepresentativeRun` when using `median-run`

[b3nk3]

Describe the bug When using the median-run aggregation method I'd expect that assertions are done against the run, that has the property of isRepresentativeRun in the manifest.json. This does not appear to be the case.

Manifest:

{
    "url": "http://localhost:3000/",
    "isRepresentativeRun": true,
    "htmlPath": "frontend/.lighthouseci/localhost-_-2024_07_26_09_57_40.report.html",
    "jsonPath": "frontend/.lighthouseci/localhost-_-2024_07_26_09_57_40.report.json",
    "summary": {
      "performance": 0.68,
      "accessibility": 0.98,
      "best-practices": 1,
      "seo": 1
    }
  },

However, when correlating this with the assertion-resutls.json it is apparent, that a different run was used for assertion.

[
  {
    "name": "minScore",
    "expected": 0.7,
    "actual": 0.69,
    "values": [
      0.57,
      0.69,
      0.68
    ],
    "operator": ">=",
    "passed": false,
    "auditProperty": "performance",
    "auditId": "categories",
    "level": "error",
    "url": "http://localhost:3000/"
  }
]

To Reproduce Steps to reproduce the behavior: Use run LHCI with filesystem target, using the following assertion:

{
          matchingUrlPattern: `^${PREVIEW_URL}/$`,
          assertions: {
            'categories:performance': ['error', { minScore: 0.7, aggregationMethod: 'median-run' }],
            'categories:accessibility': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
            'categories:best-practices': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
            'categories:seo': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
          },
        },

Expected behavior

Assertion should be done against the run with the isRepresentativeRun property in the manifest.json

As per the description of median-run in the Aggregation methods section of the config documentation:

Use the value of the run that was determined to be "most representative" of all runs...

Environment (please complete the following information):

• OS: macOS, ubuntu
• Version: @lhci/cli@0.14.x

[b3nk3]

pinging for an update...

[b3nk3]

Still looking for an answer on this

[paulirish]

@b3nk3 Sorry Ben.. It's unlikely we'll be able to dig into this anytime soon. Very happy to review PRs, though. Apologies we can't investigate ourselves.