Second run and beyond falls back to old Chrome headless mode

[dcsaszar]

Describe the bug

The Chrome --headless=new option is only effective on the first run because of repeated --headless=new --headless=new ... for the other runs, which apparently triggers the old headless mode.

To Reproduce Steps to reproduce the behavior:

1. Pick a URL that reports a significantly lower or higher performance with the --headless=new mode compared to --headless.
   • I use https://dev.d33869rf57mkjo.amplifyapp.com/ as an example here (found in https://github.com/GoogleChrome/lighthouse/issues/6708).
   • http://www.lumme.info/ (from https://github.com/GoogleChrome/lighthouse/issues/14957) should also show an effect (only terminate with --headless=new)
2. Use this .lighthouserc.yml:

   ci:
   assert:
   assertions:
     categories:performance: [error, minScore: 0]
   includePassedAssertions: true
   collect:
   url:
     - https://dev.d33869rf57mkjo.amplifyapp.com/
   upload:
   outputDir: .lighthouseci/results
   target: filesystem

3. npx lhci autorun
4. Check the three reported values
   • The first value matches the score of the browser devtools Lighthouse.
   • The second and third score are different from the first.

Checking assertions against 1 URL(s), 3 total run(s)

1 result(s) for https://dev.d33869rf57mkjo.amplifyapp.com/ :

  ✅  categories.performance passing for minScore assertion
        expected: >=0
           found: 1
      all values: 1, 0.73, 0.72

Expected behavior

• All three scores should be similar.
• All three scores should match the browser devtools Lighthouse result.

The expected behavior can be enforced by adding under collect::

    headful: true
    settings:
      chromeFlags: --headless=new

Checking assertions against 1 URL(s), 3 total run(s)

1 result(s) for https://dev.d33869rf57mkjo.amplifyapp.com/ :

  ✅  categories.performance passing for minScore assertion
        expected: >=0
           found: 1
      all values: 1, 1, 1

Environment (please complete the following information):

• OS: macOS 14.5
• Browser: Chrome
• Version: 127.0.6533.89

Additional context

console.log({chromeFlagsAsString}) in static computeArgumentsAndCleanup(url, options) logs the problem.

[organich]

Describe the bug

The Chrome --headless=new option is only effective on the first run because of repeated --headless=new --headless=new ... for the other runs, which apparently triggers the old headless mode.

To Reproduce Steps to reproduce the behavior:

1. Pick a URL that reports a significantly lower or higher performance with the --headless=new mode compared to --headless.

• I use https://dev.d33869rf57mkjo.amplifyapp.com/ as an example here (found in ☔ Pagespeed Insights results differ from lighthouse in chrome lighthouse#6708).
• http://www.lumme.info/ (from Lighthouse stalls when testing bfcache in old headless mode lighthouse#14957) should also show an effect (only terminate with --headless=new)

2. Use this .lighthouserc.yml:

ci:
  assert:
    assertions:
      categories:performance: [error, minScore: 0]
    includePassedAssertions: true
  collect:
    url:
      - https://dev.d33869rf57mkjo.amplifyapp.com/
  upload:
    outputDir: .lighthouseci/results
    target: filesystem

3. npx lhci autorun
4. Check the three reported values

• The first value matches the score of the browser devtools Lighthouse.
• The second and third score are different from the first.

Checking assertions against 1 URL(s), 3 total run(s)

1 result(s) for https://dev.d33869rf57mkjo.amplifyapp.com/ :

  ✅  categories.performance passing for minScore assertion
        expected: >=0
           found: 1
      all values: 1, 0.73, 0.72

Expected behavior

• All three scores should be similar.
• All three scores should match the browser devtools Lighthouse result.

The expected behavior can be enforced by adding under collect::

    headful: true
    settings:
      chromeFlags: --headless=new

Checking assertions against 1 URL(s), 3 total run(s)

1 result(s) for https://dev.d33869rf57mkjo.amplifyapp.com/ :

  ✅  categories.performance passing for minScore assertion
        expected: >=0
           found: 1
      all values: 1, 1, 1

Environment (please complete the following information):

• OS: macOS 14.5
• Browser: Chrome
• Version: 127.0.6533.89

Additional context

console.log({chromeFlagsAsString}) in static computeArgumentsAndCleanup(url, options) logs the problem.