[BUG] Lighthouse performance is returning NULL most of the time

[wafendy]

Environment

• chrome-aws-lambda Version: 7.0.0
• puppeteer / puppeteer-core Version: 7.1.0
• OS: Mac
• Node.js Version: 12.x
• Lambda / GCF Runtime: nodejs12.x

Expected Behavior

Google Lighthouse executed successfully and always returns Performance score

Current Behavior

Google Lighthouse executed successfully BUT most of the time Performance score is null.

Steps to Reproduce

I created a github repo to help reproduce the error.

https://github.com/wafendy/aws-lambda-lighthouse-docker

main branch: Using the latest version which is v7.0.0 last-known-working branch: Using version v3.1.1

v3.1.1 is last known stable version for my setup.

Possible Solution

chrome-aws-lambda v3.1.1 is the last known working version, but it has may security vulnerabilities and no more updates to addresss the vulnerabilities.

Perhaps, I'm using it wrongly or there is missing configuration.

[dochne]

We can replicate this issue, it seems to fail approximately 25% of the time for us.

Something similar seems to have been flagged here https://github.com/GoogleChrome/lighthouse/issues/11537. with a response of "retry it a few times", which isn't exactly the solution we were aiming for 😂

[ACAHNN]

In case this is helpful for anyone. I was having this issue when driving lighthouse via puppeteer. What fixed it for me was removing an argument that is passed to chromium. Specifically, I was using the --single-process flag. Commenting this out did the trick for me.

[code]

await puppeteer.launch({
        ignoreHTTPSErrors: true,
        headless: true,
        args: [//'--single-process',
           '--no-zygote',
           '--no-sandbox',
          ]
    });

[configuration]

node: 14.16.0 lighthouse: 7.2.0 puppeteer: 8.0.0 platform(s): macOS Big Sur (local), Ubuntu 20.04 (remote EC2 instance)

[wafendy]

When --single-process argument is removed, my local docker can run just fine without any error.

However if I run the same code on AWS lambda, I'm getting the following error:

2021-04-26T15:26:56.873+08:00Copy2021-04-26T07:26:56.873Z   045ca199-78fa-4231-8913-930254a5ed9e    ERROR   Error: connect ECONNREFUSED 127.0.0.1:9222
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16) {
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 9222
} 

[rozenmd]

If you want meaningful (not highly variable) results, don't use AWS Lambda to run Lighthouse. Look at EC2, or ECS.

I started with AWS Lambda when I built OnlineOrNot.com, but results would yoyo up to 50% between runs (and that's with running each test 5 times).

[louist7]

We're having this same problem at the minute, @wafendy did you ever find a solution for this?

[dev-67]

Did anyone get a solution for this? @wafendy @louist7

[louist7]

We found that increasing the memory on the lambda function to 2048 MB, and the timeout to 300 seconds, improved the reliability of the results enough for us to be able to use it.