Inconsistent results across devices

[Paul-Hebert]

I got inconsistent results between two different computers. Both were Macbooks running Big Sur on the same network.

Here's what I got on my newer M1 device:

Here's the same site on my older non-M1 device:

(Note this is also more consistent to the Lighthouse scores in the browser. The browser scores were consistent across devices)

The Lighthouse docs mention variability but this seems pretty extreme.

I saw similar results on a number of sites. I tried making sure to use the same version of Node, reinstalling Chrome, restarting my machine, etc.

I'm not sure if this is related to it being an M1 machine or something else?

[spaceninja]

I was recently able to run Lighthouse Parade on my M1 and it seemed to run fine. ¯_(ツ)_/¯

[calebeby]

I re-ran this on my mac (not an M1) and asked Danielle run it on hers (M1) and we are still seeing this issue:

Non-M1:

M1:

Command used:

nvm use 14
npx lighthouse-parade https://cloudfour.com/ --max-crawl-depth 3 --exclude-path-glob "/thinks/*" --exclude-path-glob "/presents/*"

[calebeby]

Ran again with the command:

npx lighthouse-parade https://cloudfour.com --max-crawl-depth 2 --exclude-path-glob "/thinks/*" --exclude-path-glob "/presents/*" --lighthouse-concurrency 1

(to avoid parallelism):

Results:

The stats that were similar between M1 and non-M1:

• Largest Contentful Paint
• Cumulative Layout Shift
• Speed Index
• Time to Interactive
• First meaningful paint

The stats that were significantly different between M1 and non-M1:

• Max potential first input delay
• Total blocking time
• Estimated Input Latency
• Minimize main-thread work

[Paul-Hebert]

Thanks Caleb! That's interesting. I wonder why that is....

[calebeby]

I am looking into it more. I've looked through lighthouse issues to see if they list anything similar, but I couldn't find anything. Gonna try updating lighthouse too

[Paul-Hebert]

We were able to dig into this more in #92

The issue crops up when "using virtualized x64 instead of natively running arm instructions" (quoting @calebeby because he has a better grasp of this than me)

I believe this means that if you're on an M1 machine you'll need to use Node 15 or 16 and confirm that your Node version is using arm64 as the underlying architecture. @calebeby , @spaceninja , does that sound correct? If so what's the best place for us to document this? Should we add a Troubleshooting section to the Readme?

[spaceninja]

Yeah, adding a section in the README should be enough. I'd imagine something like:

---

Mac M1 Note

Users running virtualized x64 versions of Node on Macs with M1 chips may see anomalous results. For best results on an M1 chip, ensure you are running the arm version of Node 15+.

When you run the command node -p process.arch you should see arm64. If you see x86, try uninstalling and reinstalling your version of Node, like so:

nvm uninstall 16
nvm install 16
nvm use 16
node -p process.arch

[calebeby]

Addressed in #93!!