Overview

There is a lot of new code here! Happy to do a video walkthrough if that is helpful.

Here is what stayed mostly the same:

The crawler code. In a follow-up PR I will probably swap out the crawler. For now, I just adapted the output so that it is an async iterator instead of an event emitter. I moved createUrlFilter to a separate file without changing it.
The CLI code/args parsing/progress displaying

Everything else was rewritten:

lighthouse.ts now manages a pool of worker_threads (Node's version of web workers). Each worker thread runs the lighthouse-worker.ts file. It figures out which worker is available to handle a lighthouse run, and tells it to run it.
lighthouse-worker.ts launches a chrome instance, and runs lighthouse whenever it is requested to.
async-iterator-queue is a data structure that has an async iterator that loops through its items, yielding as many as it has. It will pause when it runs out of items. As push is called more, the new items are yielded as well. When finish is called, the iterator exits. It is used to adapt the simplecrawler output to an async iterator for use in main.ts
main.ts wires everything together. It is called from the cli code. It manages the state and pulls in the URLs from the crawler, requests a worker from lighthouse.ts, and sends the URL to the worker. Then it takes the result and updates the state, and passes the result to the OutputWriter.
output-writer/index.ts contains the OutputWriter interface. Right now there is a CSV OutputWriter and a stub for a google-sheets OutputWriter. It also has a function to merge several OutputWriters together so the code in main can pretend as though there is only one OutputWriter even when there are distinct outputs.

This PR targets next. Here are other upcoming changes I'm hoping to do in separate PR's before release:

google-sheets output format
Replace simplecrawler
Add more flags for customizing lighthouse (and/or allow reading config file)
Write documentation (in this PR I deleted partially-out-of-date documentation so I wouldn't forget to update it once all the CLI flags /etc are all settled. I'll copy it back and update it in a follow up PR).
I would like to set up a test suite that runs a static HTML server and runs lighthouse-parade against it, which we can use as a non-changing reference, which will be helpful to review lighthouse updates, etc.
Bundle the JS output instead of directly using output from tsc
I will write changesets.

For review

I would suggest turning off whitespace changes, which will help just a little bit. Also GitHub has the nice new file tree on the left of the review page, it is probably helpful to use that.

Testing

Create a baseline lighthouse run to compare against:

git checkout main
npm i
npm run build
npm i -g lighthouse-parade
npm link
lighthouse-parade ...

Try the new changes

git checkout lighthouse-node-api
npm i
npm run build
npm link
lighthouse-parade ...

Then you can compare the generated CSV data and make sure the differences are minor. The new CSV format won't work with the existing google sheets template, so you can just review the CSV directly or in Numbers or something for now.

cloudfour / lighthouse-parade

Use lighthouse node API and restructure output writing #111

Overview

For review

Testing

🦋 Changeset detected