CI: detect perf/size regressions (build size bot, lighthouse...)

[slorber]

Integrate Build Size Bot

Integrate build size bot (or alternatives) to run on PRs to detect bundle size and performance regressions whenever changes are made to the repository.

Example of Next.js PR

[hamzahamidi]

Is it Ok to use the code source of the action created by Next.js ?

[slorber]

@hamzahamidi thanks for your contributions.

The issues with MLH tags are for MLH interns joining us for 2 months. As they just arrived I'd like to see how we assign the issues with the MLH tags first, and maybe delegate those issues to other contributors if MLH interns are already busy.

If you are looking to help on something, maybe we can get in touch on Discord so that we find an issue for you?

[jcs98]

Hello @slorber may I take this up?

[slorber]

yes thanks, let me know if you need me to do something about it

[jcs98]

Hi, This is what I have in mind: On a new PR, the GitHub action will run yarn build and then Calculate the build size as the sum of all the lib folders in the 15 docusaurus packages And also the time it took to execute the build

Can you please tell me if this is okay or something else is expected?

Example comment by the build-size-bot:

slorber commented 3 years ago

Hi @jcs98

Tracking the size of our NodeJS code is probably not the most important thing to track.

What matters most are:

• tracking build time (on our init template + our v2 website)
• tracking static output size (size of HTML files, size of JS bundles etc...)

The system should be able to prevent major perf regressions such as:

• we included lodash in theme-classic by mistake, and client side JS code increase significantly
• we have a perf issue in node code (like sync/blocking fs access), and this increase build time significantly

Also, it's worth looking at the existing tools. Maybe some tools can already provide us useful information. I'm pretty sure there are Webpack stats plugins that we could use.

Can you do some research of existing things you can use? The goal is not necessarily to implement everything from scratch, but figure out a simple/good enough solution with a good ROI. Don't hesitate to look at the setup of existing frameworks. NextJS is an example, but we don't need to go this far.

jcs98 commented 3 years ago

Thank you @slorber I have started looking at a few things.

Can you please confirm if "tracking static output size" is calculating the size of the folder website/build or something else? Sorry if I am missing something here.

I also looked at some existing tools:

• bundlesize that can be integrated with circle ci
• the one that next.js uses (https://github.com/vercel/next-stats-action)
• compressed-size-action (although this seems to be quite verbose in my attempt here)

Would any of these work here?

slorber commented 3 years ago

Yes, we'll likely want to track if the site of a Docusaurus site grows, mostly in website/build.

We need to define the relevant metrics for our usecase, and figure out how to measure them and track them over time to avoid regressions.

Some metrics I can think of:

• Build time
• Webpack chunks bundle sizes (can maybe found in Webpack manifest / stats output?)
• Lighthouse score
• Accessibility scores
• ... anything relevant

We may even try to partner with external saas that offers free plans for open-source (like calibreapp.com?) to get high-quality tracking.

slorber commented 3 years ago

Don't hesitate to try multiple services at the same time, we can try many of them, and after a while only keep the ones that we find the most useful.

Just try to get a not too verbose output, because measuring the size of all the /build folder is probably not very relevant for our usecase.

You can also try SaaS services as well, and see if they offer open-source free licenses.

jcs98 commented 3 years ago

Thank you for the direction! I'll try a few different things and keep you posted.

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

Package Name	Build Size	Build Time
docusaurus-init	120KB	32s
docusaurus-migrate	40KB	12s
...	...	...
Total	300KB	82s

[slorber]

Hi @jcs98

Tracking the size of our NodeJS code is probably not the most important thing to track.

What matters most are:

• tracking build time (on our init template + our v2 website)
• tracking static output size (size of HTML files, size of JS bundles etc...)

The system should be able to prevent major perf regressions such as:

• we included lodash in theme-classic by mistake, and client side JS code increase significantly
• we have a perf issue in node code (like sync/blocking fs access), and this increase build time significantly

Also, it's worth looking at the existing tools. Maybe some tools can already provide us useful information. I'm pretty sure there are Webpack stats plugins that we could use.

Can you do some research of existing things you can use? The goal is not necessarily to implement everything from scratch, but figure out a simple/good enough solution with a good ROI. Don't hesitate to look at the setup of existing frameworks. NextJS is an example, but we don't need to go this far.

[jcs98]

Thank you @slorber I have started looking at a few things.

Can you please confirm if "tracking static output size" is calculating the size of the folder website/build or something else? Sorry if I am missing something here.

I also looked at some existing tools:

• bundlesize that can be integrated with circle ci
• the one that next.js uses (https://github.com/vercel/next-stats-action)
• compressed-size-action (although this seems to be quite verbose in my attempt here)

Would any of these work here?

[slorber]

Yes, we'll likely want to track if the site of a Docusaurus site grows, mostly in website/build.

We need to define the relevant metrics for our usecase, and figure out how to measure them and track them over time to avoid regressions.

Some metrics I can think of:

• Build time
• Webpack chunks bundle sizes (can maybe found in Webpack manifest / stats output?)
• Lighthouse score
• Accessibility scores
• ... anything relevant

We may even try to partner with external saas that offers free plans for open-source (like calibreapp.com?) to get high-quality tracking.

[slorber]

Don't hesitate to try multiple services at the same time, we can try many of them, and after a while only keep the ones that we find the most useful.

Just try to get a not too verbose output, because measuring the size of all the /build folder is probably not very relevant for our usecase.

You can also try SaaS services as well, and see if they offer open-source free licenses.

[jcs98]

Thank you for the direction! I'll try a few different things and keep you posted.