Feature: Compare different URLs

[schlessera]

Right now, the UI for the LHCI server allows comparisons only for the same URL across different commits.

I'm currently working on the AMP for WordPress plugin, and we have a different use case.

We have different subsystems where we want to measure the impact, and for doing so we have URL arguments to toggle them on and off. This is done because LHCI server only allows for a single report per URL on a given commit hash, so using different URLs for different variations of the same version of the code makes sense.

Example URLs:

'http://localhost:8080/',
'http://localhost:8080/?amp=1&amp_optimizer=0&amp_ssr=0',
'http://localhost:8080/?amp=1&amp_optimizer=1&amp_ssr=0',
'http://localhost:8080/?amp=1&amp_optimizer=1&amp_ssr=1',

The UI on the LHCI server does not allow us to see diffs from one URL to the other (whether that'd be on the same commit or a different one). It only allows for comparisons in time (different commits for the same location), but we'd also need comparisons across locations

Sticking to the same commit in that case would already be enough for our purpose, in case making both dimensions flexible at the same time proves difficult in terms of UX.

[patrickhulce]

Thanks for filing @schlessera! I like this feature request as a generic run comparison view where any URL from any build could be compared with any other.

[amedina]

This is a great feature and it will play a key role on evaluating and tracking optimizations towards improving core vitals, specifically in our case for sites powered by the AMP plugin.

/cc @paulirish @egsweeny

[gauthampughazhendhi]

@paulirish I am also in need of this feature for my use-case. Any timeline on when it would be available as part of the stable release?

[patrickhulce]

There is no timeline for anyone working on this in the near future. It's open source though so feel free open a PR ;)

[gauthampughazhendhi]

@patrickhulce any development guide available for this to better understand the existing implementation?

[patrickhulce]

There's architecture documentation that's definitely a must, but nothing really specific to the server UI implementation. I'll add some notes on it to CONTRIBUTING.md 👍

[patrickhulce]

FYI some guidance is now in @gauthampughaz #359

[lazysergey]

I also need this and not because I need to compare different url, but because I need to compare same path but for different environments - I need to compare results on production and feature-branch urls to realise what impact will have the branch, but seems I am able to compare only exactly same urls - urls below are incomparable:

https://mydomain.com/xxxx/ https://some-feature-branch.mydomain.com/xxxx/

I didn't find any workarounds to allow different urls comparison, but it feels essential to me when I want to compare results of feature-branch build deployed to some subdomain or dev server, not to the production itself

I have exactly same setup on feature-branches so it is safe and valid to compare them with production

[patrickhulce]

@lazysergey that situation is exactly what --urlReplacementPatterns is for.

[Ovicakov]

Hi ! Thanks @patrickhulce for explanations. We have also have to compare different URLs from our dev environnement (we are working with Travis), i.e. https://master.domain.xx vs https://my-feature.domain.xx. We expected --urlReplacementPatterns was doing that but in fact, it is not the case. Actually, with urlReplacementPatterns the url names are the same, but there is still the error : "oops this base build is missing a run for this URL".

What could be the source of this error ? Maybe we missed something ? Thank you !

[patrickhulce]

Actually, with urlReplacementPatterns the url names are the same

Then there is a bug in your replacement pattern. What exact replacement pattern strings are you using?

[Ovicakov]

Then there is a bug in your replacement pattern

It is not the goal of urlReplacementPatterns to get similar url names to be capable of compare it ? I am a bit confused with this feature.

And here, our exact pattern : s/([a-zA-Z0-9-_]+).domain/lhci.domain/g, which works with https://repl.it/@hamstu/test-replaceUrlPatterns#index.js.

[patrickhulce]

It is not the goal of urlReplacementPatterns to get similar url names to be capable of compare it ? I am a bit confused with this feature.

Ah apologies, I misread your comment 😊 They are the same. Are they 100% the same, down to the query string too?

The only known source of "oops this base build is missing a run for this URL" is if the base build does not have an exact match for the URL string. Note that the base build must have been uploaded after the urlReplacementPatterns change was made and it is not retroactive for previous base builds.

[amalitsky]

Have related question regarding Network Request and Network Round Trip Times sections of report comparison view.

Looks like urlReplacementPatterns doesn't have the expected effect when comparing lists of requests. I'm constantly getting smth like +36 / -36 statistics where every network request is considered brand new even when url path and query remain the same.

Domains are different since one is staging while other one is production.

I.e. first and last lines here:

Network Round Trip section is even more puzzling. Everything is supposedly removed and added here even when URLs are the same:

Server Backend Latencies list behaves the same way.

Am I missing the point of those views and diff statistic?

This might be the code related to this functionality: https://github.com/GoogleChrome/lighthouse-ci/blob/main/packages/server/src/ui/routes/build-view/lhr-comparison.jsx#L57-L114