Sitespeed.io runs are faster

[bharat129]

Hi @soulgalore ,

We are running sitespeed.io using docker-compose file (Unix VM) inside organization's firewall. Browser : Chrome Connectivity Cable

I have not made any changes in default configuration. I am constantly getting largest pagetiming metric : Last Visual Change around 8 sec.

But when I run the same test on public instance of WPT, it gives 13-15 second of fully loaded time.

Its now very difficult to explain the huge difference in timings. I know you have mentioned not to compare tools on https://www.sitespeed.io/documentation/sitespeed.io/best-practice/#difference-in-metrics-between-webpagetest-and-sitespeedio

1. Is it possible the sitespeed.io run on unix VM is not using the connectivity as cable since I can very well see from the HAR that resources are downloaded faster in sitespeed run than public WPT ?

2. What metric we should trust which will provide time similar to WPT's fully loaded time?

Docker Command: docker-compose -f /www/sitespeed/docker-compose.yml run -v /www/sitespeed:/sitespeed.io sitespeed.io https://www.xxxx.ca/ --browsertime.browser=chrome --browsertime.iterations=3 --speedIndex --browsertime.timeouts.script=80000 --browsertime.proxy.http=proxy.xxx.com:80 --browsertime.proxy.https=proxy.xxx.com:80 --browsertime.connectivity.profile=cable --graphite.host=graphite

Regards, Bharat

[beenanner]

Hey @bharat129 it could be possibly related to this?

We are running sitespeed.io using docker-compose file (Unix VM) inside organization's firewall. ... But when I run the same test on public instance of WPT, it gives 13-15 second of fully loaded time.

Which would need to also go through your firewall. I'm not sure what specs the public WPT instances run on, but you should also make sure your Unix VM is spec similar if you are trying to compare. Though I wouldn't recommend it. ;-) You could also look into setting up a private WPT instance to compare public WPT vs private WPT.

https://github.com/WPO-Foundation/webpagetest-docs/blob/master/user/Private%20Instances/README.md

[bharat129]

@beenanner , you are correct. That's why there is no time in DNS and other metric SSL, TTFB, Content download are taking less than half from outside.

Can you please share the details for below: What metric we should trust which will provide time similar to WPT's fully loaded time?

[soulgalore]

Hey @bharat129 how do you set up your connectivity, using Docker networks?

Best Peter

[bharat129]

@soulgalore , we have not setup any connectivity using Docker networks. How to do it?

[cgoldberg]

I'd bet network connectivity/location/topology/conditions accounts for much of the observed difference.

[bharat129]

@cgoldberg , you mean to say if I change the connectivity, the results will be the same? Since still my sitespeed server is inside the fiewall?

[cgoldberg]

you can't really make the connectivity identical, so I don't see the point of the comparison. Consider that WPT and your local container are located in different places and are taking different networks across the internet to the system under test.. If you were doing a comparison on a local (controlled) network under idle conditions with identical test environments, it might be a valid comparison.

[soulgalore]

@bharat129 checkout https://www.sitespeed.io/documentation/sitespeed.io/connectivity/#docker-networks

If you don't setup any connectivity, you will have a hard time finding regressions and it will vary within the tool.

[soulgalore]

Closing this because of inactivity, let me know if you need more help!