Transfer sizes seem incorrect

[arnaudvalle]

Original issue (https://github.com/sitespeedio/sitespeed.io/issues/1467)

==========

Hello there,

I've been trying to run sitespeed on a page behind login (I believe that's all good now) but I am seeing different results (on requests numbers for example) between doing it "manually" in my Chrome browser (and looking in the Network tab) and when doing it via the sitespeed command.

I've setup a test page with a login / password for you to have a look here: http://lumsites-sandbox.appspot.com/a/lumapps/home/sitespeed-homepage-test-private (see credentials in the preScript login script I use).

Here's the command I use:

docker-compose run sitespeed.io --preScript sitespeed-login.js http://lumsites-sandbox.appspot.com/a/lumapps/home/sitespeed-homepage-test-private -n 3 -b chrome --browsertime.chrome.args no-sandbox --browsertime.chrome.args disk-cache-dir=/dev/null --browsertime.chrome.args disk-cache-size=1 --browsertime.chrome.args media-cache-size=1 --browsertime.pageCompleteCheck 'return (function() {try { return (Date.now() - window.performance.timing.loadEventEnd) > 15000;} catch(e) {} return true;})()' --graphite.host=graphite --summary-detail

These are the versions it uses on my machine:

Versions OS: linux 4.9.8-moby nodejs: v6.9.1 sitespeed.io: 4.4.2 browsertime: 1.0.0-beta.25 coach: 0.31.0

Which gives me the following results:

Total requests 31 Image requests 3 CSS requests 4 Javascript requests 7 Font requests 0

When I do the same process manually (logging in, then going to the test page and looking up the numbers in Chrome > Network with cache disabled) I get this:

Total requests 51 Image requests 4 CSS requests 4 Javascript requests 21 Font requests 5

I suspect it has something to do with the 'warm cache' due to the login page visited during the preScript step, but if that's the case I can't see how to clear that cache completely before running the tests on my target page.

Thanks in advance for your help or any hint on what I may have done wrong.

For info, I get different results depending on the version of the image of sitespeed I specify in my docker-compose file. The 'Transfer size' values look especially different.

==========

sitespeed 4.0.3

Score / Metric Median ✗ Overall score 67 ✗ Performance score 65 ✗ Accessibility score 65 ✗ Best Practice score 76 √ Fast Render advice 95 ! Avoid scaling images advice 90 √ Compress assets advice 100 ! Optimal CSS size advice 90 ✗ Total size (transfer) 1.9 MB √ Image size (transfer) 642.6 KB ✗ Javascript size (transfer) 1.1 MB ✗ CSS size (transfer) 97.0 KB Total requests 43 Image requests 13 CSS requests 1 Javascript requests 11 Font requests 0 200 responses 43 Domains per page 7 Cache time 10 minutes Time since last modification -1 second RUM Speed Index 5121 First Paint 4840 ms Backend Time 1530 ms Frontend Time 7155 ms

==========

sitespeed 4.5.1

Score / Metric Median

✗ Overall score 69 ✗ Performance score 68 ✗ Accessibility score 65 ✗ Best Practice score 76 √ Fast Render advice 95 ! Avoid scaling images advice 90 ✗ Compress assets advice 80 ! Optimal CSS size advice 90 √ Total size (transfer) 987.1 KB √ Image size (transfer) 664.9 KB ✗ Javascript size (transfer) 126.5 KB ✗ CSS size (transfer) 113.3 KB Total requests 46 Image requests 17 CSS requests 4 Javascript requests 7 Font requests 0 200 responses 46 Domains per page 8 Cache time 45 minutes Time since last modification -1 second RUM Speed Index 4440 First Paint 2015 ms Backend Time 1389 ms Frontend Time 6617 ms

[mscrivo]

We've got the exact same issue.

[sri85]

I am facing the same issue where i am seeing the calls to get the css and js are missing from the report.

[tobli]

Is there a url where it happens consistently? That would be helpful for debugging.

[sri85]

Unfortunately the app I am testing is private , but in the issue itself i see that there is a test URL.

In my case what i see is that the calls to get the CSS and JS are missing , rest all are in tact.

[soulgalore]

@sri85 and you also hit another URL first as in the original issue? Do you see them if you use Firefox?

[sri85]

Yes i use prescript to login into my application . Yes this issue is seen in firefox as well.

[sri85]

@soulgalore I am attaching the screenshot to highlight the difference in data i am seeing from the chrome network tab and HAR file generated by sitespeed. Just to give you a bit more context what I do in my preScript Step 1: Go to login page the URL which looks https://<myaswesome>/a/appname/accountname/login after i log in i land in a page the url of which looks something like https:///a/mam/a/appname/accountname/#!/browse This is what i see in the chrome network tab when i navigate to https:///a/mam/a/appname/accountname/#!/browse manually

This is what sitespeed shows me in the report

As you can see the HAR entries in sitespeed report have skipped the JS and CSS file .

[soulgalore]

But are you sure devtools is right? :) Asking since you said you have the same behavior if you run with Firefox.

Just making sure that you have the cache turned on in devtools or are you trying to disable the cache as in the first comment in this issue?

[sri85]

But are you sure devtools is right? :) Asking since you said you have the same behavior if you run with Firefox. 😃 Good one

Just making sure that you have the cache turned on in devtools or are you trying to disable the cache as in the first comment in this issue?

I have disabled the cache in devtools

[soulgalore]

Ok, so it works as it should then? :) If you do a prescript with login, we have the same cache for the next access (as long as you have some kind of cache headers). Between runs we empty the cache.

[sri85]

@soulgalore The css and js that are loaded in landing page are loaded only after the users have logged in , so in theory they are not cached when the user logs in for the first time.

So to make it more clear

Iteration 1:

1. Login
2. Landing page(Loads all the CSS and JS)
3. Refresh the page / log out and log in again
4. This time since the user has already JS/CSS downloaded when they logged in for the first time so this time it will be loaded from cache.

In my scenario I am doing only Step 1 and Step 2 and that too i am running only for 1 iteration so I will be expect this to be shown in HAR as its not cached and i am accessing it for the first time.

[sri85]

we have the same cache for the next access (as long as you have some kind of cache headers) This was magic word that rung a bell and i added -browsertime.chrome.args disk-cache-dir=/dev/null --browsertime.chrome.args disk-cache-size=1 --browsertime.chrome.args media-cache-size=1 and now I am able to see all the requests

@soulgalore you are a legend! Thanks for all the help

[soulgalore]

@sri85 aha ok hehe, good then :)

[mscrivo]

Adding those args fixed the issue for us as well, thanks @sri85 and @soulgalore

[knaos]

Hey, guys. I had the same issue as @sri85 and @mscrivo . It got fixed by adding the chrome args flags. But I don't understand why this issue was happening? Can you please explain? When using a preScript, Browsertime was not showing all the network calls as if it was starting to record from a bit later OR the cached items did not get populated into the results.

[soulgalore]

@knaos so what are you seeing, people has used this thread to talk about different things :)

If you disable the cache (with the CLI params), no requests will be cached in the browser and you will get them in the HAR file. If you run as default and you on your login page has the same assets as on your next page (and you have some caching headers), these will be cached in the browser and you will not get them in the HAR file.

[knaos]

@soulgalore First, I want to say that this tool-chain is amazing! Thank you! I was not able to see the cached files in the HAR. Is this by design and why? In Chrome, you still see the cached files in the HAR but it says it was loaded from cache, either disk or memory.

[knaos]

If you disable the cache (with the CLI params), no requests will be cached in the browser and you will get them in the HAR file.

@soulgalore Which CLI param is that?

[soulgalore]

Thanks for the kind words @knaos !

Yeah what is actually right I'm not sure, need to check the HAR spec. Do you remember @tobli what we thought about it for Chrome? I know for Firefox I added so we strip cached requests but that was because it was missing cache information, so you could not understand if the response was cached or not (I think the response code was 200). Only the timings indicated they where cache hits. But that can have changed in latest Firefox, let me check that for the coming Browsertime 3.0.

Cli params, the ones people talked earlier in the thread, by disabling the cache. Best is to look for official Chrome documentation to check what's the correct way to do it.

[syed1988]

Hello, it seems like I am running into similar issues. I ran sitespeed against google.com and the summery reports that CSS size (transfer) as 0 byte. But look at the network tab of chrome developer's tool i'm seeing a couple of MBs of CSS.

sitespeed report:

[syed1988]

Here is the command i used to run sitespeed: docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:9.3.4 https://www.google.com --chrome.args disk-cache-dir=null --chrome.args disk-cache-size=1 --chrome.args media-cache-size=1 Also seeing similar discrepancy with javascript files.

[soulgalore]

Hi @syed1988 thanks, I think it is something else? I tried with Firefox and I get the same result (0 bytes). Looking in network tab in FF I don't see any css request either.

Best Peter

[syed1988]

@soulgalore any ideas on why the two browser reporting two different metrics. I'm seeing similar thing with Javascript Size metrics for one of the pages of our site. All things being equal, Chrome reports Javascript Size to be 952 b, whereas, Firefox reports it to be 3.6 MB.