Log actual data loaded.

[adoxa]

Just want to give you something to look at before I go to bed. :)

After creating the stream listener, I discovered it didn't work with local files (or things like about:). "Life-of-request info" (lori) did, and it just used a progress listener. Turns out adding the listener to the browser works just fine. Atm it still uses onDataAvailable, but I think switching to onProgressChange would be better (losing the difference between network & cache, but data is uncompressed size, whereas progress is compressed). Alternatively, keep 'em all, creating a replacement for the Web Developer/FireBug/ySlow info (although I've only looked at the first of those).

[kustodian]

I will try it out and tell you how it works. We can discussion a little bit later what data is best to be displayed.

Also regarding the about:* pages, I already made an issue #15 to remove ESB from those pages, since ESB cannot provide any useful information on those pages.

[sipertruk]

So for an exhaustive report of progress we need to attach the listener to the browser. I wish the mozilla doc stated the fact. Switching to onProgressChange would be better indeed , I can't see any purpose of showing the cache data. For knowing your cache is working maybe..

[kustodian]

If you ask me, I would exclude the cached data if possible. If I would like to see the real size and the real speed of downloading of the page, I would refresh it with CTRL+F5, or disable cache.

[adoxa]

I'm a little confused - what do you want to do with cached data? onProgressChange gets the size of files, cached or not (so if the site downloads 10K and gets 20K from cache, the total will be 30K). onDataAvailable can distinguish, so it's possible to show both the 10K and the 20K separately, although being data size, this is the uncompressed size, not the compressed size that was downloaded; for that you could use contentLength or onProgressChange (since contentLength may be -1). For myself, I'd like to know what was actually downloaded, which of course is the more involved one.

[adoxa]

Another difference between progress & data is that progress stops once the document is loaded, but data keeps going, so it would be possible to keep updating the loaded value.

[sipertruk]

Maybe it's useless but I found that page showing the same observer way with a different name of service : https://developer.mozilla.org/en-US/docs/Monitoring_HTTP_activity.

The network-monitor in the web console use that : http://hg.mozilla.org/mozilla-central/file/a18ff2303b09/toolkit/devtools/webconsole/network-monitor.js

Perhaps it's the same stuff after all..

[kustodian]

I thought about this and displaying compressed data for downloads is not needed. When you think about it, the only thing that matters is how much data was transferred from the server to the browser. It doesn't matter if it's transferred faster by using compression or not. Even the download/torrent clients which support compression, don't display how much compressed data was downloaded, but how much actual data was download. Because of that I would say that we should ignore compressed data and count actual data. As far as I can see that will be even easier to implement, which is a plus.

What we should try to do is to display how much data was downloaded and the download speed without counting cache. I would suggest that it would be best to have 3 fields for this:

• Loaded: uncompressed downloaded data
• Cached: how much data was read from cache
• Speed: Loaded/s

Maybe we should rename Loaded to something else (e.g. Transferred)?

[adoxa]

But how much was transferred from the server to the browser is the compressed value. For example, it appears AlterVista automatically compresses, so my 14K index is a 3.5K download and that's the value I want to see.

[kustodian]

I understand your concern, but if it's hard to implement, it's not that big of a problem. Loaded means how much data was loaded by the browser and if the browser downloaded 3.5K and it uncompressed it to 14K, 14K is the actual loaded data. Isn't it?

[adoxa]

Right, but I'm more interested in the downloaded value. I'm in the process of incorporating the other observer mentioned by sipertruk, which also includes socket data received & sent, and displaying all the values (as well as logging everything).

[kustodian]

If you manage to do that, what we could do is use that for Speed and for Loaded we could display it like Loaded: 3.5KB (14KB) or something like that. The tooltip could explain what is each value.

[adoxa]

It depends on the cache. Excluding cache altogether is easy, since either of the http observers allow that (the observer from above gets downloaded, the stream listener gets loaded). But then, without cache, it's not data loaded, only downloaded. The progress listener will get both cached and downloaded data; the stream listener has separate topics for cached and loaded. Then there's also the headers (both sent & received). I think they all get those, though (except maybe progress, sometimes). I would like to use the socket data to get "raw" downloads, but it looks like the value accumulates, and there's no explicit reset. But that doesn't really matter. Anyway, I'll remove the sockets and push what I have soon...

[adoxa]

This now shows most things: P is the progress value (probably what you originally intended); D is the data from the stream listener: "total(network+cached)" (probably what you want now); H is the data from the http observer: "total(headers+data)" (or maybe this); R is the request headers sent from the observer. In the log, Data is from the stream listener, C for cached & N for network; Http is from the observer, R for the request header (with the first line); H for the response header (with the first line) and C for the completed response (the data); the numbers are the size of the current data followed by the total size of data so far.

[sipertruk]

Putting all the method altogether is a good idea, now let's choose :) I haven't looked enough everything but at first sight the stream listener looks best of course. The value seems reliable enough.

I noticed in the logs that on the first OnDataAvailable after reloading a page this.browser.esbValues.loaded is not at 0. So the total count misses the first increment.

On reloading some pages images are not counted anywhere, neither in cache nor network. Then the cache value loses a part of its utility.

[adoxa]

I was thinking of turning them all into elements and letting the user mix & match:

• Data downloaded (P, this would be my default; possibly with a cached suboption);
• Data loaded (D, with a suboption for cached - total/cached; I guess this is what you & kustodian would prefer as default);
• Data received (H);
• Data sent (R).

Of course, that raises the issue of just what is meant by "speed", then. If only one of the above is chosen, that would be it; if none, speed would automatically hide, too. If multiple, well, I guess that would require all the corresponding speed elements. Alternatively, just fix it to received (or received+sent).

The latest version does not use loaded, it got renamed to loadedProgress. Assuming that what's you meant, I haven't noticed it, got an example site?

Yeah, I wondered about images not showing up. Turns out they're "Not Modified", so neither onDataAvailable nor onProgressChange are called and Http (C) just gets 0; onStart is called, so request.contentLength might work for some, but not others (either because it's unknown, or it's the P value, not the D, as some images use [such as an SVG on developer.mozilla.org]).

[sipertruk]

For speed I would prefer to use data received so I am close to the real speed for the online data transfert.

What I said was very unclear and I was wrong. I meant this.browser.esbValues.loadedCached and this.browser.esbValues.loadedNetwork. They're set at 0 at onStateChange for STATE_START so the first Data line of the log after a reload shouldn't be like (I used this page, issue 17) : "Data (N) : 89 1856756" "text/e... but "Data (N) : 89 89" "text/e... Now I realize what I meant is impossible, it's just the former listener's log, I guess this page is doing something when we hit reload.

[sipertruk]

The listener keep on logging scripted and plug-in activities after the initial loading is finished. Switching tab and back will update values. If there's no more Loaded value update intended after a onStateChange(STATE_STOP) it could be disabled.

(Or detect that the loading is finished in the listener and update the value at a time interval ? Not sure if that's were we want to go.)

[kustodian]

I'm not sure if updating ESB fields once the loading is finished is the right way to go about this.

[adoxa]

Actually, again, I'm the opposite. :smile: I was thinking of another option (or not) to keep track of data after the initial load, but as a separate value - for example, data received like 32 KB/5 KB means it received 32 K during the initial load, and then subsequently received 5 K (and related to that, once things get sorted I'll add a "Units" option group, so you could choose 5,000 B [using toLocaleString], 5 KB [decimal prefix], or 4.88 KiB [binary prefix, default]).

As for the initial data, hm, yes, but it seems to get reset again just afterwards. This is what I got:

NS_BINDING_ABORTED: Component returned failure code: 0x804b0002 (NS_BINDING_ABORTED) [nsIStreamListener.onStopRequest] extendedstatusbar.js:1008
The connection to https://live.github.com/_sockets/NDcwMjM4OjZkNjM2OGRmYmVjMGMwMzFmZTg5MzNmMWEzOGRmMTI3OjU4MmMxNjI2YmFkNjA5ODc2ZTNkNWI1ZTEyMTQyZGY0MWNjYWVlNmVkNjJlYzNkYmQzYzc0NWRlMDEyNTk0MDc=--3add3693685ee99f0d1299a8fbd47be693ffeb21 was interrupted while the page was loading. github-e4591d88d498b849f06ae49e0a07d6c6401ac7c7.js:3
"Start:" "https://github.com/kustodian/extended-statusbar/pull/17#issuecomment-48757738" extendedstatusbar.js:591
"Data end :     -1       0" "https://live.github.com/_sockets/NDcwMjM4OjZkNjM2OGRmYmVjMGMwMzFmZTg5MzNmMWEzOGRmMTI3OjU4MmMxNjI2YmFkNjA5ODc2ZTNkNWI1ZTEyMTQyZGY0MWNjYWVlNmVkNjJlYzNkYmQzYzc0NWRlMDEyNTk0MDc=--3add3693685ee99f0d1299a8fbd47be693ffeb21" extendedstatusbar.js:1005
"Http (C) :     12      12" extendedstatusbar.js:943
"Http (R) :    995     995" "GET /kustodian/extended-statusbar/pull/17 HTTP/1.1
" extendedstatusbar.js:943
"Data (N) :     89  725335" "<no type>" "https://live.github.com/_sockets/NDcwMjM4OjZkNjM2OGRmYmVjMGMwMzFmZTg5MzNmMWEzOGRmMTI3OjA4Y2JmZjllMzJjZDIzM2NkNzI2ZGVjMjY0MjY3ZmJlYWE4Zjk5MzI3Yzk3NTJmNmEyNWVjOTBkNDNhYjFlNTE=--40521e46c1bfc97408a86b296539b8f73eb064a1" extendedstatusbar.js:989
"Http (H) :   1651    1651" "HTTP/1.1 200 OK
" extendedstatusbar.js:943
"Data beg :     -1       0" "text/html" "https://github.com/kustodian/extended-statusbar/pull/17#issuecomment-48757738" extendedstatusbar.js:996
"Progress :   1364    1364" "text/html" "https://github.com/kustodian/extended-statusbar/pull/17#issuecomment-48757738" extendedstatusbar.js:827
"Data (N) :   3174    3174" "text/html" "https://github.com/kustodian/extended-statusbar/pull/17#issuecomment-48757738" extendedstatusbar.js:989

I guess there's no problem with having the additional elements? I'll call it "Data retrieved" rather than "downloaded" to better indicate that it includes cached data, and I'll hold off actually coding it until the order branch gets merged.

[adoxa]

I've decided against including cached data, after all, and have also stopped separating the headers and content, so the data display is a little more compact, now. Logging has been removed, but there's now a "Details" context menu item to display a window with everything - still yet to do sorting, resizable columns and persisting the "raw" states.

[kustodian]

So what exactly is everything now? :D

I'm not sure what R/L mean. I guess R is retrieved which I guess is compressed data downloaded. I don't know what L (loaded) means, since it is always 0 for me. The up/down arrows are uploaded and downloaded data (uncompressed)?

Do we really want to have a Details window? Most of that information is only useful for developers and most of it is accessible one way or another in the developer tools or Firebug. I'm not sure if we really need that much information. If you really want to display that much information, I would rather make it a tooltip for the Loaded: label and make it display something like top 10 most largest files, but I'm still not sure if we need that much information.

[adoxa]

Expand the commit message...

Okay, you're right, I've simplified things - just used the browser progress listener, removed everything else.

[adoxa]

Added the browser progress listener directly, skipping all the testing.