Steep performance degradation in RAW mode

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Use serve + RAW mode
2. Use modules with a large number of files (~300 per module)

Loading the average closure file takes about 160ms per file on a real fast 
desktop computer.
It used to be a lot less than that, 10x less or more.
On some computers it takes minutes to load the application in RAW mode, it used 
to take seconds.

Not sure exactly when the problem was introduced, I skipped the last release 
because of the UTF-8 encoding issue reported earlier, but it is one of the last 
2 official releases.

Tested only on Windows so far, all major browsers.

Original issue reported on code.google.com by stefand...@gmail.com on 27 Nov 2011 at 4:54

[GoogleCodeExporter]

Hi, on one hand, this is not good, but on the other hand, I've heard anecdotal 
reports of performance degradation, so hopefully this will help identify the 
problem!

Can you be more explicit about which releases from the list you have tried that 
work/do not work?

http://code.google.com/p/plovr/downloads/list

Also, are you using the built-in Closure Library or your own (via the 
http://plovr.com/options.html#closure-library option)?

Any other details you can provide may help diagnose the problem.

Thanks,
Michael

Original comment by bolinf...@gmail.com on 27 Nov 2011 at 10:08

[GoogleCodeExporter]

Will test on Mac too to see if the problem is Windows specific.
The code showing the problem is the latest source code, including the
November release.
The code without the problem is the April release, I skipped the September
release, so I do not know if that exhibited the problem or not.
If it helps, I can test using the September release too, just let me know.

Original comment by stefand...@gmail.com on 28 Nov 2011 at 7:27

[GoogleCodeExporter]

Yes, please let me know if the September release exhibits the issue or not. It 
will dramatically reduce the number of changes that I will need to consider.

Original comment by bolinf...@gmail.com on 28 Nov 2011 at 3:39

[GoogleCodeExporter]

The September release does not have the latency problem.
For the September release most http script requests are resolved in 5-6
msec.
The November release takes 400-500 msec per request, both on Mac and
Windows.
Measured on the same machine, a Mac Pro / Quad Core Xeon 2.8 GHz.
The 160msec response time per request I reported earlier was for a faster
machine.
Let me know if I can help with testing or diagnostics.

Original comment by stefand...@gmail.com on 28 Nov 2011 at 4:59

[GoogleCodeExporter]

Aha! I think I have a hypothesis for the cause of the regression:

http://code.google.com/p/plovr/source/detail?r=248ec0556f87

The call to getAllDependencies() on every request to /input is not cheap!

Original comment by mbo...@gmail.com on 29 Nov 2011 at 4:09

[GoogleCodeExporter]

Hi, I just committed 
http://code.google.com/p/plovr/source/detail?r=38639496546895645ba3000035e1df01c
5b30556 which I thought might help fix this issue, but testing on my own code, 
there does not seem to be a significant difference. However, if you notice, 
this code includes a println in InputFileHandler. When I run this on my project 
(which has 4 modules which contain ~150 JS files altogether), the get() method 
in InputFileHandler claims to take <10ms for most inputs (according to what is 
printed out), though Google Chrome claims that these files take anywhere from 
50-1000ms to load (I must be reading the Network tab wrong?).

This is on a Lenovo Thinkpad running Ubuntu 10.10. I noticed that files in 
modules other than the root module claim to load faster, though I don't know 
whether that's contention for the network when the page initially loads or 
what. (Or it could be the use of document.write() for the initial script tags?)

Feel free to build from head, or please try the precompiled binary at 
http://bolinfest.com/plovr/plovr-386394965468.jar which contains the println. 
I'm curious what sorts of numbers print out for that debugging message in 
InputFileHandler. Specifically, are you seeing the same thing as me: numbers 
<10ms printed to the command line from plovr, but much larger numbers in Chrome 
Developer Tools / Firebug?

Any insight would be helpful!
--Michael

PS Unfortunately, the largest codebase I have for testing this sort of stuff 
uses the newer features of Soy (such as {let} and array/map literals), so I 
cannot use the September 2011 release 
(http://code.google.com/p/plovr/downloads/detail?name=plovr-96feca4d303b.jar) 
to build my code as a point of comparison.

Original comment by mbo...@gmail.com on 29 Nov 2011 at 5:57

[GoogleCodeExporter]

Actually, on my code at least, most of the time is spent "blocking," i.e., 
waiting for other requests to finish since the user agent has a max connection 
limit per hostname. So blocking is variable, but the other sources of latency 
seem to be fairly consistent across requests:

Sending is generally 0ms
Waiting is almost always 3ms or less
Receiving is generally around 40ms

What is the timing breakdown for your requests? I realize that 40ms is nowhere 
near 160ms or 16ms.

Out of curiosity, does your desktop have a solid state drive or a more 
traditional hard disk? If the latter, then perhaps my recent fix would help you 
more than it has helped me.

Original comment by bolinf...@gmail.com on 29 Nov 2011 at 6:21

[GoogleCodeExporter]

Seems like it solved it for our codebase
(loading closure + ~120 of our own js files takes ~2sec on an early 2011 MBP 
without an ssd)

Original comment by Tobias.Raeder on 29 Nov 2011 at 4:11

[GoogleCodeExporter]

Thanks a lot Michael, I can confirm the problem is resolved.
Sorry I did not get to test it sooner.

The latency reported on a laptop varies quite a bit, but most requests are
resolved in under 10 msec.
This is using a standard HDD and the stock Mac Book Pro.

Original comment by stefand...@gmail.com on 30 Nov 2011 at 11:17

[GoogleCodeExporter]

Excellent! As soon as I get this ETag stuff squared away 
(https://groups.google.com/forum/#!topic/plovr/yWiGfVG-hq4 
http://code.google.com/p/chromium/issues/detail?id=105824), I will put together 
a new release.

Original comment by mbo...@gmail.com on 2 Dec 2011 at 11:01

[GoogleCodeExporter]

Original comment by bolinf...@gmail.com on 2 Dec 2011 at 11:01

• Changed state: Fixed