document the differences between Flash/Java on various platforms

[GoogleCodeExporter]

The Flash and Java clients produce markedly different performance results, 
depending on the specifics of the OS/Flash version/Browser. It would be good to 
document this somewhere. Even better if we can include the "known good" or 
"known bad" list in the web UI somewhere. This would be especially useful at 
speeds in the 100s of Mbps.

e.g. in testing just now, on OS X with FF, I saw a roughly 5-10Mbps performance 
degradation on a 100M limited test host.

Original issue reported on code.google.com by AaronMat...@gmail.com on 13 Mar 2015 at 2:50

[GoogleCodeExporter]

Relatedly, it'd be good to test these things in reproducible network 
environments (e.g. on a clean 1Gbps network, what do we see?. On a network with 
1% loss, what do we see? On a machine with 30ms RTT, what do we see?)

Original comment by AaronMat...@gmail.com on 13 Mar 2015 at 2:55

[GoogleCodeExporter]

Would following test plan be good?
Measure performance results depending on:
 - network type (10mb/100mb/1gb),
 - packet loss (e.g. starting from 1% to 10% with step=1%)
 - delay value: (e.g. 10ms/30ms/100ms/300ms/500ms/1s/2s)
mixing these things so test would look as follows:
1. Network: 100mb, packet loss: 1%, RTT: 30ms
2. Network: 100mb, packet loss: 2%, RTT: 100ms
etc

I can test mentioned things on my machine with Linux system and Firefox/Chrome 
browsers using newest flash version (on linux it is quite old anyway: 
11.2.202.451) and java 7u76.

What do you think about such tests? Is it what you had in mind or should it be 
performed another way (any additional notes would be then very appreciated)? 

Original comment by skost...@soldevelo.com on 19 Mar 2015 at 10:23

[GoogleCodeExporter]

Original comment by skost...@soldevelo.com on 19 Mar 2015 at 12:31

• Changed state: Accepted

[GoogleCodeExporter]

That's pretty much what I was thinking. You could use netem on a linux host 
acting as a router to create a reproducible network setup with various 
performance characteristics.

As to the end host, do you have access to a Windows host or a Mac? Those are 
the stats that are probably more interesting to the user base.

Original comment by AaronMat...@gmail.com on 19 Mar 2015 at 5:15

[GoogleCodeExporter]

Unfortunately I don't have access to device with Mac system but I can test it 
using Windows instead then. 

Original comment by skost...@soldevelo.com on 20 Mar 2015 at 8:36

[GoogleCodeExporter]

Attaching test results. Below short information how tests were performed:
System: Windows 8.1
Browsers: Firefox 35.0.1, Chrome 41.0.2272.101
Java version: 8u40
Flash version: 17.0.0.134

In attached file there are 3 sections: first is with results from Firefox and 
comparing flash/java performance, second the same for Chrome and third is 
comparison of results obtained in these two browsers. All results are average 
number of 5 tests done in the same environment. Also I have not tested in 
detail for 10mb network as it's rather obsolete today and also not tested 
further for remaining delay/packet loss values as download values in results 
for such configuration were too low to mention (few or below 1 mb/s). 
Configuration was changed using netem on Linux machine which performed as 
web10g server using newest code from trunk r1217 (Ubuntu 14.04 LTS with 3.9.1 
kernel and 2.0.7 userland) which I directly connected with second machine with 
Windows.

Do these results satisfy requirements mentioned in ticket description or 
something should be added/changed ?

Original comment by skost...@soldevelo.com on 23 Mar 2015 at 9:44

Attachments:

• ndt-test-results.ods

[GoogleCodeExporter]

Could you include the standard deviation of the 5 tests? These numbers are all 
over the place...

Original comment by AaronMat...@gmail.com on 23 Mar 2015 at 2:30

[GoogleCodeExporter]

It turned out that for some configurations deviation was too high so I 
performed additional tests and now for all configurations its minimum 10 
(instead of 5), I have performed as many attempts as were needed to reduce 
relative standard deviation below 10%. Results show that the biggest difference 
is between flash version integrated with Chrome and the one which comes from 
Adobe (~50%). Also upload values differences for 0% packet loss and 0ms delay 
is quite high. All details are visible in spreadsheet where you can define 
thresholds and see cells automatically being colored indicating which ones 
match defined requirements. Is that ok now or something should be 
added/changed? 

Original comment by skost...@soldevelo.com on 25 Mar 2015 at 2:44

Attachments:

• ndt-test-results.ods

[GoogleCodeExporter]

This all looks good. Thanks! Would it be possible to get some Linux numbers too?

Original comment by AaronMat...@gmail.com on 25 Mar 2015 at 3:25

[GoogleCodeExporter]

New spreadsheet with Linux system results added (tested on Java and Flash 
plugins for Firefox and Flash only for Chrome as Java support has been removed 
since version 35). Results show that there is huge difference between very old 
version of Flash in Linux's Firefox (11.x) when compared to results from newest 
one from Windows (17.x). Also results overall differ quite a lot between both 
systems. Are these numbers enough or is something missing?

Original comment by skost...@soldevelo.com on 27 Mar 2015 at 1:43

Attachments:

• ndt-test-results.ods

[GoogleCodeExporter]

After some investigation it turned out that for 100mb/s network speed results 
between Windows and Linux differed so much because of duplex config, for Linux 
it was full-duplex while for Windows half. I then configured Windows to work in 
full-duplex mode too and attached file contains updated results for 100mb/s for 
Windows. 

Original comment by skost...@soldevelo.com on 30 Mar 2015 at 1:51

Attachments:

• ndt-test-results.ods

[GoogleCodeExporter]

These look great. Thanks! Would it be possible to include some warnings in the 
web client if Linux + Flash, or Chrome + Flash get used?

Original comment by AaronMat...@gmail.com on 31 Mar 2015 at 12:12

[GoogleCodeExporter]

I've added proper changes on web-client-browser-os-warnings branch. If they 
look ok for you I could merge them into trunk. 

Original comment by skost...@soldevelo.com on 31 Mar 2015 at 1:34

[GoogleCodeExporter]

Looks good. Closing.

Original comment by AaronMat...@gmail.com on 31 Mar 2015 at 3:02

• Changed state: Done