document the differences between Flash/Java on various platforms

[aaronmbr]

Original issue 175 created by aaronmbr on 2015-03-13T14:50:55.000Z:

The Flash and Java clients produce markedly different performance results, depending on the specifics of the OS/Flash version/Browser. It would be good to document this somewhere. Even better if we can include the "known good" or "known bad" list in the web UI somewhere. This would be especially useful at speeds in the 100s of Mbps.

e.g. in testing just now, on OS X with FF, I saw a roughly 5-10Mbps performance degradation on a 100M limited test host.

[aaronmbr]

Comment #1 originally posted by aaronmbr on 2015-03-13T14:55:47.000Z:

Relatedly, it'd be good to test these things in reproducible network environments (e.g. on a clean 1Gbps network, what do we see?. On a network with 1% loss, what do we see? On a machine with 30ms RTT, what do we see?)

[aaronmbr]

Comment #2 originally posted by aaronmbr on 2015-03-19T10:23:04.000Z:

Would following test plan be good? Measure performance results depending on:

• network type (10mb/100mb/1gb),
• packet loss (e.g. starting from 1% to 10% with step=1%)
• delay value: (e.g. 10ms/30ms/100ms/300ms/500ms/1s/2s) mixing these things so test would look as follows:
  1. Network: 100mb, packet loss: 1%, RTT: 30ms
  2. Network: 100mb, packet loss: 2%, RTT: 100ms etc

I can test mentioned things on my machine with Linux system and Firefox/Chrome browsers using newest flash version (on linux it is quite old anyway: 11.2.202.451) and java 7u76.

What do you think about such tests? Is it what you had in mind or should it be performed another way (any additional notes would be then very appreciated)?

[aaronmbr]

Comment #3 originally posted by aaronmbr on 2015-03-19T12:31:04.000Z:

<empty>

[aaronmbr]

Comment #4 originally posted by aaronmbr on 2015-03-19T17:15:49.000Z:

That's pretty much what I was thinking. You could use netem on a linux host acting as a router to create a reproducible network setup with various performance characteristics.

As to the end host, do you have access to a Windows host or a Mac? Those are the stats that are probably more interesting to the user base.

[aaronmbr]

Comment #5 originally posted by aaronmbr on 2015-03-20T08:36:18.000Z:

Unfortunately I don't have access to device with Mac system but I can test it using Windows instead then.

[aaronmbr]

Comment #6 originally posted by aaronmbr on 2015-03-23T09:44:54.000Z:

Attaching test results. Below short information how tests were performed: System: Windows 8.1 Browsers: Firefox 35.0.1, Chrome 41.0.2272.101 Java version: 8u40 Flash version: 17.0.0.134

In attached file there are 3 sections: first is with results from Firefox and comparing flash/java performance, second the same for Chrome and third is comparison of results obtained in these two browsers. All results are average number of 5 tests done in the same environment. Also I have not tested in detail for 10mb network as it's rather obsolete today and also not tested further for remaining delay/packet loss values as download values in results for such configuration were too low to mention (few or below 1 mb/s). Configuration was changed using netem on Linux machine which performed as web10g server using newest code from trunk r1217 (Ubuntu 14.04 LTS with 3.9.1 kernel and 2.0.7 userland) which I directly connected with second machine with Windows.

Do these results satisfy requirements mentioned in ticket description or something should be added/changed ?

[aaronmbr]

Comment #7 originally posted by aaronmbr on 2015-03-23T14:30:06.000Z:

Could you include the standard deviation of the 5 tests? These numbers are all over the place...

[aaronmbr]

Comment #8 originally posted by aaronmbr on 2015-03-25T14:44:32.000Z:

It turned out that for some configurations deviation was too high so I performed additional tests and now for all configurations its minimum 10 (instead of 5), I have performed as many attempts as were needed to reduce relative standard deviation below 10%. Results show that the biggest difference is between flash version integrated with Chrome and the one which comes from Adobe (~50%). Also upload values differences for 0% packet loss and 0ms delay is quite high. All details are visible in spreadsheet where you can define thresholds and see cells automatically being colored indicating which ones match defined requirements. Is that ok now or something should be added/changed?

[aaronmbr]

Comment #9 originally posted by aaronmbr on 2015-03-25T15:25:23.000Z:

This all looks good. Thanks! Would it be possible to get some Linux numbers too?

[aaronmbr]

Comment #10 originally posted by aaronmbr on 2015-03-27T13:43:20.000Z:

New spreadsheet with Linux system results added (tested on Java and Flash plugins for Firefox and Flash only for Chrome as Java support has been removed since version 35). Results show that there is huge difference between very old version of Flash in Linux's Firefox (11.x) when compared to results from newest one from Windows (17.x). Also results overall differ quite a lot between both systems. Are these numbers enough or is something missing?

[aaronmbr]

Comment #11 originally posted by aaronmbr on 2015-03-30T13:51:17.000Z:

After some investigation it turned out that for 100mb/s network speed results between Windows and Linux differed so much because of duplex config, for Linux it was full-duplex while for Windows half. I then configured Windows to work in full-duplex mode too and attached file contains updated results for 100mb/s for Windows.

[aaronmbr]

Comment #12 originally posted by aaronmbr on 2015-03-31T12:12:46.000Z:

These look great. Thanks! Would it be possible to include some warnings in the web client if Linux + Flash, or Chrome + Flash get used?

[aaronmbr]

Comment #13 originally posted by aaronmbr on 2015-03-31T13:34:45.000Z:

I've added proper changes on web-client-browser-os-warnings branch. If they look ok for you I could merge them into trunk.

[aaronmbr]

Comment #14 originally posted by aaronmbr on 2015-03-31T15:02:38.000Z:

Looks good. Closing.