search

geneura-papers / 2016-DCAI

13th International Conference on Distributed Computing and Artificial Intelligence (DCAI'16) + Review ISAFM
0 stars 0 forks source link

Results from new experiments #13

Open vrivas opened 7 years ago

vrivas commented 7 years ago

Experiments using different navigators running individually

(Included in file: https://github.com/geneura-papers/2016-DCAI/blob/master/jsRBFNN/resultados_isafm_2017/resultados_por_navegador_ejecutado_individualmente.ods)

ERROR:  MSE     
ORDER   AVERAGE STDEV   BROWSER
1   9,21E-04    7,22E-07    FIREFOX 50.1.0 (Linux)
2   9,72E-04    3,19E-08    CHROMIUM 55.0.2883.87
3   1,13E-03    1,02E-06    CHROME 55.0.2883.87
4   1,25E-03    8,92E-06    FIREFOX 50.1.0 (Windows)
5   1,73E-03    4,02E-05    FIREFOX 47
6   1,52E-02    2,16E-03    IE 11.0.9600.18537
7   4,22E-02    4,17E-02    EDGE 38.14393.0.0
8   5,46E-02    5,42E-02    SAFARI 10.0.1
ERROR:  RMSE        
ORDER   AVERAGE STDEV   BROWSER
1   3,01E-02    1,79E-04    FIREFOX 50.1.0 (Linux)
2   3,11E-02    8,24E-06    CHROMIUM 55.0.2883.87
3   3,26E-02    2,02E-04    CHROME 55.0.2883.87
4   3,36E-02    1,20E-03    FIREFOX 50.1.0 (Windows)
5   3,68E-02    3,76E-03    FIREFOX 47
6   1,15E-01    1,96E-02    IE 11.0.9600.18537
7   1,61E-01    1,63E-01    EDGE 38.14393.0.0
8   1,93E-01    1,73E-01    SAFARI 10.0.1
ERROR:  MAE     
ORDER   AVERAGE STDEV   BROWSER
1   2,35E-02    1,35E-04    FIREFOX 50.1.0 (Linux)
2   2,49E-02    6,89E-06    CHROMIUM 55.0.2883.87
3   2,66E-02    1,40E-04    CHROME 55.0.2883.87
4   2,67E-02    1,03E-03    FIREFOX 50.1.0 (Windows)
5   2,90E-02    2,65E-03    FIREFOX 47
6   9,40E-02    1,70E-02    IE 11.0.9600.18537
7   1,37E-01    1,19E-01    EDGE 38.14393.0.0
8   1,62E-01    1,19E-01    SAFARI 10.0.1
ERROR:  MdAE        
ORDER   AVERAGE STDEV   BROWSER
1   1,90E-02    1,62E-04    FIREFOX 50.1.0 (Linux)
2   2,12E-02    1,56E-05    CHROMIUM 55.0.2883.87
3   2,26E-02    1,32E-03    FIREFOX 50.1.0 (Windows)
4   2,35E-02    1,18E-04    CHROME 55.0.2883.87
5   2,47E-02    3,03E-03    FIREFOX 47
6   8,62E-02    3,32E-02    IE 11.0.9600.18537
7   1,43E-01    1,65E-01    EDGE 38.14393.0.0
8   1,63E-01    1,53E-01    SAFARI 10.0.1
ERROR:  MAPE        
ORDER   AVERAGE STDEV   BROWSER
1   1,25E+00    4,52E-01    FIREFOX 50.1.0 (Linux)
2   1,32E+00    2,29E-02    CHROMIUM 55.0.2883.87
3   1,43E+00    3,07E+00    FIREFOX 50.1.0 (Windows)
4   1,47E+00    5,26E-01    CHROME 55.0.2883.87
5   1,55E+00    8,18E+00    FIREFOX 47
6   5,41E+00    5,97E+01    IE 11.0.9600.18537
7   7,90E+00    4,34E+02    EDGE 38.14393.0.0
8   9,46E+00    4,43E+02    SAFARI 10.0.1
ERROR:  MdAPE       
ORDER   AVERAGE STDEV   BROWSER
1   1,00E+00    3,51E-01    FIREFOX 50.1.0 (Linux)
2   1,15E+00    4,31E-02    CHROMIUM 55.0.2883.87
3   1,21E+00    4,04E+00    FIREFOX 50.1.0 (Windows)
4   1,22E+00    2,68E-01    CHROME 55.0.2883.87
5   1,27E+00    6,90E+00    FIREFOX 47
6   4,39E+00    1,04E+02    IE 11.0.9600.18537
7   7,74E+00    5,16E+02    EDGE 38.14393.0.0
8   8,76E+00    4,72E+02    SAFARI 10.0.1
ERROR:  RMSPE       
ORDER   AVERAGE STDEV   BROWSER
1   1,61E+00    7,08E-01    FIREFOX 50.1.0 (Linux)
2   1,65E+00    2,13E-02    CHROMIUM 55.0.2883.87
3   1,81E+00    3,86E+00    FIREFOX 50.1.0 (Windows)
4   1,86E+00    9,01E-01    CHROME 55.0.2883.87
5   1,99E+00    1,31E+01    FIREFOX 47
6   6,97E+00    7,74E+01    IE 11.0.9600.18537
7   9,62E+00    6,48E+02    EDGE 38.14393.0.0
8   1,18E+01    7,05E+02    SAFARI 10.0.1
ERROR:  RMdSPE      
ORDER   AVERAGE STDEV   BROWSER
1   1,00E+00    3,51E-01    FIREFOX 50.1.0 (Linux)
2   1,15E+00    4,31E-02    CHROMIUM 55.0.2883.87
3   1,21E+00    4,04E+00    FIREFOX 50.1.0 (Windows)
4   1,22E+00    2,68E-01    CHROME 55.0.2883.87
5   1,27E+00    6,90E+00    FIREFOX 47
6   4,39E+00    1,04E+02    IE 11.0.9600.18537
7   7,74E+00    5,16E+02    EDGE 38.14393.0.0
8   8,76E+00    4,72E+02    SAFARI 10.0.1
ERROR:  sMAPE       
ORDER   AVERAGE STDEV   BROWSER
1   1,25E+00    4,32E-01    FIREFOX 50.1.0 (Linux)
2   1,31E+00    2,42E-02    CHROMIUM 55.0.2883.87
3   1,43E+00    3,04E+00    FIREFOX 50.1.0 (Windows)
4   1,48E+00    5,60E-01    CHROME 55.0.2883.87
5   1,55E+00    8,30E+00    FIREFOX 47
6   5,45E+00    4,64E+01    IE 11.0.9600.18537
7   7,35E+00    3,05E+02    EDGE 38.14393.0.0
8   8,74E+00    2,89E+02    SAFARI 10.0.1
ERROR:  sMdAPE      
ORDER   AVERAGE STDEV   BROWSER
1   1,00E+00    3,48E-01    FIREFOX 50.1.0 (Linux)
2   1,15E+00    4,49E-02    CHROMIUM 55.0.2883.87
3   1,20E+00    3,97E+00    FIREFOX 50.1.0 (Windows)
4   1,23E+00    2,71E-01    CHROME 55.0.2883.87
5   1,27E+00    6,79E+00    FIREFOX 47
6   4,29E+00    8,73E+01    IE 11.0.9600.18537
7   7,27E+00    4,03E+02    EDGE 38.14393.0.0
8   8,24E+00    3,68E+02    SAFARI 10.0.1
ERROR:  MASE        
ORDER   AVERAGE STDEV   BROWSER
1   1,28E+00    4,05E-01    FIREFOX 50.1.0 (Linux)
2   1,35E+00    2,11E-02    CHROMIUM 55.0.2883.87
3   1,45E+00    4,25E-01    CHROME 55.0.2883.87
4   1,46E+00    3,08E+00    FIREFOX 50.1.0 (Windows)
5   1,58E+00    7,93E+00    FIREFOX 47
6   5,14E+00    5,08E+01    IE 11.0.9600.18537
7   7,51E+00    3,56E+02    EDGE 38.14393.0.0
8   8,85E+00    3,55E+02    SAFARI 10.0.1
ERROR:  RMSSE       
ORDER   AVERAGE STDEV   BROWSER
1   1,64E+00    5,34E-01    FIREFOX 50.1.0 (Linux)
2   1,70E+00    2,64E-02    CHROMIUM 55.0.2883.87
3   1,78E+00    6,06E-01    CHROME 55.0.2883.87
4   1,84E+00    3,58E+00    FIREFOX 50.1.0 (Windows)
5   2,01E+00    1,12E+01    FIREFOX 47
6   6,28E+00    5,84E+01    IE 11.0.9600.18537
7   8,78E+00    4,84E+02    EDGE 38.14393.0.0
8   1,05E+01    5,16E+02    SAFARI 10.0.1
JJ commented 7 years ago

:+1:

vrivas commented 7 years ago

Surprisingly (or not?) best results are achieved with latest versions of Firefox and Chromium (both of them running under Linux), followed in any case by the same navigators under Windows. "Worst" browsers seem to be IE and Edge and... Safari!!!!

JJ commented 7 years ago

You should run node too to compare.

vrivas commented 7 years ago

OK... but the library is intended to allow running EA in browsers :/

JJ commented 7 years ago

But, what's the point of making the comparison? What you want to prove is that browsers are fast enough, right? So reviewer number 3 will say that browsers are slower than desktop applications, so it's better to pre-empt that objection by testing it also for node.

vrivas commented 7 years ago

Results analysed by @eparrasg for MSE (the measure used by De Jhon in his initial paper) Rejected means there exist NO signficant differences wrt control browser. Non rejected means just the opposite.

Control Browser: firefox-50-1-0-macs    ρ   α/i Hypothesis
firefox-50-1-0-androids         0,00003 0,00714 Rejected
firefox-47-0-linuxes            0,00140 0,00833 Rejected
chromium-51-0-2704-79-linux     0,00259 0,01000 Rejected
chrome-55-0-2883-87-windows     0,00811 0,01250 Rejected
edge-14-14393               0,01992 0,01667 Non Rejected
safari-10-0-2-macs          0,71500 0,02500 Non Rejected
chrome-55-0-2883-95-macs        0,71500 0,05000 Non Rejected

We have similar analysis for every measure (13 measures) . The control browsers is always Firefox 50 in Mac, and the Rejected and non rejected appear more or less in the same browsers.

JJ commented 7 years ago

Great. Add that to the paper.

vrivas commented 7 years ago

There was an error in the data sent by @eparrasg: the null hypothesis was just the inverse, and the real results are:

Control Browser: firefox-50-1-0-macs    ρ   α/i Hypothesis
chrome-55-0-2883-95-macs        0,00003 0,00714 Rejected
safari-10-0-2-macs          0,00140 0,00833 Rejected
edge-14-14393               0,00259 0,01000 Rejected
chrome-55-0-2883-87-windows     0,00811 0,01250 Rejected
chromium-51-0-2704-79-linux     0,01992 0,01667 Non Rejected
firefox-47-0-linux          0,71500 0,02500 Non Rejected
firefox-50-1-0-android          0,71500 0,05000 Non Rejected

I.e., the same but now Rejected means worst than the control browser, while Non rejected means no differences wrt control browser.

By the way, I can't imagine way chrome in Mac is so bad :/