search

gerich-home / it-depends

Lightweight dependency tracking library for JavaScript
http://it-depends-js.github.io/
Other
1 stars 2 forks source link

Fix and enable performance tests for AppVeyor #37

Open gerich-home opened 8 years ago

gerich-home commented 8 years ago

AppVeyor yields quite a different results that local performance tests: https://ci.appveyor.com/project/gerich-home/it-depends/build/1.0.220

Compare to local run on my machine:

'subscribe to computed with 1000 subscribers and unsubscribe them'
  1.05x slower
'subscribe to computed with 1 subscribers and unsubscribe them'
  1.40x slower
'subscribe to computed with 3 subscribers and unsubscribe them'
  1.14x slower
'subscribe to computed with 500 subscribers and unsubscribe them'
  1.01x faster
'computed updated 1000 times with 1000 subscribers'
  2.09x faster
'computed updated 1000 times with 1 subscribers'
  2.73x faster
'computed updated 1000 times with 500 subscribers'
  2.62x faster
'computed updated 1 times with 1000 subscribers'
  10.05x faster
'computed updated 1 times with 1 subscribers'
  10.42x faster
'computed updated 1 times with 500 subscribers'
  10.05x faster
'computed updated 3 times with 1000 subscribers'
  2.20x faster
'computed updated 3 times with 1 subscribers'
  1.50x faster
'computed updated 3 times with 500 subscribers'
  2.22x faster
'computed updated 50 times with 1000 subscribers'
  2.15x faster
'computed updated 50 times with 1 subscribers'
  1.83x faster
'computed updated 50 times with 500 subscribers'
  2.29x faster
'read observable 1000 times'
  2.43x faster
'read observable 100 times'
  2.03x faster
'read observable 1 times'
  1.92x faster
'subscribe to observable with 1000 subscribers and unsubscribe them'
  1.09x slower
'subscribe to observable with 1 subscribers and unsubscribe them'
  1.30x slower
'subscribe to observable with 3 subscribers and unsubscribe them'
  1.19x slower
'subscribe to observable with 500 subscribers and unsubscribe them'
  1.09x slower
'write observable 1000 times with 1 subscribers'
  12.82x faster
'write observable 1000 times with 3 subscribers'
  8.28x faster
'write observable 1000 times with 50 subscribers'
  2.62x faster
'write observable 1 times with 1000 subscribers'
  9.91x faster
'write observable 1 times with 1 subscribers'
  10.32x faster
'write observable 3 times with 1000 subscribers'
  1.83x faster
'write observable 50 times with 1000 subscribers'
  1.87x faster

Slow tests are significantly slower than on my local machine (up to 5.5 times), while fast tests are significantly faster (up to 72 times).

Thus AppVeyor results are not representative. What I see is that the deviation is much higher on AppVeyor (up to +-40%) when on my local machine it is quite slow (up to +-4.5%).

This behavior needs further investigation.

One can compare the behavior on AppVeyor and some alternative CI server (TravisCI?)

gerich-home commented 8 years ago

Different results were observed on machine with similar configuration as mine as well...

VanDalkvist commented 8 years ago

Here is TravisCI it-depends build Build Status - see add-travis branch

Here is the comparison of travis and appveyor for node 0.12 - https://www.diffchecker.com/ol73vjvt

And also a comparison between node 5 and node 0.12 - https://www.diffchecker.com/dvxwxhzx

It seems actually that some tests under travis build look like slower than appveyor build. But also there are tests that faster. I think it greatly depends on machines configuration on a build server. Also it works under separate container mostly and it has different processor time to build and execute. So, I think it's normal behaviour for cloud CI servers. Maybe local results will be more stable than cloud CI results.

VanDalkvist commented 8 years ago

Last version performance logs from TravisCI

node: 0.12 travis
  Running 'computed diamond updated 1000 times with 1000 hidden dependencies'
       'knockout' is etalon
       'itDepends' at 1.07x slower
  Running 'computed diamond updated 1000 times with 100 hidden dependencies'
       'itDepends' at 1.65x faster
       'knockout' is etalon
   Running 'computed diamond updated 1000 times with 500 hidden dependencies' 
       'itDepends' at 1.09x faster
       'knockout' is etalon
   Running 'subscribe to computed with 1000 subscribers and unsubscribe them' 
       'knockout' is etalon
       'itDepends' at 1.94x slower
   Running 'subscribe to computed with 1 subscribers and unsubscribe them' 
       'knockout' is etalon
       'itDepends' at 2.78x slower
   Running 'subscribe to computed with 3 subscribers and unsubscribe them' 
       'knockout' is etalon
       'itDepends' at 2.40x slower
   Running 'subscribe to computed with 500 subscribers and unsubscribe them'
       'knockout' is etalon
       'itDepends' at 2.25x slower
   Running 'computed updated 1000 times with 1000 subscribers' 
       'itDepends' at 1.90x faster
       'knockout' is etalon
   Running 'computed updated 1000 times with 1 subscribers' 
       'knockout' is etalon
       'itDepends' at 1.23x slower
   Running 'computed updated 1000 times with 500 subscribers' 
       'itDepends' at 1.55x faster
       'knockout' is etalon
   Running 'computed updated 1 times with 1000 subscribers' 
       'knockout' is etalon
       'itDepends' at 2.78x slower
   Running 'computed updated 1 times with 1 subscribers' 
       'knockout' is etalon
       'itDepends' at 2.34x slower
   Running 'computed updated 1 times with 500 subscribers' 
       'knockout' is etalon
       'itDepends' at 3.02x slower
   Running 'computed updated 3 times with 1000 subscribers' 
       'itDepends' at 2.97x faster
       'knockout' is etalon
   Running 'computed updated 3 times with 1 subscribers' 
       'knockout' is etalon
       'itDepends' at 1.46x slower
   Running 'computed updated 3 times with 500 subscribers' 
       'itDepends' at 1.61x faster
       'knockout' is etalon
   Running 'computed updated 50 times with 1000 subscribers'
       'itDepends' at 2.24x faster
       'knockout' is etalon
   Running 'computed updated 50 times with 1 subscribers'
       'knockout' is etalon
       'itDepends' at 1.14x slower
   Running 'computed updated 50 times with 500 subscribers'
       'itDepends' at 2.11x faster
       'knockout' is etalon
   Running 'read observable 1000 times' 
       'itDepends' at 2.24x faster
       'knockout' is etalon
   Running 'read observable 100 times' 
       'itDepends' at 2.45x faster
       'knockout' is etalon
   Running 'read observable 1 times' 
       'itDepends' at 22.72x faster
       'knockout' is etalon
   Running 'subscribe to observable with 1000 subscribers and unsubscribe them' 
       'knockout' is etalon
       'itDepends' at 2.36x slower
   Running 'subscribe to observable with 1 subscribers and unsubscribe them' 
       'knockout' is etalon
       'itDepends' at 3.67x slower
   Running 'subscribe to observable with 3 subscribers and unsubscribe them' 
       'knockout' is etalon
       'itDepends' at 2.26x slower
   Running 'subscribe to observable with 500 subscribers and unsubscribe them'
       'knockout' is etalon
       'itDepends' at 2.40x slower
   Running 'write observable 1000 times with 1 subscribers' 
       'itDepends' at 1.65x faster
       'knockout' is etalon
   Running 'write observable 1000 times with 3 subscribers' 
       'itDepends' at 1.55x faster
       'knockout' is etalon
   Running 'write observable 1000 times with 50 subscribers' 
       'itDepends' at 2.64x faster
       'knockout' is etalon
   Running 'write observable 1 times with 1000 subscribers' 
       'knockout' is etalon
       'itDepends' at 1.69x slower
   Running 'write observable 1 times with 1 subscribers' 
       'itDepends' at 2.48x faster
       'knockout' is etalon
   Running 'write observable 3 times with 1000 subscribers' 
       'itDepends' at 4.54x faster
       'knockout' is etalon
   Running 'write observable 50 times with 1000 subscribers' 
       'itDepends' at 3.31x faster
       'knockout' is etalon