Closed Barsonax closed 4 years ago
Benchmark is always executed in Release mode. I don't know what causes the regression.
I just executed the benchmark again (Release mode, all other applications closed). This is the result:
Strange when I execute this I get different results. Also your results also differ quite alot from the ones in the readme.
EDIT: ah I see the readme is now updated again.
EDIT: did you run the benchmark without the debugger attached?
No debugger attached. Same machine as always.
Strange these results don't make sense. For some reason Singularity is performing way worse in the basic benchmarks but fine in the advanced (the generic one is faster than singleton...).
Have to investigate further what causes this. Might be hard if I cannot reproduce this though.
These are the results I get:
Added grace as a comparison since iam running this on a different machine. As you can see the perf here is pretty much identical for the most part to grace but somehow not when you run it.
Also my own benchmarkdotnet benchmarks do not indicate any difference so iam at a loss here why this is happening.
In this commit: https://github.com/danielpalme/IocPerformance/commit/ef47502a8fc8d01895bec5a33632467dcaabb301#diff-04c6e90faac2675aa89e2176d2eec7d8 the performance in all benchmarks (even the prepare and register) became worse quite drastically. Its too big for just random variation so something else is happening here.
Also noticed that in that commit the results are quite different from the run you did while Singularity did not had a version change.
I think you need to go on eBay and buy the following:
Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz Memory: 15,89GB
Joking, but joking seriously.
The other option is to ask Daniel to ship you his compiled binaries and then you run the benchmark that way. Probably a cheaper option to start.
@jzabroski I think you miss the fact that between runs the variation in the results is also way too high suddenly (even for the same version!)
I am not sure what variation you are referring to. When I look at his screenshot and the current readme, they are identical.
Thats because his screenshot and the current readme are from the same run...
check the results here, also singularity 0.14.0: https://github.com/danielpalme/IocPerformance/blob/ef47502a8fc8d01895bec5a33632467dcaabb301/README.md
Some of the results have a difference of like 50%....
Old run:
Current:
If this is normal variation that makes these benchmarks worthless.
@danielpalme can you try running only the singularity container benchmark and nothing else? Just to rule out another container's side effect ruining the benchmark?
I executed the benchmark again. Now Singularity performs much better again. Maybe some background process on my machine affected the performance.
Grace 7.0.0 Single Multi
Singleton 25 46
Transient 34 57
Combined 51 81
Complex 62 76
Property 95 103
Generics 50 77
IEnumerable 288 268
Conditional 40 63
Child Container 44819 26587
Asp Net Core 695 603
Interception With Proxy 812 496
Prepare And Register 140
Prepare And Register And Simple Resolve 787
Singularity 0.14.0 Single Multi
Singleton 24 45
Transient 38 61
Combined 50 88
Complex 65 82
Property
Generics 49 75
IEnumerable 292 220
Conditional
Child Container
Asp Net Core 507 555
Interception With Proxy
Prepare And Register 27
Prepare And Register And Simple Resolve 443
Jup thats the performance iam used to 👍. Lets keep a eye out for when this happens again (maybe it was caused by a side effect of another container?).
I have noticed a rather big sudden performance regression for Singularity in this benchmark which I cannot reproduce (by checking out this repo and running the benchmarks) and also iam not aware of any changes made to the code that might have caused this.
Just speculating here but was the benchmark accidentally run in debug mode?