Performance regression in singularity

[Barsonax]

I have noticed a rather big sudden performance regression for Singularity in this benchmark which I cannot reproduce (by checking out this repo and running the benchmarks) and also iam not aware of any changes made to the code that might have caused this.

Just speculating here but was the benchmark accidentally run in debug mode?

[danielpalme]

Benchmark is always executed in Release mode. I don't know what causes the regression.

I just executed the benchmark again (Release mode, all other applications closed). This is the result:

[Barsonax]

Strange when I execute this I get different results. Also your results also differ quite alot from the ones in the readme.

EDIT: ah I see the readme is now updated again.

EDIT: did you run the benchmark without the debugger attached?

[danielpalme]

No debugger attached. Same machine as always.

[Barsonax]

Strange these results don't make sense. For some reason Singularity is performing way worse in the basic benchmarks but fine in the advanced (the generic one is faster than singleton...).

Have to investigate further what causes this. Might be hard if I cannot reproduce this though.

[Barsonax]

These are the results I get:

Added grace as a comparison since iam running this on a different machine. As you can see the perf here is pretty much identical for the most part to grace but somehow not when you run it.

Also my own benchmarkdotnet benchmarks do not indicate any difference so iam at a loss here why this is happening.

In this commit: https://github.com/danielpalme/IocPerformance/commit/ef47502a8fc8d01895bec5a33632467dcaabb301#diff-04c6e90faac2675aa89e2176d2eec7d8 the performance in all benchmarks (even the prepare and register) became worse quite drastically. Its too big for just random variation so something else is happening here.

Also noticed that in that commit the results are quite different from the run you did while Singularity did not had a version change.

[jzabroski]

I think you need to go on eBay and buy the following:

Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz Memory: 15,89GB

Joking, but joking seriously.

[jzabroski]

The other option is to ask Daniel to ship you his compiled binaries and then you run the benchmark that way. Probably a cheaper option to start.

[Barsonax]

@jzabroski I think you miss the fact that between runs the variation in the results is also way too high suddenly (even for the same version!)

[jzabroski]

I am not sure what variation you are referring to. When I look at his screenshot and the current readme, they are identical.

[Barsonax]

Thats because his screenshot and the current readme are from the same run...

check the results here, also singularity 0.14.0: https://github.com/danielpalme/IocPerformance/blob/ef47502a8fc8d01895bec5a33632467dcaabb301/README.md

Some of the results have a difference of like 50%....

Old run:

Current:

If this is normal variation that makes these benchmarks worthless.

[Barsonax]

@danielpalme can you try running only the singularity container benchmark and nothing else? Just to rule out another container's side effect ruining the benchmark?

[danielpalme]

I executed the benchmark again. Now Singularity performs much better again. Maybe some background process on my machine affected the performance.

Grace 7.0.0                                  Single      Multi
 Singleton                                       25         46
 Transient                                       34         57
 Combined                                        51         81
 Complex                                         62         76
 Property                                        95        103
 Generics                                        50         77
 IEnumerable                                    288        268
 Conditional                                     40         63
 Child Container                              44819      26587
 Asp Net Core                                   695        603
 Interception With Proxy                        812        496
 Prepare And Register                           140
 Prepare And Register And Simple Resolve        787

Singularity 0.14.0                           Single      Multi
 Singleton                                       24         45
 Transient                                       38         61
 Combined                                        50         88
 Complex                                         65         82
 Property
 Generics                                        49         75
 IEnumerable                                    292        220
 Conditional
 Child Container
 Asp Net Core                                   507        555
 Interception With Proxy
 Prepare And Register                            27
 Prepare And Register And Simple Resolve        443

[Barsonax]

Jup thats the performance iam used to 👍. Lets keep a eye out for when this happens again (maybe it was caused by a side effect of another container?).