Performance matrix for framework comparision

[sh-waqar]

It would be great if we would have some performance matrices to compare different frameworks (frontend/backend) just like TodoMVC has. Since the specs are same and for every framework same tasks are being followed we can write some automated scripts and check the performance for the tasks this way.

[jamesbrewerdev]

Personally, I don't think this is a good idea. The problem is that benchmarks are misleading and contribute to the "Rails doesn't scale" nonsense that is too prevalent already.

Since you mentioned TodoMVC, let's use that as an example.

If you Google "todomvc performance", the first result (for me, at least) is: https://github.com/featurist/todomvc-perf-comparison

Now I don't know whether this is a good benchmark or not, but I'm lazy so let's assume that it is.

Here's the conclusion posted by the author of that repository:

Overall:

Mercury, Vue and Mithril are king. Elm is a close fourth, except on IE9 where it is very slow.

Ember is the slowest overall; Angular, React and Backbone are trailing too.

Here's the question we have to ask: What does this tell you?

It tells me that Mercury, Vue, and Mithril are fantastic at rendering a list of 50 items, making a change to each of those items, and then deleting each of those items.

The problem is that this is almost meaningless. There are some cases where rendering speed is the most important thing, but those cases are few and far between.

What is infinitely more important is the ability to scale your team. That means that, if your team already knows Angular 1.x, then switching to anything else requires re-training the entire team. That's a massive time sink for what is, in my experience, a very minimal gain.

There is nothing inherently bad about benchmarking, but let's make sure we're benchmarking something useful. Otherwise we're going to get users coming in, seeing that Mithril is the fastest at rendering 50 items on Chrome 36 for OS X, and deciding that they should be using Mithril based on that fact alone.

We have a responsibility to make sure we aren't helping users shoot themselves in the foot. And, in that regard, a performance matrix is the wrong way to go.

[EricSimons]

I agree with most of @brwr's sentiments here in that performance comparisons will likely be super unfair, esp because of how much creative freedom RealWorld devs have when building out the apps.

However, I'm not completely opposed to implementing something like this. Does TodoMVC officially endorse/support those performance measurements? If so, how do they ensure fairness of each implementation's efficiency?

[jamesbrewerdev]

As far as I can tell, the benchmarks are not endorsed by TodoMVC. Nor does TodoMVC have any of their own.

[sh-waqar]

@brwr - you have made some excellent points and I agree with them. It can create some confusions for new developers and they can get some wrong impression about the framework. There is already much dispute with frameworks that which one is the "Best".

@EricSimons - No the benchmarks for TodoMVC are no officially endorsed by them, they are created by other developers.

I think it would be okay to close this issue.

[EricSimons]

Okay sounds good — thanks @sh-waqar! Would love to hear more suggestions from you in the future :)