TypeScript vs JavaScript

[timocov]

Hi everyone,

seems that it is not fair to have TypeScript in this tools because it compiles to esnext (not es5). In this case performance, energy consumption and memory usage depends on NodeJS (which run esnext code).

I guess that compiled TypeScript code is almost the same as JavaScript code written to solve the same problem (and should have almost the same results with pure JavaScript) and here we should have one of solutions:

1. JavaScript (es5) and JavaScript (esnext)
2. JavaScript (es5) and TypeScript (compiled to es5 not esnext)

[carbontwelve]

What you're suggesting crossed my mind and I came here to submit as an issue myself however doesn't TypeScript even when compiled to ECMA still add additional checks in order to ensure strict types?

If so, the additional "burden" of TypeScript's checks on top of the vanilla ECMA do contribute to energy consumption, cpu usage, etc in a way that should be displayed in the list against vanilla JavaScript.

[timocov]

however doesn't TypeScript even when compiled to ECMA still add additional checks in order to ensure strict types?

No, TypeScript compiler does not emit any type information into runtime (you can try it in the playground).

I believe that only a little bit things can be measured against es5 - the code which is used functions from https://github.com/Microsoft/tslib (for example spread operator, (async) generators) (but I'm not sure whether we should have such comparison).

[kibertoad]

What is even more concerning is that Typescript implementation uses classes and Javascript one does not. Classes introduce quite a bit of runtime overhead, so this skews results heavily. Which is a strange decision to make, considering that Javascript supports classes as well (at least in Node 7 it should).

[evanshortiss]

I just noticed that the Mandelbrot JS example forks worker processes and the TS version doesn’t. Odd to compare single vs multi core in this way since both compile to JS and can (and should) use the same features?

[States]

At the time of this study, back in 2017, those solutions were the most performance efficient ones on CLBG. While many different varying solutions are submitted to CLBG, we only obtained the fastest running ones - if that was single or multi thread, it was irrelevant. The deciding process is detailed in the published paper if you have the time to check it.

[adri326]

After reading the fannkuch-redux example, it strikes me that two different solutions were implemented for typescript and javascript, despite requiring nothing but arrays and numbers:

• node version
• typescript version

The typescript version should have been an exact copy of the javascript version, with only type annotations added. It also seems like the solution is using the new Array syntax, which is known to cause arrays to be a lot slower because they are treated as dictionaries.

8c8
< function fannkuch(n: number) {
---
> function fannkuch(n) {

Copying the javascript version and adding the above type annotation gives me a 40% speed boost and makes it as fast as the javascript version. I am also unable to replicate the performance gap between javascript and typescript on a modern nodejs version, but this might be due to the use of new Array, which back then had major performance issues.

To me, the claim that those solutions were the most performance efficient ones on CLBG is false for this example. I'm guessing that the fannkuch-redux solutions were simply too scary for anyone to look in depth into it. It saddens me that the results are biased because of this.

It looks like this example slipped through the sieve of having the most efficient solution in a language, it saddens me that the results got biased because of this.

[States]

Well I'm glad you are able to get a more runtime efficient version than the best version 5 years ago with a 5 year old compiler. Feel free to go over to CLBG and as Isaac Gouy, the maintainer, would say - if you think you can write a better version, submit it. Before even submitting, look over the rules as they are very well explained. The TS fannkuch version was even contributed by him, as you can see in the source-code, and he aimed to replicate the JS version as close as possible.

Now I'm not going to waste time arguing with you over our 'claim'. I stated facts, and happily with a quick search or even contacting the maintainer himself, feel free to view the version used at the time of the study and verify yourself if we are falsifying anything we stated. I'll even do you a favor and do the searching for you:

The conference was on October'17, the submission was on June'17, and our work was possibly 3-4 months prior to the submission date, so Feb/March/April was when we ran everything. On the way back time machine archive, you got yourself the version we used, and the most efficient at the time (https://web.archive.org/web/20170425064751/https://benchmarksgame.alioth.debian.org/u64q/typescript.html).

The beauty of open source research is you can feel free to fork this, update it as you wish, modernize all the versions and compilers and re-run the study. As many have already done, and submitted updated or revisited peer reviewed papers, concluding the same facts. Now just stating we are falsifying anything, well I won't be quiet about that.

[evanshortiss]

No one is saying results are falsified (Edit: sounds like maybe they did, and I disagree with that). They're saying the methodology for TS doesn't make sense.

The study is running different JS (the vanilla JS, vs. JS compiled from TS) code to accomplish the same task, in the same node JS engine. The TS compiler outputs JS, so you should have simply use the same code and labelled the category "JavaScript/TypeScript". TS not using thread/processes is a critical omission, since it had that capability available.

Would the same issue apply for the C vs C++ examples since C code is valid in a C++ program?

I don't have anymore to add other than I think it's a great study! The issue people have ( I suspect ) is that it's still being referenced and this data point is pretty misleading due to a flaw in the study. There's nothing wrong with noting a mistake and owning it.

[adri326]

I got carried away in my comment and I must apologize for that. I did not mean to say that you falsified the results, and there is proof that you did not, what I meant to say is that the dataset has a flaw and I tried to pinpoint it.

My opinion on the js/ts issue aligns with @evanshortiss's: comparing javascript with typescript doesn't make much sense if one can tweak the typescript code to transpile to the same code as javascript.

[States]

Glad that confusion is out of the way! Thanks for the apology.

I have previously tried addressing that point on the JS vs. TS before in other forums, as this work comes up every so often. I too have chatted with CLBG's maintainer to try to understand why the difference was so crazy on the fannkuch example (all other ones had minor differences). Just to also point out, the TS and JS solutions were all "stand-alone" and written from scratch, sometimes by different authors as one would expect. In general, the JS solutions were more efficient. That one fannkuch difference was very heavy, and as such, the final normalized ranking was probably not as fair as with the other languages. However, I did leave all 10 individual benchmark data online https://sites.google.com/view/energy-efficiency-languages/results for a better detailed view.

The used TS fannkuch example was created by the maintainer (and coincidentally the JS version too, with slight adaptations by other authors. He tried to get them as similar as possible to each other, but he stated that a known issue at the time was an incorrect type-checking which could have been the reason for the huge bottleneck. We never got down to the actual issue, as the differences from that "slow" version was basically the same as a more recent (at the time we chatted) "faster" version other than the array type. Issues with TS is probably a reason why it is no longer included on CLBG I would assume.

A quick test which can always be done is to just update the fannkuch example with however one believe it should be, and just use the framework I setup to test it very quickly. We ended up revisiting this problem using more idiomatic solutions and the ranking was basically the same (albeit we didn't have TS in that one and a few others were absent). From what I understood, another research team is currently revisiting this study of mine and re-producing it with the modern compilers, so maybe somethings might mix up a little, but I doubt there would be huge jumps to be honest.