search

01alchemist / TurboScript

Super charged typed JavaScript dialect for parallel programming which compiles to WebAssembly
Apache License 2.0
494 stars 35 forks source link

Benchmark Results? #77

Closed MichaReiser closed 7 years ago

MichaReiser commented 7 years ago

I'm just curious. Do you have any benchmark that compares TurboScript with JS or Emscripten?

nidin commented 7 years ago

Not yet. I am planning to do some encoding/ decoding benchmark.

MaxGraey commented 7 years ago

I also prepared n-body bench, but I'm waiting wasm's sqrt op.

nidin commented 7 years ago

@max then I will try to add sqrt support today :)

MaxGraey commented 7 years ago

@nidin check slack plz. I have some questions.

nidin commented 7 years ago

MaxGraey's N-Body Benchmark results WebAssembly vs JavaScript image

MaxGraey commented 7 years ago

It's without binaryen optimizations?

nidin commented 7 years ago

Yes. I tried binaryen but was getting errors. I don't know what is the problem.

MaxGraey commented 7 years ago

It seems a good result as for unoptimized version! I think next we should add same bench test for c++ (emscripten).

MichaReiser commented 7 years ago

Thank you. That's what I expected :( JavaScript is faster than most of us think it is. Furthermore, I believe that the JITs have still some room concerning optimizing WASM.

To be honest. I'm a little bit released to see these results. This would have been trouble to my master thesis if your code would have been significantly faster than JS ;)

It seems a good result as for unoptimized version! I think next we should add same bench test for c++ (emscripten).

That's a good idea. Just ensure that you use at least -O2

MaxGraey commented 7 years ago

@MichaReiser can you add n-body test to your bench tests? You can find sources here. It will be interesting how LLVM optimize this bench.

MichaReiser commented 7 years ago

@MaxGraey I just wanted to ask you if you can give me the JS source. So, thanks! Will take a look soon.

MichaReiser commented 7 years ago

hmm... If i look at the result... WASM and JS do not return the same results

MichaReiser commented 7 years ago

Chrome JS: 1.54775s 🎉 WASM: 1.91057s

Firefox: JS: 1.360125s 🎉 WASM: 1.3785s

I prefer the results of Firefox. In my case, both implementations return the same result.

A side note. WebAssembly is also about predictability concerning performance. And this is true in this case. The variance of the WASM implementation is lower than the one of the JS implementation (at least using chrome. This is not the case for FF)

js x 0.64 ops/sec ±1.52% (8 runs sampled) wasm x 0.51 ops/sec ±0.97% (7 runs sampled)

MacBook Mid 2014

nidin commented 7 years ago

Good numbers.

winksaville commented 7 years ago

Where is the benchmark code, I'll create a C/C++ version if I have time today?

MaxGraey commented 7 years ago

@winksaville code

winksaville commented 7 years ago

This http://benchmarksgame.alioth.debian.org/u64q/program.php?test=nbody&lang=typescript&id=3 from The Computer Language Benchmark Game http://benchmarksgame.alioth.debian.org/ looks similar, but not exactly.

Where did you get yours? I'm guessing wherever that was there is already a C version, like this one http://benchmarksgame.alioth.debian.org/u64q/program.php?test=nbody&lang=gcc&id=4 from The Computer Language Benchmark Game.

On Thu, May 25, 2017 at 9:03 AM Max Graey notifications@github.com wrote:

@winksaville https://github.com/winksaville code https://github.com/MaxGraey/TurboScript/blob/MaxGraey/add-benchmark/benchmark/n-body/ts/index.ts

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/01alchemist/TurboScript/issues/77#issuecomment-304049095, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-hHBvkph5w3Rq8qbeW1LDeB-gVtu9pks5r9aZUgaJpZM4NV0E- .

MichaReiser commented 7 years ago

I ran the same benchmark with optimizations disabled:

Chrome js: 1.63425s wasm: 3.4968s

Firefox js: 1.463375s wasm: 3.14s

So, the optimization make a huge difference!

Something I want to add. I believe nbody is not a good benchmark to show that TurboScript can be faster. The problem here is, that nbody makes no use of CPU instructions that only WASM offers. Furthermore, a JIT compiler can generate at least as efficient native code for this implementation as any offline compiler.

winksaville commented 7 years ago

I suspect the reason is that it's early days for the internal wasm compilers, but time will tell. Sorry about my previous post, I did it using the google inbox "link" command which obviously doesn't translate well to github markup :(

On Thu, May 25, 2017 at 9:31 AM Micha Reiser notifications@github.com wrote:

I ran the same benchmark with optimizations disabled:

Chrome js: 1.63425s wasm: 3.4968s

Firefox js: 1.463375s wasm: 3.14s

So, the optimization make a huge difference!

Something I want to add. I believe nbody is not a good benchmark to show that TurboScript can be faster. The problem here is, that nbody makes no use of CPU instructions that only WASM offers. Furthermore, a JIT compiler can generate at least as efficient native code for this implementation as any offline compiler.

  • There are no dynamic JS features. So, the optimizer can do a good job
  • JIT optimizer have more information at hand. E.g. is a function only called with a specific value and, therefore, can some checks be removed
  • A jit optimizer can perform more aggressive inlining as the code overhead has no affect on the compiled file (it's just in the memory).

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/01alchemist/TurboScript/issues/77#issuecomment-304056245, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-hHBuBYDEZa2yGUzx7cM7k2mHaqRb_ks5r9azbgaJpZM4NV0E- .

MaxGraey commented 7 years ago

@MichaReiser LLVM do the same job: aggressive inlining, dead code illumination, precompute constant expressions and much more and I think it should do this better then JIT which need balance between quality of optimization and time of compilation in memory.

MaxGraey commented 7 years ago

n-body bench is very common test for any language. I think computing fib or isPrime is more syntactic

MichaReiser commented 7 years ago

@MichaReiser LLVM do the same job: aggressive inlining, dead code illumination, precompute constant expressions and much more and I think it should do this better then JIT which need balance between quality of optimization and time of compilation in memory.

The point about balancing is true. However, if you look at a function that is generally only called with non null values. However, it also accepts null values. An offline compiler cannot remove those branches (except the function is inlined) as there might be the case where the function is actually called with null. A JIT compiler, on the contrary, can remove all branches for null if the function has never been called with null (if the function is called once with a null value, a deopt is needed). So there are cases where JIT compilers can achieve better results.

Yes, n-body is a common benchmark. But I just do not expect any benefits over plain JavaScript. It does not use any features that wasm supports better than JS

MichaReiser commented 7 years ago

Just a short update

I implemented the nbody benchmark using emscripten:

Chrome:

nbody...
  js x 0.65 ops/sec ±1.19% (8 runs sampled)
  emcc x 0.62 ops/sec ±0.54% (8 runs sampled)
  wasm x 0.49 ops/sec ±0.57% (7 runs sampled)

Firefox:

nbody...
  js x 0.76 ops/sec ±1.81% (8 runs sampled)
  emcc x 0.81 ops/sec ±0.92% (8 runs sampled)
  wasm x 0.70 ops/sec ±1.05% (8 runs sampled)

I need to investigate what causes the difference between Emscripten and my implementation. But it should be possible to be, at least, as fast as JavaScript.

The source code of the C++ implementation is located here (I also inlined most of the variables in the JavaScript implementations).