alexp-sssup
commented
6 years ago Hello Alon. Sorry for the delay, we needed some time to reproduce results and gather all required data.
We have been pleasantly surprised that Cheerp is now integrated in emscripten's benchmarks. We have been maintaining our own branch of emscripten to run tests, which we have now rebased on 1.37.36 and released. You can find it here.
About size results, the differences arise from the test cases being slightly different. I will focus on test_primes and test_memops on this discussion, other benchmarks are available in the branch linked above.
To reproduce our results you need to simply remove printf statements from the test source. Both in test_primes and test_memops printf is used to report an invalid test size and to print the test results. In our modified version we remove the printf calls and directly return the numerical result of the test. As an example here is the source of memops with our changes:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int N, M;
int arg = argc > 1 ? argv[1][0] - '0' : 3;
switch(arg) {
case 0: return 0; break;
case 1: N = 1024*1024; M = 55; break;
case 2: N = 1024*1024; M = 400; break;
case 3: N = 1024*1024; M = 800; break;
case 4: N = 1024*1024; M = 4000; break;
case 5: N = 1024*1024; M = 8000; break;
default: /*printf("error: %d\\n", arg);*/ return -1;
}
int final = 0;
char *buf = (char*)malloc(N);
for (int t = 0; t < M; t++) {
for (int i = 0; i < N; i++)
buf[i] = (i + final)%256;
for (int i = 0; i < N; i++)
final += buf[i] & 1;
final = final % 1000;
}
//printf("final: %d.\\n", final);
return final;
}Of course, we use the same modified sources when building with Emscripten and with Cheerp.
About larger scale tests, llvm-ar is not supported, we link libraries using llvm-link as documented here.
Various test cases need some patching to run with Cheerp. The branch we published contains all required patches. It should be noted that the tests were originally ported to be compiled to plain JS (Cheerp genericjs mode), so the patches could be heavily reduced if only the wasm target is of interest.
kripken
Hi Cheerp devs :)
I see your website says "30% smaller than Emscripten" so I was curious to measure that. Running emscripten's
tests/test_benchmark.pyscript, I see the following results which are very different:.wasmand.jsfiles together, but it's also true when just looking at the wasm.)test_primesand 27% smaller ontest_memops.test_box2d, emscripten is 23% smaller there.That's for size. Speed-wise, the results are a mix (some a little faster, some a little slower, some about the same).
Several of the tests hit problems when running the Cheerp output:
test_base64hitsError: this should be unreachable.test_fasta_floathitsRuntimeError: index out of bounds.test_havlakhangs.test_box2dhitsRuntimeError: indirect call signature mismatch.I also couldn't get some tests to build in Cheerp:
test_linpack,test_bullet,test_lua_binarytrees,test_lua_scimark,test_zlib. For exampletest_zlibsaysMaybe the script doesn't build them right? It calls Cheerp's llvm-ar etc., and the commands work with emscripten and native builds, but maybe something more needs to be done for Cheerp? Or am I hitting Cheerp limitations - would zlib, Lua, etc. need to be ported to Cheerp first?
This is on Cheerp
1516806243-1~xenial(which is the latest nightly build I see for Xenial, from Jan 24) and emscripten 1.37.36 (last tagged release, from Mar 13).These results are very different from the ones you reported, perhaps we are not measuring the same thing somehow? (All the details of how I got the measurements mentioned above are in the linked script that runs those benchmarks, I basically just ran that script as-is except for uncommenting the line to enable running Cheerp.) Or maybe our results are about different versions?