Comparison chart not working, raw data access question

[rochus-keller]

https://awfy-speed.stefan-marr.de/comparison/ is a great idea, but somehow I don't manage to see the bars; neither when all executables are selected, nor when I make the subset I'm interested in. Here is what I see in Firefox 78: Looks the same in Chromium.

Is there a way to get access to the raw data behind the graphs? E.g. I would like to know what options SOM++ was compiled with, and which versions were used of the executables and benchmarks.

Many thanks.

[smarr]

Sorry, the Codespeed instance hasn't been maintained for quite a while now.

But to answer your question on SOM++, this is the setup used:

    SOMpp:
        path: awfy/implementations/SOMpp/
        executable: som.sh
        args: " -H512MB "
        build: [make]

This means, it's just the standard flags. And, as you can see, the setup is so old, it's even from before the cmake changes. Ugh, looks like the SOM++ readme also needs an update :(

The results are roughly equivalent to what you'd get with cmake .; make today. But, very old...

So, I fear, the best option is to run the things locally that you're interested in, to get reliable numbers.

[rochus-keller]

Thanks. I was able to build SOM++ with both CMake (-m64 replaced by -m32 and -std=gnu++1y by -std=gnu++11) and also

this QMake project: SOMpp.pro.zip But to compare with your results I needed to know how you built your executable of course. Default options is ok, which means with the copying collector (which is about a factor two slower than with the M&S collector on my machine).

Can you please provide me with a link where I can get the times of each benchmark of each executable (i.e. the data table supposedly used by the diagram, whose contents I cannot see).

Thanks.

[smarr]

I am sorry, there is no recent data. I would have to run the experiments myself.

It sounds like you already have the setup. So, perhaps you can run the benchmarks yourself?

I would need to update the SOM++ benchmark CI, and rerun the benchmarks. I fear, I won't get to this any time soon, sorry.

[rochus-keller]

I don't need recent data, just any data on which e.g. the figure on your website or the som-st.github.io site is based. I assume CSOM and SOM++ are still the same version. So I can compare with the data measured on my machine (i.e. I know how SOM++ with default options performs on my machine and then know where to put my LjSOM on your figure). See https://github.com/rochus-keller/som/#a-som-to-luajit-bytecode-compiler-and-debugger.

[smarr]

Comparing across languages seems problematic, and CSOM, as well as SOM++ have been updated recently quite a bit to be able to execute all tests and benchmarks. Using old data would likely be problematic.

I'd recommend to base any comparison on a consistent set of data points from the same machine and comparable versions of SOM implementations.

[rochus-keller]

Unfortunately not all executables run on my machine (e.g. I don't have the required Java version and don't want to upgrade because I don't use Java otherwise). I'm still trying to find out which executables work; I was e.g. successful with Crystal 0.35, but this comparison makes no sense since it's a highly optimized statically AOT compiled implementation. I'd hoped to be able to use my SOM++ measurement as a reference for scaling and comparison with your measurements; don't see why this should not work to get a decent ball park figure.

[smarr]

Sorry, I fear I can't help with it. There's no raw data to work from. The system on which the benchmarks used to run got retired.

And I haven't had the time to migrate things to the new ReBenchDB setup, where you'd be more lucky, and could download raw data.