Some of benchmarks results off for native default build

[lidija-b]

I was looking into native builds and tested on mips and I want to check if there is something I'm missing or some problem i missed because these results don't look like what is written as example in doc. So results were off for some of size benchmarks but other look as expected.

So for mips64 gets when running benchmark: (cflages '-c', '-O2', '-fdata-sections', '-ffunction-sections' ldflags '-O2', '-Wl,-gc-sections')

--absolute                          *default*
    Benchmark           Speed           Benchmark           Speed
    ---------           -----           ---------           -----
    aha-mont64              0           aha-mont64         4000000.00
    crc32                 100           crc32              4013000.00
    cubic                 400           cubic               10.39
    edn                     0           edn                3984000.00
    huffbench               0           huffbench          4108000.00
    matmult-int             0           matmult-int        4020000.00
    minver                800           minver               5.00
    nbody                 300           nbody               12.59
    nettle-aes              0           nettle-aes         3988000.00
    nettle-sha256           0           nettle-sha256      4000000.00
    nsichneu                0           nsichneu           4001000.00
    picojpeg                0           picojpeg           3747000.00
    qrduino                 0           qrduino            4210000.00
    sglib-combined          0           sglib-combined     4025000.00
    slre                    0           slre               4005000.00
    st                    300           st                  13.84
    statemate               0           statemate          4000000.00
    ud                    200           ud                  20.01
    wikisort              300           wikisort            14.36
    ---------           -----           ---------           -----
    Geometric mean          0           Geometric mean     71672.15
    Geometric SD          436.03            Geometric SD       375.21
    Geometric range        45           Geometric range    26891982.52
    All benchmarks run successfully         All benchmarks run successfully

And for size results are: (cflags '-c', '-Os', '-fdata-sections', '-ffunction-sections' ldflags '-Os', '-Wl,-gc-sections', '-nostartfiles', '-nostdlib')

--absolute                          *default*
    Benchmark            size           Benchmark            size
    ---------            ----           ---------            ----
    aha-mont64          1,904           aha-mont64           1.81
    crc32               1,184           crc32                5.15
    cubic              19,344           cubic                7.83
    edn                 3,200           edn                  2.20
    huffbench           3,296           huffbench            2.02
    matmult-int         1,552           matmult-int          3.70
    minver              2,432           minver               2.26
    nbody               1,840           nbody                2.60
    nettle-aes          5,184           nettle-aes           1.80
    nettle-sha256       6,288           nettle-sha256        1.13
    nsichneu           23,904           nsichneu             1.59
    picojpeg           18,896           picojpeg             2.35
    qrduino            13,872           qrduino              2.28
    sglib-combined      5,712           sglib-combined       2.46
    slre                6,256           slre                 2.58
    st                  2,000           st                   2.27
    statemate           8,832           statemate            2.39
    ud                  2,192           ud                   3.12
    wikisort            9,232           wikisort             2.19
    ---------           -----           ---------           -----
    Geometric mean      4,787           Geometric mean       2.47
    Geometric SD            2.48            Geometric SD         1.51
    Geometric range     9,936           Geometric range      2.08
    All benchmarks sized successfully       All benchmarks sized successfully

It doesn't look to be empty executable issue.

When i was my testing was based on Embench 05rc1(#42) and arm32. Have tested with same process on raspberypi and got the expected results.

[PaoloS02]

Hello. Have you specified the clock speed of your machine when you built your tests? You should do that with the --cpu-mhz option, e.g.:

./build_all.py --arch native --chip default --board default --cflags='-c -O2 -fdata-sections -ffunction-sections' --ldflags='-O2 -Wl,-gc-sections' --user-libs='-lm' --cpu-mhz 2000 --clean

This will make embench repeat the body of the tests as many times as required to have the results to be consistent when compared to other processors regardless of the clock speed.

You might also want to check whether you can deactivate any sort of default boost on your machine, just to be more precise about the clock speed you're using.

[PaoloS02]

Hello again. In addition to what I said last time I'd like to say that embench is currently designed for embedded systems. Even if a native option is available, a version of Embench suitable for systems with an OS is yet to come. I strongly encourage you to get involved with the monthly calls (send me an email to paolo.savini@embecosm.com so I can send you the details of the call) and/or to subscribe to the mailing list: https://embench.org/.

I would close this for now. Don't hesitate to get in touch here or on the mailing list for any doubt.

Thank you for your input Paolo

[jeremybennett]

In the absence of further comments, I am marking this as closed.