embench / embench-iot

The main Embench repository
https://www.embench.org/
GNU General Public License v3.0
254 stars 104 forks source link

Reproduced baseline program size is larger than expected #143

Closed bzbplumb closed 2 years ago

bzbplumb commented 3 years ago

I am trying to reproduce the baseline code size values. Would it be possible to get more details on how to build the code size baseline?

I am getting values larger than the baseline - see below - using the gcc compiler version 9 and 10. I have included the build and size log file.

Taking the aha-mont64 module as an example. The module code size is small ~888bytes but the total size is bigger than expected 2.38kb.

Any help would be much appreciated

build-2021-09-17-113501.log size-2021-09-17-113954.log size-2021-09-17-113958.log d

jeremybennett commented 3 years ago

Hi @bzbplumb The results should be identical for identical set ups. The precise versions of compilers etc matter for this, and are specified in the the iot-results repository details for each target. For example for the Arm Cortex M4 when configured for speed.

I've assigned to @PaoloS02 who did much of the original work on ensuring reproducibility.

bzbplumb commented 3 years ago

Hi Jeremy,

Thanks. I have included the version output of the 9 & 10 gcc compilers.

GNU Tools for Arm Embedded Processors 9-2019-q4-major.txt GNU Arm Embedded Toolchain 10.3-2021.07.txt

I also tried the build_all command line from the run_all.py script with the same result. ./build_all.py --builddir build --logdir logs --clean --verbose --arch=arm --chip=cortex-m4 --board=generic --cc=arm-none-eabi-gcc --cflags="-c -mcpu=cortex-m4 -mthumb -mfloat-abi=soft -Os" --ldflags="-nostartfiles -nostdlib" --dummy-libs="crt0 libc libgcc libm"

bzbplumb commented 2 years ago

Hi Jeremy and Paolo,

I found the cause of my woes - it was all user error. The replicated baselines values I am getting are close to the actual values. Geometric Mean=0.95 for both version 9 & 10 of the ARM gcc compiler

Command line: ./build_all.py --builddir build --logdir logs --clean --verbose --arch=arm --chip=cortex-m4 --board=generic --cc=arm-none-eabi-gcc --cflags="-c -mcpu=cortex-m4 -mthumb -mfloat-abi=soft -Os -ffunction-sections -fdata-sections" --ldflags="-nostartfiles -nostdlib -march=armv7 -Wl,-Map=arm-m4.map -Wl,--gc-sections" --dummy-libs="crt0 libc libgcc libm"

baseline_results.csv

Regards Bill