embench / embench-iot

The main Embench repository
https://www.embench.org/
GNU General Public License v3.0
248 stars 101 forks source link

[fix]: KeyError Exception thrown by benchmark_size.py when missing baseline data #201

Open madhu2000u opened 1 month ago

madhu2000u commented 1 month ago

When collecting the relative results, it compares with baseline data. Recently, Depthconv benchmark was added with this commit 07282ee but a baseline data was not added to the baseline-data/size.json.

This commit does not add a baseline data for Depthconv into baseline-data/size.json but rather adds error handling in benchmark_size.py should any new benchmarks get added but a baseline data is not included.

simonpcook commented 1 month ago

If this is triggering when trying to collect data, and the baseline data doesn't get updated, I think this is a very sensible change.

One question though I'd like input from others is if this is merged should this be an error or a warning? I can see arguments either way on this.

jeremybennett commented 1 month ago

Baseline data must be present for all benchmarks. Its absence should be a catastrophic fail. We need to get the baseline measurement added. I have this on my todo list, but you can propose a value if you have access to the reference platform (STM32F407 @ 16MHz with FPU disabled).

Would you raise an issue to ensure this is dealt with promptly. Thanks.

madhu2000u commented 1 month ago

Thank you @simonpcook and @jeremybennett for your review and inputs. I do not have the reference platform to propose a value. But I have raised an issue (#202) for this as you suggested.

Also, do you have any suggestions in the code addition to handle such errors or is this good to go for a merge?

Thank you!