search

ukri-excalibur / excalibur-tests

Performance benchmarks and regression tests for the ExCALIBUR project
https://ukri-excalibur.github.io/excalibur-tests/
Apache License 2.0
18 stars 15 forks source link

[sombrero] Simplify classes used in benchmarks #239

Closed giordano closed 9 months ago

giordano commented 9 months ago

Address https://github.com/ukri-excalibur/excalibur-tests/issues/237#issue-2010042384. @mirenradia Can you please check this solves the issues you reported in the first post?

mirenradia commented 9 months ago

This resolves all of the issues I mentioned in #237 except for the num_tasks problem and I can now run the ITT-sn benchmark (after manually setting num_tasks appropriately). However, I do get the following warning:

WARNING: skipping test 'SombreroBenchmarkScaling': test has one or more undefined parameters
giordano commented 9 months ago

I can reproduce that, as far as I understand it comes from the fact case_filter.generate(scaling_config) in https://github.com/ukri-excalibur/excalibur-tests/blob/40b4f4a874010ebfd822a46d08fcf1975020496d/benchmarks/apps/sombrero/sombrero.py#L103 is an empty list [], and so the parameter params is undefined. I'm not entirely sure how that benchmark is supposed to work, it was contributed by @mmesiti, maybe he can shed some light on this?

In any case I'm going ahead and merge this PR, as it addresses the previously reported issues.