[epic] Build benchmarks

[mastoffel]

Build a set of simulators/datasets to benchmark autoemulate. Let's aim at having 2-3 datasets from 3 different fields. If we only include small-medium sized datasets, it should be ok to include them in autoemulate.

• [x] biomedical (cardiac)

• [ ] engineering

• [ ] climate change (putatively)

• [x] #136

[kallewesterling]

What are the steps needed here? How can we bring this to a finish line over the next three sprints? (Or however many sprints we think we'll need for this issue.)

[mastoffel]

What are the steps needed here? How can we bring this to a finish line over the next three sprints? (Or however many sprints we think we'll need for this issue.)

Added some more specifics. Should be doable in one sprint.

[kallewesterling]

Great - thanks! I think we still need to break it down into several issues, unless this is not an epic but rather a single issue now that it has been refined? Let's chat about it in today's planning meeting.

[kallewesterling]

An example for how to write something for loading data, if we need it (i.e. if there are bigger data packages that we need to load) can be found in the DeepSensor codebase. I liked how @tom-andersson made it easy to load external datasets for the package.

[aranas]

Great that we have multiple datasets across domains that can be easily loaded, is the actual benchmarking also documented somewhere already?

[aranas]

notes from aug 12 discussion dimensions to benchmark

• fit
• stability (eg of eigenvalues)
• rank order of parameters

[aranas]

potential dataset for benchmarking for engineering application https://www.researchgate.net/publication/380289575_A_Comprehensive_Dataset_of_the_Aerodynamic_and_Geometric_Coefficients_of_Airfoils_in_the_Public_Domain