facebook / Ax

Adaptive Experimentation Platform
https://ax.dev
MIT License
2.38k stars 312 forks source link

Move `outcome_names` from `BenchmarkRunner` to `BenchmarkTestFunction` #3021

Closed esantorella closed 2 weeks ago

esantorella commented 2 weeks ago

Summary: Context: This will enable constructing the BenchmarkRunner based on the BenchmarkProblem and BenchmarkMethod rather than asking the user to provide it. In addition to making things simpler (it's weird that a runner is part of a problem!), that will enable the Runner to be aware of aspects of the method, such as parallelism.

This will also enable us to return metrics in a dict format ({outcome_name: value}) if we choose to do so in the future. That may be simpler since the data already gets processed into dicts by the runner.

Note that for problems based on BoTorch problems, names are usually already set programmatically, so that logic moves to the test problem.

This diff:

Update usages:

Differential Revision: D65497700

facebook-github-bot commented 2 weeks ago

This pull request was exported from Phabricator. Differential Revision: D65497700

codecov-commenter commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.61%. Comparing base (8231273) to head (68473c1).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3021 +/- ## ======================================= Coverage 95.60% 95.61% ======================================= Files 483 483 Lines 49012 49046 +34 ======================================= + Hits 46859 46893 +34 Misses 2153 2153 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

facebook-github-bot commented 2 weeks ago

This pull request was exported from Phabricator. Differential Revision: D65497700

facebook-github-bot commented 2 weeks ago

This pull request was exported from Phabricator. Differential Revision: D65497700

facebook-github-bot commented 2 weeks ago

This pull request was exported from Phabricator. Differential Revision: D65497700

facebook-github-bot commented 2 weeks ago

This pull request was exported from Phabricator. Differential Revision: D65497700

facebook-github-bot commented 2 weeks ago

This pull request has been merged in facebook/Ax@ffedab0ccb7c1be53e3e04ba9acbb0fad96a56be.