facebook / Ax

Adaptive Experimentation Platform
https://ax.dev
MIT License
2.38k stars 312 forks source link

Implement configure_runner, configure_metric #3104

Open mpolson64 opened 16 hours ago

mpolson64 commented 16 hours ago

Summary: configure_runner and configure_metric allow users to attach custom Runners and Metrics to their experiment.

configure_runner is fairly straightforward and just sets experiment.runner

configure_metric is more complicated: given a list of IMetrics it iterates through and tries to find a metric with the same name somewhere on the experiment. In order it checks the Objective (single, MOO, or secularized), outcome constraints, then tracking metrics. If no metric with a matching name is found then the provided metric is added as a tracking metric.

Differential Revision: D66305614

facebook-github-bot commented 16 hours ago

This pull request was exported from Phabricator. Differential Revision: D66305614

codecov-commenter commented 16 hours ago

Codecov Report

Attention: Patch coverage is 94.96021% with 19 lines in your changes missing coverage. Please review.

Project coverage is 95.77%. Comparing base (4aa1d65) to head (c7b8aeb).

Files with missing lines Patch % Lines
ax/preview/api/client.py 81.81% 12 Missing :warning:
ax/preview/api/utils/instantiation/from_string.py 94.11% 4 Missing :warning:
ax/preview/api/utils/instantiation/from_config.py 94.23% 3 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3104 +/- ## ========================================== + Coverage 95.43% 95.77% +0.33% ========================================== Files 495 500 +5 Lines 49956 50316 +360 ========================================== + Hits 47674 48188 +514 + Misses 2282 2128 -154 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


🚨 Try these New Features: