facebook / Ax

Adaptive Experimentation Platform
https://ax.dev
MIT License
2.38k stars 312 forks source link

Validation and narrower try/except in BenchmarkMetric #3100

Closed esantorella closed 18 hours ago

esantorella commented 21 hours ago

Summary: Context:

BenchmarkMetric.fetch_data originally had a broad try/except block just because this the usual way to define metrics. The try/except is actually not needed at all; with benchmarking, there are no potentially-flaky RPCs and we completely understand what should happen when fetching data. BenchmarkMapMetric has no try/except. The following cases are possible when fetching data:

This PR:

Differential Revision: D66301697

facebook-github-bot commented 21 hours ago

This pull request was exported from Phabricator. Differential Revision: D66301697

codecov-commenter commented 20 hours ago

Codecov Report

Attention: Patch coverage is 96.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 95.43%. Comparing base (5d1e441) to head (6d512ca).

Files with missing lines Patch % Lines
ax/benchmark/benchmark_metric.py 94.11% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3100 +/- ## ========================================== + Coverage 95.41% 95.43% +0.01% ========================================== Files 495 495 Lines 49941 49956 +15 ========================================== + Hits 47653 47674 +21 + Misses 2288 2282 -6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


🚨 Try these New Features:

facebook-github-bot commented 19 hours ago

This pull request was exported from Phabricator. Differential Revision: D66301697

facebook-github-bot commented 17 hours ago

This pull request has been merged in facebook/Ax@095d7ed2b59716ec10e387b0705a7bd90547d869.