MapData support

[esantorella]

Summary: BenchmarkMapMetric

• A MapMetric, returning MapData.
• Receives an entire learning curve and looks to the backend simulator to see whether to return partial data, if the trial has still been running or has been early-stopped.

BenchmarkMapTestFunction

• Now produces a 2d tensor rather than a 1d tensor, with the second dimension being the progression along the learning curve or time series.
• Always produces the entire learning curve or time series. If we only observe partial data, BenchmarkMapMetric will handle that.

LearningCurveTestFunction A BenchmarkTestFunction that decays smoothly to the value of some underlying base test function. This may not be interesting to benchmark, but is useful for testing. We may want to delete this.

BenchmarkRunner

• Always produces the entire learning curve or time series. If we only observe partial data, BenchmarkMapMetric will handle that.
• Now works with numpy arrays rather than torch tensors. This may be controversial, but results eventually wind up as numpy arrays, and Ax always needs numpy as a dependency and doesn't always need torch, so I figure the more we can avoid torch, the better.
• Adds IID noise to each element of a time series. We can add more sophisticated noise generators in the future.

benchmark_problem.py Adds an option to generate a LearningCurveTestFunction from a BoTorch problem. We may want to delete this if we don't want to keep LearningCurveTestFunction around.

BenchmarkTrialMetadata Now stores Ys and Ystds as numpy arrays rather than lists of floats.

Differential Revision: D64198634

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D64198634

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D64198634

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 98.47328% with 4 lines in your changes missing coverage. Please review.

Project coverage is 95.70%. Comparing base (241ad71) to head (0db4288).

Files with missing lines	Patch %	Lines
ax/benchmark/benchmark_metric.py	90.24%	4 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #3058 +/- ## ======================================== Coverage 95.69% 95.70% ======================================== Files 487 487 Lines 49351 49525 +174 ======================================== + Hits 47225 47396 +171 - Misses 2126 2129 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D64198634

[facebook-github-bot]

This pull request has been merged in facebook/Ax@0cab5d2074c347a0d7a781050b48009a783f86dd.