search

brandontrabucco / design-bench

Benchmarks for Model-Based Optimization
MIT License
80 stars 19 forks source link

oracle.expect_normalized_y for TFBind8 #9

Closed christopher-beckham closed 2 years ago

christopher-beckham commented 2 years ago

Hi,

I am not sure as to why for various oracles in TFBind8, the field task.oracle.expect_normalized_y is set to True. I assumed that this means that we should expect that the oracle's predictions will lie in the same range as the normalised y variable, but this does not appear to be the case.

If I histogram task.oracle.internal_dataset.y then I get the following:

image

(I assume internal_dataset is exactly the dataset used to train the approximate oracle, without any extra preprocessing.)

It can also be seen here, if we plot y_i (normalised) against oracle.predict(x_i) for all values in the training set. It looks like the range of predictions for the oracle lie in [0,1]:

image

Can you clarify whether this is a bug or whether I have misunderstood what this field is meant to imply? Thanks!

christopher-beckham commented 2 years ago

Never mind, I looked at the code, it looks like the oracle internally outputs a normalised y, but it will de-normalise it for you automatically afterwards.