NEI not outperforming EI?

[EvanClaes]

Issue description

I'm evaluating the performance of NEI vs EI on two in silico bioprocess models. When there is no noise on the observations, EI clearly outperforms a random exploration and a simple GP maximization algorithm (the maximum expected value of the GP is taken as the next query). However, when I add noise (Gaussian with standard deviation of 1% the observed value), the performance of both EI and the GP max algorithm becomes equal to random exploration.

So I tried NEI (with a fixed noise GP), but this did not improve the performance (still equal to random exploration). Is there anything I can check to see what is going wrong?

I also checked the q-NEI tutorial to start from there and then adjust towards my case, but the figure at the end there also seems to show that NEI is not outperforming EI...?

[eytan]

Are you using the default model? How are you normalizing your inputs? Is it possible that you are injecting more noise than you think you?

It might be helpful to compare the GP model fits w/ and w/o the noise. If the GP ends up just predicting the mean then I would expect most algorithms to end up doing random exploration. https://botorch.org/tutorials/batch_mode_cross_validation has some code for doing cross validation / ocular tests of the GP fit.

e

On Fri, Sep 9, 2022 at 8:55 AM EvanClaes @.***> wrote:

Issue description

I'm evaluating the performance of NEI vs EI on two in silico bioprocess models. When there is no noise on the observations, EI clearly outperforms a random exploration and a simple GP maximization algorithm (the maximum expected value of the GP is taken as the next query). However, when I add noise (Gaussian with standard deviation of 1% the observed value), the performance of both EI and the GP max algorithm becomes equal to random exploration.

So I tried NEI (with a fixed noise GP), but this did not improve the performance (still equal to random exploration). Is there anything I can check to see what is going wrong?

I also checked the q-NEI tutorial https://botorch.org/tutorials/closed_loop_botorch_only to start from there and then adjust towards my case, but the figure at the end there also seems to show that NEI is not outperforming EI...?

— Reply to this email directly, view it on GitHub https://github.com/pytorch/botorch/issues/1391, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAW34LUIHFDLMADPJT6IRTV5MXVXANCNFSM6AAAAAAQIU7MSE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[EvanClaes]

Hello Eytan,

Thanks for your feedback, it helped. The noise specification was indeed the problem. I was giving the model the noise of the observed values, not of the standardized ones. I properly scaled the noise and now it is working perfectly.

PS: Any idea why the tutorial is not showing the NEI as better than EI?

[eytan]

That’s a good question. Perhaps the noise level isn’t very high? The standard errors are also very wide, so there aren’t many replicates. 10 random restarts can also be a bit low especially in the constrained setting. So there might be a larger gap if one increases either of those numbers.

On Fri, Sep 9, 2022 at 2:10 PM EvanClaes @.***> wrote:

Hello Eytan,

Thanks for your feedback, it helped. The noise specification was indeed the problem. I was giving the model the noise of the observed values, not of the standardized ones. I properly scaled the noise and now it is working perfectly.

PS: Any idea why the tutorial is not showing the NEI as better than EI?

— Reply to this email directly, view it on GitHub https://github.com/pytorch/botorch/issues/1391#issuecomment-1242311729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAW34MZAXWUJELJP72XSNDV5N4SNANCNFSM6AAAAAAQIU7MSE . You are receiving this because you commented.Message ID: @.***>

[Balandat]

Yeah I think it's just too noisy to say much - and running a proper comparison with a lot of iterations is a bit too costly to run as a tutorial. Maybe we can clarify this in the text of the tutorial and point folks to other evaluations we've done as part of papers?

[esantorella]

Closing in favor of #1861 for clarity