Active Kriging Monte Carlo Simulation

[JCSadeghi]

Hi emukit team,

I am considering implementing results from this paper https://www.sciencedirect.com/science/article/pii/S0167473011000038 in emukit. I don't think it's too different from your quadrature implementation.

The main additions would be:

1. probability of misclassification aquisition function
2. integrate the GP over an arbitrary probability density using Monte Carlo simulation
3. use the Monte Carlo simulation to define an exit criterion for the learning loop (as in the paper)

I noticed that most of your tutorials (e.g. sensitivity analysis) are using uniform random variables for the inputs, and #130 has already discussed integrating over a normal density. I'm wondering if you have any suggestions/plans for the best way to achieve 2. Perhaps I should make use of the distributions from numpy?

Any advice is much appreciated,

Kind Regards, Jonathan

[javiergonzalezh]

Hi Jonathan,

Thanks a lot for your interest! Absolutely, it would be great having these points implemented in Emukit. This is a possible way to do it but I am sure that @mmahsereci and @apaleyes have some interesting ideas too.

1 and 2 are quite generic so they can go into the main library. 1 seems easy and can be implemented as an acquisition. 2 would be, indeed, very nice to have, as this would allow to compare Monte Carlo methods with Bayesian quadrature (BQ). This CR from @mmahsereci already creates the abstraction for BQ although only the uniform measure is implemented so far. Using a similar abstraction for Monte Carlo would be nice. Using numpy distributions should fine, I think.

For 3, I recommend to use the examples folder at it is a bit more specific and we don't know yet how it fits with the rest of the library. We can always integrate it later.

@mmahsereci @apaleyes do you think that this make sense?

[apaleyes]

Sounds good. I agree that pt.3 should be separated in the examples, but seems like it may graduate to the main library pretty quickly, although that may require a bit of restructuring in the package.

[JCSadeghi]

Thanks for your input - I'm working on some other projects at the moment, but I will make a prototype when I have some spare time 👍

[mmahsereci]

Hi Jonathan, sounds good! I had a look to the paper in the meantime. It seems most ideas are fairly general in the sense that they could be used, e.g., for experimental design as well. If you don’t mind, I would like to get your feedback on a few points to make sure I understand the method.

• the integration (pt.2) you mentioned is the integration of the indicator function of failures, predicted by the GP mean, i.e., it is not the integral of the GP itself. Is my understanding correct here?
• the active learning scheme is fairly agnostic towards its goal (i.e., estimating failure rate, in contrast to e.g., learning the failure function) apart from the stopping condition (coefficient of variation) where the failure rate explicitly appears.

Please let me know if I misunderstood. Thanks again for your input.

[JCSadeghi]

1. Yes, that is correct - the GP mean represents the so-called performance function (g(x)) of the system. We integrate the indicator function (g(x)<0) with Monte Carlo simulation. There is a big community who wish to solve problems of this form, but currently most available methods are only implemented in MATLAB/legacy code/closed source (see proceedings of ICOSSAR, ICVRAM, ESREL conferences, for nice applications).

2. The probability of misclassification acquisition function is used because it has been shown to be a good way to accurately learn the indicator function. It is generally accepted in the community that if the indicator function is known and can be evaluated cheaply, via a surrogate, then in most cases a brute force monte carlo simulation will yield a satisfactory solution (see e.g. first order reliability method or importance sampling for intuition). Other researchers have defined more "well motivated" acquisition functions which behave in a similar way.

For context, g(x) is usually expensive to evaluate - in many cases it can take days for each evaluation. Also for most systems in practice, the failure rate is usually very small so a huge number of samples are required for standard MC simulation without a GP emulator.

Another advantage of using a GP emulator is that one can then solve the "Reliability Based Design Optimisation" problem.

I hope this explains why I am interested in an implementation!

[mmahsereci]

Yes, that explains. Thanks. Also for the pointers in 1.