Getting started

[javdrher]

All right, lets get started.

As you may be aware, I posted a demo prior to the approval of the project here: https://github.com/javdrher/optdemo In the meantime I also added tests for that, achieving 99% test coverage according to the xcoverage plugin. My initial idea was to commit this demo as a project start and configure the repository in the same style as GPflow (protected master branch, contributions through pull requests, with status checks for successful tests and code coverage). Next steps would be documentation, improving some of the tests, solve some issues and then decide on what features to add next.

However, I'd be happy to hear any comments and suggestions, regarding things you like and dislike about the demo so we start from something everyone feels comfortable with. I'm aiming to further initiate the repo somewhere next week.

[javdrher]

@1Reinier , I just noticed your issue in the demo repository. I coded it with python 3.5 and TF 1.0.1 so I'm guessing some renaming issues are to be expected. My idea was to maintain compatibility with the TF version currently supported by GPflow as this project will naturally depend on it.

Did the cholesky decomposition faillure come from _optimize_all in the Acquisition class? It's an issue i ran into during development. It's caused by optimizing the hypers again, after adding for instance a new data point. Currently I locally solved it by increasing the likelihood variance, however I noticed that this isn't a very stable solution when changing for instance to a different python version. It's one of the issues with the demo I'm aware of which needs a fix

edit: also I remember a paper stating that starting from the hyperparameters found in the previous iteration is actually not a good practice in BO, keeping that in mind, completely resetting the hypers might be a better approach.

[1Reinier]

Yeah keeping TF 1.0 support makes sense. Perhaps update this as soon as GPflow ports to TF 1.1+. I think the Cholesky issue did indeed pop up in acquisition. There were also some object member issues, where it was locating an acquisition object on some other object. Can send you the traceback later.

Overall I think the demo is a very good starter for GPflowOpt! Code quality seems better than GPyOpt for sure.

[yangvz]

I think it'll be better if the framework was more flexible. In particular:

• Allow ask-and-tell, as in skopt. That is, you can run the black-box function somewhere else, and feed the result back in to the BO model to get the next point.
• Support correlated constraints in addition to domain constraints. For example, x1+x2=1.
• Support qEI, where the BO model possibly gives you multiple points for the next (simultaneous) run of the black-box function.

[icouckuy]

Thanks for the suggestions.

• ask-and-tell (I call it the human simulator. i.e., it can also be time-consuming measurements) is definitely an interesting use case that will be considered. At the moment we just want to get a first stable release as soon as possible to invite collaboration.

As a sidenote you have the 'optimization as a service' approach as in https://github.com/Yelp/MOE

While interesting, I think this would fall out of the scope of GPFlowOpt.

• expensive constraints (probability of feasibility, expected violation, etc.) and 'cheap' constraints (passed to the scipy optimizer) are on the roadmap
• similarly we are thinking about more criteria like qEI. Note that qEI increases the dimensionality of the optimization a lot (and isn't analytically for >2 points). Alternatively we can think also about a simple multi-start local optimizer to find multiple local optima.

[yangvz]

@icouckuy

To be fair, MOE isn't maintained by me :P, and I agree that it's a bit of an overkill to be a implemented as a service. qEI with q>2 is analytically tractable, but unrealistically expensive to compute. Simple heuristics like KB and CL can be implemented first, which is nothing but q runs of EI optimizations. Also, it's interesting too that multi-start optimizer can be parallelized.

I will look forward to the first release.

[javdrher]

hi @yangnw, thanks for your interest and feedback!

As you have noticed we are currently in the process of stabilizing the code and making sure its properly tested (and they pass). Right now there is some code for expensive black-box constraints, but the white-box constraints you mention are to be added. My idea was to incorporate this into the domain class, and have the optimizer classes translate it into the optimization (constraint functions for scipyoptimizer, rejection sampling for MCOptimizer etc)

I had interest to look into the multi-step approaches if people showed interest in them, which seems to be the case ;) . You raise an interesting point about the parallel multi-start optimization. That would definitely be cool, and it could be achieved in the StagedOptimizer by (1) extend it to pass multiple solutions to the next stage (2) support parallel optimization in optimizers. The only risk is some rather low-level code to get multiprocessing going, and it might be slightly non-trivial to test properly.