Error when running 'constrained' example

[peterzh]

Hi I get the following error when running the 'constrained' example using PESC using the latest origin/PESC code. All other examples appear to work fine for me. It appears to do a single fit but then produces errors when trying to identify the next point to search.

I looked into the code and it seems model.options doesn't have a key 'binomial_trials'. Also model._one_minus_epsilon is not defined.

Any help would be appreciated, Thanks.

##@##:~/Spearmint$ python spearmint/main.py examples/constrained/

Getting suggestion for y_at_least_x, y_at_most_10, branin...

Suggestion:     
                NAME          TYPE       VALUE
                ----          ----       -----
                X             float      6.781006    
                Y             float      11.715088   
Submitted job 1 for tasks(s) y_at_least_x, y_at_most_10, branin with local scheduler (process id: 56519).
Current time: 2015-08-02 17:55:27
Status: 1 pending, 0 complete.
ID(s) of pending job(s) for y_at_least_x, y_at_most_10, branin: 1
Waiting for results...

Fitting GP to 0 data for y_at_least_x task...
Fitting GP to 1 data for NaN task...
Fitting GP to 0 data for y_at_most_10 task...
Fitting GP to 0 data for branin task...
Computing current best...

No feasible solution found (yet).

Maximum total probability of satisfying constraints = 0.05659
  Probability of satisfying       y_at_least_x constraint: 0.33181
  Probability of satisfying                NaN constraint: 0.49298
  Probability of satisfying       y_at_most_10 constraint: 0.34597

At location:    
                NAME          TYPE       VALUE
                ----          ----       -----
                X             float      6.781006    
                Y             float      11.715088   
Traceback (most recent call last):
  File "spearmint/main.py", line 514, in <module>
    main()
  File "spearmint/main.py", line 339, in main
    recommendation = chooser.best()
  File "/Users/##/Spearmint/spearmint/choosers/default_chooser.py", line 820, in best
    val_o, loc_o = self.bestObservedConstrained()
  File "/Users/##/Spearmint/spearmint/choosers/default_chooser.py", line 1242, in bestObservedConstrained
    all_constraints_satisfied = np.all([self.constraintSatisfiedAtObservedInputs(c) for c in self.constraints], axis=0)
  File "/Users/##/Spearmint/spearmint/choosers/default_chooser.py", line 1295, in constraintSatisfiedAtObservedInputs
    sat = values/float(model.options['binomial_trials']) >= model._one_minus_epsilon
KeyError: 'binomial_trials'

[JasperSnoek]

Thanks Peter, redirecting this to @mgelbart

[mgelbart]

I fixed this. Those examples shouldn't even be there... I only meant for the "toy" example to be there but they must have got copied in from the main branch. In any case the constrained one should be working now too. Let me know if there are more issues.

[peterzh]

Hi @mgelbart , thanks! One quick question: If the 'constrained' example shouldn't be there, then am I right in saying that this code is designed to handle objective functions which sometimes return NaN values? For example an objective function such as:

import numpy as np

def main(job_id, params):
    x = params['X']
    if cos(x)==0:
        return np.nan
    else:
        return tan(x)

I'm trying to optimise a black-box objective function where some parameter values give NaNs, but these constraints are unknown. Thanks!

[mgelbart]

Spearmint does work with NaNs, but the particular combination of PESC and NaNs may give less good results. I suggest you try out both PESC and EI for your problem just in case EI does better. My guess is that PESC will do much better than EI when there are no NaNs and maybe similar when there are NaNs.