Getting stuck after raised ValueError.

[ili3p]

When a function evaluation doesn't complete (error in the external objective function or some other issues), I raise ValueError. However, after raising ValueError pySOT doesn't continue. I assume is getting stuck in some infinite loop, since the process continue to use the CPU 100%. I left it for several hours and didn't change.

To reproduce follow these steps:

virtualenv venv_test
. ./venv_test/bin/activate
pip install numpy
pip install inspyred
git clone git@github.com:dme65/pySOT.git
cd pySOT/
# edit the Ackley class so it raises ValueError
vim pySOT/test/Ackley.py   
python setup.py install
# edit test_simple.py so it uses the new Ackley
vim pySOT/test/test_simple.py
cd pySOT/test/
python test_simple.py

Output:

(venv_test)ilija@deep03:~/for_test/pySOT/pySOT/test$ python test_simple.py

Number of threads: 1
Maximum number of evaluations: 1000
Search strategy: CandidateDYCORS
Experimental design: Latin Hypercube
Ensemble surrogates: Cubic RBF, domain scaled to unit box
10-dimensional Ackley function
Global optimum: f(0,0,...,0) = 0
[-12.61363636  -4.65909091   6.47727273  -9.43181818   9.65909091
 -12.61363636   8.06818182  -4.65909091  -6.25        16.02272727]
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "build/bdist.linux-x86_64/egg/poap/controller.py", line 426, in run
    self.handle_eval(record)
  File "build/bdist.linux-x86_64/egg/poap/controller.py", line 453, in handle_eval
    self.finish_success(record, self.objective(*record.params))
  File "/home/ilija/for_test/pySOT/pySOT/test/Ackley.py", line 19, in objfunction
    raise ValueError('Dimension mismatch')
ValueError: Dimension mismatch

I have attached the modified Ackley.py and test_simple.py.

Archive.zip

[dme65]

If a function evaluation fails the record should be marked as a failure via finish_failure, so what I think happens is that POAP is waiting for you to either mark the record as a success or as a failure, but that won't happen since you raised an exception that no one is going to catch. The dimension missmatch exception that is being raised in some of the test problems will be removed in the next version since the input checking is now taken care of by the strategies and the exception was never taken care of anyway.

I'm in the process of making the strategies more flexible so that they don't keep trying to evaluate the same input over and over again until the evaluation completes, but rather gives the user more power when it comes to dealing with failed evaluations.

[ili3p]

I'm in the process of making the strategies more flexible so that they don't keep trying to evaluate the same input over and over again until the evaluation completes, but rather gives the user more power when it comes to dealing with failed evaluations.

Great! I'll open an issue for this as a feature request, so I get notified when you are done with it.

It would be also great if you can add the ability to save the state of the program after every function evaluation, so if a server crash or similar, I'm able to resume the program like nothing happened. I'll open another feature request for this as well.

Thanks a lot for everything, it really is a very useful and well designed toolbox.