handle evaluation error

[jschueller]

the current code makes the assumption that the objective is always well defined however in real life the evaluation may fail (finite difference solver convergence, numerical error etc)

I dont think prima can handle that yet, for example at OpenTURNS our cobyla version (translated in C from the old fortran 77 version) can end the minimization according to a boolean returned by the objective: https://github.com/openturns/openturns/blob/master/lib/src/Base/Optim/algocobyla.c#L310

it didnt occured to me when I wrote #69, but the termination boolean is needed the objective function level (the one at #69 can stay as we can have a termination bool not related to the objective function call, ie the user clicks stop in a GUI)

so I propose to add a boolean to the OBJ, OBJCON Fortran callbacks similarly to what is done in #69 probaby a lot of the example/test/matlab code has to be changed because of this what do you think ?

[zaikunzhang]

Hi @jschueller Julien,

Thank you for raising this extremely important point.

I fully agree that function evaluation failures should be handled properly. This is vital for the robustness and reliablity of the solvers.

Indeed, this point has been mentioned in the previous discussions with you about how to test PRIMA and its interfaces / translations to other languages. Maybe that point did not catch your attention since we had many things to discuss. See "2. TOUGH tests" of "What kind of tests are sufficient for the porting or translation of PRIMA?" (BTW, I highly recommend you to read all the "discussions" I wrote on PRIMA and also write down your thoughts as discussions, so that we can understand each other better, and future contributors will understand us better).

The "TOUGH test" is a test on problems where function evaluations fail or return exceptional values randomly. See slide 23 of my talk at ICIAM 2023 for details. The following MATLAB code implements a generator of such problems:

https://github.com/libprima/prima/blob/main/matlab/tests/private/tough.m

The handling of function evaluation failures (and exceptional function values) has been implemented in the MATLAB interface:

https://github.com/libprima/prima/blob/main/matlab/interfaces/private/evalobj.m https://github.com/libprima/prima/blob/main/matlab/interfaces/private/evalcon.m

The idea is to

1. catch the failure (and raise a warning) if it occurs, and then take NaN as the function value.
2. after 1 is done, apply a moderated extreme barrier to handle the exceptional values, basically NaN and Inf (see Sec. 4.5 and 5.2 of my paper with Tom @ragonneau on Powell's methods).

The Fortran implementation, however, only handles exceptional function values but not evaluation failures, as Fortran does not handle exceptions well (no try ... catch ...):

https://github.com/libprima/prima/blob/main/fortran/common/evaluate.f90

I will not implement handling of evaluation failures in the Fortran implementation unless there exists a simple and straightforward implementation. This is due to the limitation of the language --- its incapability of handling exceptions (it should be handled by the caller, which might be an interface to another language, e.g., MATLAB, Python, C, ...).

Meanwhile, in the interface / translation of PRIMA to any language with a built-in mechanism of handling exceptions, evaluation failures and exceptional function values must be handled in a way similar to the MATLAB interface. In addition, the interface / translation should eventually implement and pass the TOUGH test, as the MATLAB interface has achieved. I insist this (indeed, I enforce this on all solvers I develop, see, e.g., https://github.com/blockwise-direct-search/bds/issues/6). Without passing the TOUGH test, the interface / translation is not complete.

Thanks.

[jschueller]

ok, so objective function errors results in f=funcmax, some penalizing value

but the nan value can be used a a simple mean to indicate failure

so I think it should be done in the Fortran layer; we can check for nan there and set funcmax

that way it will benefit all the interfaces as well (and not duplicate that logic)

[zaikunzhang]

ok, so objective function errors results in f=funcmax, some penalizing value but as you mentionned the nan value (or some bool) can be used a a simple mean to indicate failure so I think it should be done in the Fortran layer (we can check for nan there and set funcmax) that way it will benefit all the interfaces as well (and not duplicate that logic)

Yes, this Fortran layer does it: https://github.com/libprima/prima/blob/main/fortran/common/evaluate.f90#L62-L64

However, I do believe the same should still be done in the evalobj / evalcon of the interfaces (MATLAB, Python, C ...). Why? Think about the future native implementation in these languages ---- such an implementation will likely share the evalobj / evalcon with the interfaces. At that time, we cannot rely on what is done in the Fortran implementation anymore.

[jschueller]

oh, then there is nothing to do in the fortran layer

[jschueller]

I modified the Python interface to return nan on exception in the C layer, maybe we could mention it in prima.h