Handle finitely many arbitrary convex constraints with DFO-LS

[mjhough]

TODO

• [x] Write tests
• [x] Update documentation
• [x] Include example problems

Note 1 [See update below]

Often get model increase error when optimal point is where two constraints intersect. But is at correct solution, so not really an error. The current example for arbitrary constraints has this problem. In the docs I edited the output to what it should look like. That is, instead of showing error, it shows success.

Update in response to Note 1

Now when using multiple projections and get a model increase error from our TR subproblem, output something like the following:

****** DFO-LS Results ******
Solution xmin = [0.9        1.15359245]
Residual vector = [3.43592448 0.1       ]
Objective value f(xmin) = 11.81557703
Needed 10 objective evaluations (at 10 points)
Approximate Jacobian = [[-1.79826221e+01  1.00004412e+01]
 [-1.00000000e+00 -1.81976605e-15]]
Exit flag = 5
Warning (trust region increase): Either multiple constraints are active or trust region step gave model increase
****************************

[lindonroberts]

I think for the model increase error, the safest thing would be to change the error output to "model increase or multiple constraints active" (rather than just model increase), to alert the user that there is a valid reason for this output

[lindonroberts]

I've had a look at the documentation and example code, and have a couple of suggestions:

• Example rosenbrock_constrained.py: add a comment in the projection functions to say explicitly the set/constraint you are projecting into, and somewhere at the top of the file include the full problem description (as it's not as obvious for the constrained problem as the other examples). Can you please also add a comment explaining that bound constraints can either be added as a projection or explicitly like before?
• On the main documentation page and overview page, please explain that the problem formulation includes bound constraints and intersections of multiple convex sets
• On the userguide page, change "Adding constraints and more output" to "Adding bounds...". Can you please add a section with your ball/box example to demonstrate the convex constraints functionality? Again, it would useful to note that bound constraints can be added as either projection or explicitly like the earlier section on that page
• Version history: can you please add that there is no change to the solver for unconstrained/bound constrained problems (i.e. only adding new functionality)?

A question that has (sadly) only just occurred to me: what do you expect to happen if you feed in inconsistent projections (i.e. projections to non-intersecting sets)? I assume Dykstra will fail to find a good location for x0, but what does DFO-LS return in this case?

Also a note for later, that we should add suitable citations to our paper once we have an arXiv link. I also need to create a contributors page to make sure you get suitable credit!

[mjhough]

I believe I have updated the docs etc with all those suggestions now.

Regarding what happens when the intersection of sets is empty, the output of DFO-LS is unpredictable. Dykstra assumes the intersection is non-empty.

[lindonroberts]

Thanks Matthew! For reference, I have now created release 1.3, which should be able to be downloaded using pip now. I made some small changes:

• Fix syntax error in Python 2.7 coming from projections.copy(). This syntax was introduced in Python 3.
• Added module-level logging to give users more control over log output