numericalalgorithmsgroup / dfols

Python-based Derivative-Free Optimizer for Least-Squares
https://numericalalgorithmsgroup.github.io/dfols/
GNU General Public License v3.0
41 stars 15 forks source link

Defining constraint vs. incorporating this info in the objective function #20

Closed ogencoglu closed 11 months ago

ogencoglu commented 11 months ago

Following your constrained optimization example in docs:

    min rosenbrock(x)
    s.t.
        -2 <= x[0] <= 1.1,
        1.1 <= x[1] <= 3,
        norm(x-c) <= 0.4

Is this the most efficient way of performing the optimization or would it be better to incorporate the same constraint into the objective function (with an if clause that returns inf or very large value)? What are the fundamental differences of these two methods in terms of computation and the way optimization works?

lindonroberts commented 11 months ago

DFO-LS is naturally designed to handle the bound constraints (e.g. -2 <= x[0] <= 1.1), so I would always recommend encoding them this way.

For the last constraint, I would not recommend modifying your objective function (e.g. it returns a very large value if the constraint is violated), because DFO-LS uses interpolation to function values to make progress. When you encode the constraint like this - by giving a projection function - DFO-LS is able to know a lot about the geometric structure of your constraint and uses that to adapt the optimization process. So, it would be my recommended approach for solving.