search

optimagic-dev / optimagic

optimagic is a Python package for numerical optimization. It is a unified interface to optimizers from SciPy, NlOpt and other packages. optimagic's minimize function works just like SciPy's, so you don't have to adjust your code. You simply get more optimizers for free. On top you get diagnostic tools, parallel numerical derivatives and more.
https://optimagic.readthedocs.io/
MIT License
270 stars 30 forks source link

Continue running `check_constraints_are_satisfied` beyond first failure #436

Closed hmgaudecker closed 4 months ago

hmgaudecker commented 1 year ago

What would you like to enhance and why? Is it related to an issue/problem?

check_constraints seems to return only the first failure it encounters.

I have a pipeline that runs for a fairly long time until these types of failures are encountered and they are non-trivial to predict (skillmodels running 3 out of 4 factor models individually to get start parameters with cross-factor constraints), so debugging takes quite a bit of time.

Describe the solution you'd like

As usual with "test failures" (same kind of problem), it would be great to see all constraints that are violated at once.

I must admit that with the function name (check_constraints_are_satisfied) and docstring ("Raises: ValueError if constraints are not satisfied`) I was expecting this type of behaviour.

Describe alternatives you've considered

If the above proves difficult, at a minimum change the docstring to:

Raises:
    ValueError for the first constraint that is not satisfied
janosg commented 1 year ago

Thanks for opening the issue.

In check_constraints_are_satisfied we can definitely do what you suggest. An EPP student is currently trying to make that code faster and I think improving the report could be a nice addition to the project (@Kaniyaki, we can discuss this at some point).

A good example of how to do it is this skillmodels code.

In other places it will be harder to keep running when something goes wrong. We currently raise very early so we can guarantee a reasonable error message and make more assumptions down the road, which reduces complexity in already very complex code.

But actually, I think your use-case is completely covered by these simple changes.

janosg commented 4 months ago

Solved by #450