Support callbacks with nonlinear models

[rschwarz]

In MathProgBase and JuMP there's a distinction between solvers for LinearQuadratic vs Nonlinear (expression-based) problems. According to the documentation, the callbacks are supported only for MIP solvers (i.e., LinearQuadratic).

For SCIP, this distinction is artificial. We initially implemented the wrapper with a focus on the LinearQuadratic interface, later adding support for the Nonlinear.

Unfortunately, when building a model using @NLconstraint, adding callback functions does not have an effect with SCIP.jl currently (see example).

@mlubin: Is this a restriction inherent in JuMP or MathProgBase? Can we work around this with little effort?

@fserra: I'm confused by the implementation of Nonlinear in SCIP.jl, in particular this code:

type SCIPMathProgModel <: AbstractLinearQuadraticModel
[...]
end

type SCIPNonlinearModel <: AbstractNonlinearModel
    m::SCIPMathProgModel
end

LinearQuadraticModel(s::SCIPSolver) = SCIPMathProgModel(s.options)
NonlinearModel(s::SCIPSolver) = SCIPMathProgModel(s.options)

The type SCIPNonlinearModel is never again used in the code. Might there be a related issue here?

[rschwarz]

The MathProgBase documentation gives the signature as

setlazycallback!(m::AbstractLinearQuadraticModel, f)

So if we actually used SCIPNonlinearModel then I would believe this to be the issue. But anyway in NonlinearModel(s::SCIPSolver) we return a SCIPMathProgModel which is in fact an AbstractNonlinearModel?!

[mlubin]

Is this a restriction inherent in JuMP or MathProgBase?

It's not in MathProgBase or JuMP only because we've never put effort into supporting callbacks for nonlinear solvers. I would support tweaking the wording to make it clear that callbacks may be implemented in nonlinear models. JuMP will need a few small changes to set up the callbacks for nonlinear model, shouldn't be too difficult.

But anyway in NonlinearModel(s::SCIPSolver) we return a SCIPMathProgModel which is in fact an AbstractNonlinearModel?!

That's not good. There should be a wrapper type which implements AbstractNonlinearModel a lot of the other solver interfaces do this.

[rschwarz]

Thanks for the info, @mlubin.

JuMP will need a few small changes to set up the callbacks for nonlinear model, shouldn't be too difficult.

Sounds good, should I open an issue for this?

There should be a wrapper type which implements AbstractNonlinearModel a lot of the other solver interfaces do this.

Can you give me an example? I've looked at the code for some solver interfaces, such as IPOPT, Xpress or KNITRO, but they all only implement one type of AbstractLinearQuadraticModel or AbstractNonlinearModel and then use a NonlinearToLPQPBridge to provide also the other. Is that what you suggest for SCIP.jl also?

The issue I see is that if we use separate subtypes for AbstractNonlinearModel and AbstractLinearQuadraticModel, there would be a lot of duplicate methods, and only loadproblem! would really make this distinction.

What I'd like to do here is something like this:

abstract SCIPModel <: AbstractMathProgModel
type SCIPNonlinearModel <: AbstractNonlinearModel end
type SCIPLinearQuadraticModel <: AbstractLinearQuadraticModel end

but have the latter concrete types also be subtype of SCIPModel so that we can use SCIPModel to provide the common implemenation of functions such as getsolvetime or numvar.

I believe that multiple inheritance is not possible in Julia, but is this maybe a use case for union types?

[rschwarz]

Another thing I just noticed: The MPB doc says that setwarmstart! takes an AbstractLinearQuadraticModel. But really, it's also called by JuMP for AbstractNonlinearModel (see _buildInternalModel_nlp).

[mlubin]

The MPB doc says that setwarmstart! takes an AbstractLinearQuadraticModel. But really, it's also called by JuMP for AbstractNonlinearModel (see _buildInternalModel_nlp).

Yes, that's worth fixing. PRs are welcome.

[rschwarz]

I tried running my example above using the change in JuMP/#878 and it kind of worked:

<<< Now try with nonlinear constraints:
### lazycb is called! (1)
[src/scip/scip.c:19502] ERROR: invalid SCIP stage <3>
[src/scip/cons_linear.c:640] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:3513] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:16568] ERROR: Error <-8> in function call
Failing with retcode 1 at 118
Failing with retcode 1 at 1374
### lazycb is called! (2)
presolving (16 rounds: 16 fast, 3 medium, 3 exhaustive):
 16 deleted vars, 20 deleted constraints, 14 added constraints, 44 tightened bounds, 0 added holes, 0 changed sides, 1 changed coefficients
 9 implications, 0 cliques
presolved problem has 13 variables (0 bin, 0 int, 0 impl, 13 cont) and 6 constraints
### lazycb is called! (3)
[...]

So the callback is definitely called, but there is some SCIP stage issue (3 = transformed). I don't know yet whether that issue should be dealt with on the CSIP side. I'm guessing it is triggered by the (all-0) warmstart solution added by JuMP for NLPs.

[rschwarz]

I tried the same again, using SCIP in debug mode (OPT=dbg):

[src/scip/scip.c:247] ERROR: cannot call method <SCIPlockVarCons> in problem transformed stage
[src/scip/scip.c:19464] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:640] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:3513] ERROR: Error <-8> in function call
[src/scip/cons_linear.c:16568] ERROR: Error <-8> in function call
[...]
julia: src/scip/cons_linear.c:770: consCatchAllEvents: Assertion `consdata->eventdata == ((void *)0)' failed.
[...]
while loading /home/bzfschwa/.julia/v0.5/PipeLayout/examples/minlp_callbacks.jl, in expression starting on line 48
[...]
consCatchAllEvents at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/cons_linear.c:770
consInitLinear at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/cons_linear.c:13771
SCIPconshdlrInit at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/cons.c:2389
SCIPsetInitPlugins at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/set.c:4308
SCIPtransformProb at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/scip.c:12817
SCIPpresolve at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/scip.c:14366
SCIPsolve at /home/bzfschwa/src/scipoptsuite/scip-3.2.1/src/scip/scip.c:14581
CSIPsolve at /home/bzfschwa/.julia/v0.5/SCIP/deps/usr/lib/libcsip.so (unknown line)
optimize! at /home/bzfschwa/.julia/v0.5/SCIP/src/mpb_interface.jl:52

@fserra: Does that mean anything to you? I double-checked and I create a "fresh" SCIPSolver object for this second run. Also, the same happens when I only run the second part.

[rschwarz]

I think that the error message and the failing assertion are actually not directly related.

[rschwarz]

OK, we have a workaround for the SCIP stage problem in CSIP, then we're good.

[rschwarz]

CSIP is updated, now we only need the change in JuMP merged.

[rschwarz]

The relevant PR JuMP.jl/#894 was already merged.