Add more powerful solver (non-linear capability)

[nhoening]

Next to Cbc, we want to ship support for a "stronger" solver in FlexMeasures, which nonlinear constraint support. The first candidate is ipopt.

With Cbc, there is a limit to the workarounds we can think of to get our storage scheduler to function well under various conditions. The main issue we're running into is that we cannot currently forbid our scheduler to make energy disappear. The only thing that is preventing that from happening is by introducing unfavourable conditions to do so. In other words, negative prices are a problem, and workarounds can reasonably be implemented only in relatively non-complex systems.

Because we plan to do optimization on larger (neighbourhood) systems, with possibly more than one energy contract, we'll really need to include a nonlinear constraint to solve this issue once and for all, for which cbc is no longer adequate.

We know by experience that getting a solver to run on any machine is using up developer time, so hopefully we can use ipopt within the FlexMeasures Docker image (like we could with Cbc). One alternative is to start using a web service like https://neos-server.org/neos/, but that would come with adding a major extra dependency (and also an external queueing system).

[ewinnington]

HiGHS might be an interesting solver to add support for, it is developing nicely for energy optimizations. https://github.com/ERGO-Code/HiGHS Would be a simple and easy one to add to the docker image.

Also, could put CBC/HiGHS in a separate docker with an API on top to make the opt more scalable and then if you need to have a more powerful solver, an this API can interface to CPLEX or Gurobi. This could be put in place for companies that have a license for those solvers. Supporting Gurobi compute servers would allow to move the optimization to a separate server cluster, typical in an Energy company's setup.

As for Energy "disappearing" during an LP/MILP optimization, this can be dealt with by having the correct lp model with slack variables.

Happy to help discuss both these topics.

[Flix6x]

Thank you for pointing us to this, and thinking along. I'm just starting to read up on this solver, but I already found that work has been done on interfacing to the solver via Pyomo. I guess next steps could be to see:

• [ ] if the pyomo interface works
• [ ] if it can solve our LP problems
• [ ] if it can handle the constraint I have in mind to solve the issue of disappearing energy under negative prices

[ewinnington]

Do you have the equations for the https://github.com/FlexMeasures/flexmeasures/blob/main/flexmeasures/data/models/planning/linear_optimization.py documented somewhere as Latex or markdown+MathJax? If so I could take a look. Or you can just export the final equation set prior to solving a problem with pprint. I can take a look at what the issue is with your formulation. CBC is super capable and should not be causing any issues with negative prices.

I uncommented the lines

# model.pprint()
# model.display()

in the docker image of FlexMeasures, but was unable to find where the model would have been logged or printed (didn't seem to print to console the model). If I had found it I wouldn't bother you :)