Investigate causes of pull request relating to objective quadratic function for allocation

SnippenE commented 5 months ago

See #1383

visr commented 5 months ago

Why is this needed in addition to #1383?

SouthEndMusic commented 5 months ago

Here's some history of the pull request in question.

The equal ratio allocation objective

This objective function was first introduced in https://github.com/Deltares/Ribasim/pull/1366, where @jarsarasty also took part in the discussion. This is quite a desired feature, as it results in the allocation results that were expected by the members of the TKI project, namely: allocate if possible to each demand within a priority the same fraction of that demand.

The problems with this objective

The above linked first PR for this objective function was merged, but then it turned out there were some problems:

A weird bug in the HiGHS code that prints dimension mismatch but doesn't throw an error (https://github.com/Deltares/Ribasim/issues/1383);
The allocation tests were failing seemingly randomly.

The first point is not a big issue, but the second is. It seems that the outcome of the optimizations is not deterministic. This is strange behavior as there is no source of randomness in the code.

Insufficiently constrained problems

Although there is no source of randomness in the code, there can be situations where the optimization problem solved by HiGHS does not have a unique solution. Therefore it could be that a test is based on one solution, but sometimes another comes out.

I find it unlikely however that this is mathematically related to the new objective function, as these unconstrained problems are also possible with the current one. My current hypothesis is that the solution method used by HiGHS for the new objective function (which is quadratic) is sufficiently different from the current one (which is linear) for this new behavior to occur.

Many efforts have been made to constrain the solution of the optimization problems, see for instance https://github.com/Deltares/Ribasim/pull/1429.

What's left of possibilities for unconstrained optimization problems

The core of the current problems with constrainedness is with subnetworks (see the allocation documentation for an explanation of the role of subnetworks). We want to aggregate the demand of a subnetwork at the main-to-subnetwork inlet(s). We currently do this by doing an optimization within the subnetwork where this inlet is the only source, and reading of the flow at the inlet. For this to work properly, we have to make sure that the aforementioned inlet is indeed the only source in the subnetwork. Subnetworks can however have internal sources, so therefore before aggregating the subnetwork demand we first use up all the capacity from the internal sources.

The problem with this is that internal sources can be created by the optimization for a certain priority, and then the demand aggregation for the priorities afterwards does not work anymore (and the solutions are not uniquely determined anymore). So a recent idea is to not use the capacity of internal sources first (for all priorities) and then do the aggregation (for all priorities), but to interweave these steps.

A related concept: source priorities

One of the open allocation issues is that we want a given preference ordering over sources to be taken into account for allocation, see also https://github.com/Deltares/Ribasim/issues/565. I mention this here because it can be a potential solution to obtain unique solutions to the optimization problems, if every source is optimized for individually in a prescribed order. This can lead to possibly (many) more optimization calls, but since the whole allocation algorithm is called much less often than the timestepping of the physical layer of Ribasim this is probably not a significant performance bottleneck.

visr commented 5 months ago

Fixed by #1386.

Deltares / Ribasim