Expression evaluator for complex logic

[SouthEndMusic]

At the 28-6-2023 samenwerkingsdag we came to the conclusion that the user should have some scripting ability for control, rather than us hardcoding many control logic frameworks which all do not quite cover the demand.

I think the user expressions should not incorporate conditions, because this would lead to quite complex parsing where new callbacks have to be added to find changes of these conditions. So the user expressions should only consist of algebra.

Since these conditions are still desirable in the control logic, the user expressions could themselves be considered control states which are evaluated at each time step, as opposed to our current static control states.

Regarding the syntax of the expressions: we could either let the user write their own Julia functions (or maybe even Python functions used in the Julia core) passed as a string, but then we have to use eval which is of course evil, so that might require a lot of validation. Or we could introduce our own syntax and parsing, but that might be a lot of work.

[Hofer-Julian]

we could either let the user write their own Julia functions passed as a string, but then we have to use eval which is of course evil

To elaborate on that: this would be straightforward to implement and the most powerful option. But depending on the situation, it would be a security risk, and I am not sure if we would be able to validate properly

[gijsber]

I think we will have two types of use case:

1. expressions that result in a new calculated setpoint value
2. expressions that result in a condition that is met resulting in some parameter setting.

I am waiting for some exact example but I foresee the following kind of rules: example 1 Haringvlietsluizen:

If ((0.33*Q.noord +Q.hij)>1500 then (tabulatedQH.haringvlietsluis=datasetA)
elseif ((0.33*Q.noord +Q.hij)>800 and H.extern_nz<1) then (tabulatedQH.haringvlietsluis=datasetB)
elseif ((0.33*Q.noord +Q.hij)<=800 and H.extern_nz<1) then (tabulatedQH.haringvlietsluis=datasetC)
elseIf ((0.33*Q.noord +Q.hij)<=800 and H.extern_nz>1) then (tabulatedQH.haringvlietsluis=datasetD)
else: tabulatedQH.haringvlietsluis=datasetE
where noord and hij are internal locations and extern_nz is an external bounadry

example 2 virtual observation Krimpen aan de IJssel and resulting setpoint for de lek at weir Hagesteijn:

Chl.kij = 0.875*Q.noord+0.05*Q.hij+0.25* Q.lek+5.35*H.extern-nz
Q_setpoint.hagesteijn=350*Chl.kij

Note that the setpoint is on the same river so it has an indirect (ie delayed) correlation to the Chl.kij

example 3 Irenesluizen/stuw driel.

if (H.irenesluizen < 0.2m and Q.bernardsluizen <10m3/s and Q.lobith >1000m3/s) 
then Q_setpoint.gate.driel = Q_setpoint.gate.irenesluizen + 30m3/s and Q_setpoint.pump.bernhardsluizen=0 and gate.bernhardsluizen=open
elseif ((H.irenesluizen < 0.2m and Q.lobith <1000m3/s) 
then (Q_setpoint.driel = Q_setpoint.irenesluizen + 20m3/s) and Q_setpoint.bernhardsluizen =30m3/s)
else Q_setpoint.driel =30m3/s and gate.bernhardsluizen=open

[SouthEndMusic]

Here's my concept for the expression evaluator.

Expression syntax

@Hofer-Julian had a nice idea for the expression syntax for 'compound variables': use postfix notation, which is easily parsable. The user can input an expression in standard algebra notation and this can then be converted to postfix notation internally. We then have to specify how variables are denoted, maybe simply by node.variable[node_id] or some additional syntax to recognize variables. Then we also specify a list of operators *,/,+,-,....

Expression input

These compound variables are not associated with any particular node type, so I am not sure where this table will fit in, but I suggest this structure:

Name	Expression
level_average	(basin.level[1] + basin.level[2])/2

which is then in the Julia core pre-simulation converted to something like

expression = ("basin.level[1]","basin.level[2]", function_add, 2.0, function_divide)

or whatever equivalent is fastest to process.

Expression parsing and usage

I propose a function

function compound_variable(p::Parameters, expression::Tuple)::Float64

with the parsing logic. I'm not sure whether we want to compute all compound variables at each time step and store them to prevent multiple evaluations, or just compute them when they are needed.

• Example 1: These compound variables are used in the conditions for discrete control (storing the compound variables as mentioned above might not work here).
• Example 2: It has to be specified that the value of a variable must be set to the value of some compound variable, with a function, say

  function set_variable!(p::Parameters, storage::Vector{Float64})

  called at the beginning of water_balance!.

• Example 3: This requires a combination of the above. Essentially it needs non-constant control states, which can then be handled by set_variable!. Maybe it is nice in the Julia core to have a variable for each node which says how its parameter values are determined, which is either 'static', 'time', or the name of some compound variable, which can be picked up by set_variable!.

[visr]

I believe this image speaks for itself.

[visr]

Useful prior art for the parsing part: https://github.com/Deltares/mathematical_expression_parser

[gijsber]

Nice. That brings me to the idea that on the data side we could think of a separation of the formula max((A+B)C, min(DE)) and the mapping of A,B,C,D, E to actual timeseries, states or coefficients from an (external) table

[gijsber]

Postponed till we can adopt MTK again

[gijsber]

the current implementation checks one model variable against a value. In a first step we should accommodate that we allow multiple variables together to be checked against a value, where we leave the AND/OR structure as is

• [ ] decide where the formulas are to be defined (in the TOML or the geopackage)
• [ ] create formula parser
• [ ] create function to apply math to multiple model variables
• [ ] creating mapping structure (in TOML?) to map coefficicent values from an external geo-package table to a coefficient used in the formula

[SnippenE]

Won't fix. not needed at this moment