Support observations

[DanielTollenaar]

@visr as a user I would like to add a layer (overlay) of observations over my Ribasim-model. On these locations I would like to show simulated and observed level and flow_rate in one graph using the Ribasim and iMOD QGIS plugins. Rough Idea is to:

• [ ] add a layer Observation with columns name (add to title), edge_id (to read flow_rate from flow_rate.arrow), node_id (to read level from basin) and observation_id (read level and/or flow_rate from observation)
• [ ] add an observation.arrow with observations with a similar structure as flow.arrow and basin.arrow

Please add your thoughts on this and/or share. The part of adding a layer I could (help to) write. On how to achieve interaction with iMOD plugin I am unsure.

[visr]

How about an Observation table with the following columns:

column	example
name	Driel
node_type	Outlet
node_id	145
variable	flow_rate
time	2020-01-01 12:30
observed	1200

I think this gives us enough flexibility to put any timeseries in there. I want to get away from referring to flow on edge IDs, see https://github.com/Deltares/Ribasim/issues/1597.

If we want to support adding results (observation.arrow), we could:

1. Make sure the simulation stops at all observed time points (add tstops)
2. Write observation.arrow with all columns above plus simulated.

The easiest way to do this is to assume the observations are instantaneous, and then we add instantaneous simulated values to that. If we don't assume that, we have yet another period we have to average values over. Also note that if you have say two observed values in a year, you see only those two simulated values, whereas you might want to check if the two observations fall nicely onto the daily simulated timeseries. In this case you want to grab basin.arrow or flow.arrow instead.

In the QGIS iMOD timeseries plugin we could then select both observed and simulated at the same time similar to this:

Ideally you could specify your observed data in the add API directly, something like:

model.tabulated_rating_curve.add(
    Node(4, Point(3.0, 0.0)),
    [tabulated_rating_curve.Static(level=[0.0, 1.0], flow_rate=[0.0, 10 / 86400]),
     observation(df)],
)

Where df is a dataframe with the variable, time, observed to which we can add the other columns automatically. We could start with just creating the full Observation table manually though.

[DanielTollenaar]

I assume you don't get TabulatedRatingCurve / Observation (Outlet / Observation, etc) but one Observation table with our observations all together? And can we go towards visualising flow_rate and level in one plot?

A more generic comment/issue; in the above example we add observation(df) while adding a TabulatedRatingCurve to a model, so building a network. As a user I want to be able to add forcing and observations to an existing model-schematisation. I say more generic, as this applies to adding forcing as well. What about keeping your network (schematisation) separated from forcing (and complementary observation)-sets?

[gijsber]

Thank you @DanielTollenaar. I always am trying to convince people that the datamodel and implementation should separate the static part (network parameterization) from the forcings part and the state part, such that external tools (Fews and others) can prepare and hand over the forcings and manage the state.

[DanielTollenaar]

Ohw yes, but if you open this box... For scenario-analysis you want to cross schematisations/networks (current state, measure-set 1, measure-set 2, etc) with different sets of forcings and initial states. That goes a bit further. As a user ideally you don't specify forcing/ini state on node-id, but as an overlay (e.g. UGRID, or regular grids in NetCDF) with continuous info and let the software do the processing to the network.

Not for now, but just to design this part with future requests in mind.

[visr]

In a way the network (Node and Edge table) and parameters (other tables) are already separated in the data model. The add API encourages creating nodes with data together, such that users don't have to look up node IDs and map parameters themselves. This doesn't mean that it is still possible to add the Observations table (indeed one table) directly after the fact though.

[visr]

By default we should probably write the results/observation.arrow if the Observation table is present. Though if users supply 10 minute timeseries this means we'll have to stop every 10 minutes, so it may be nice to be able to disable writing this file as well in the TOML. We currently don't support this yet for any results though, so best to tackle it generically.

Moving this to the Ribasim repository since it primarily concerns a Ribasim feature that Ribasim-NL needs.