Adding generic network model setup method

Kind of request

Adding new functionality to add generic network model setup method.

Enhancement Description

For now in hydromt architecture, there is no dedicated modules for a network object. Network is stored as one of the geoms as a geodataframe. However, this is not sufficient. For many 1D network manipulation, it has advantages to use a more complex data structure such as a graph and unstructured grids.

The motivation to use graph is that it enables 1) storing edges and nodes information at once 2) complex network methods such as networkx to deal with network topology, including creation, manipulation, and study of the structure, dynamics, and functions of complex networks. --> needed for delft3dfm/hybridurb/potentially facilitate wfow + ribasim finding upstream/downstream more easily
The motivations to use unstructured grids is that it enables storing 1) storing edges and nodes information at once 2) complex multidimensional data to the network, such as xugrid , to facilitate index, slice, add and remove data in a more intuitive and less error prone manner. --> needed for all plugins mentioned.

Therefore, it would be a nice addition to have a dedicated network object and utilities around to facilitate geodataframe, graph and Ugrid representations of a network.

There are many plugins who intensively deal with network would benefit from this effort. That includes Ribasim, Delwaq, Delft3D FM and hybridUrb (research project). It would be good to develop this module together/ in parallel with these plugins.

Use cases Hydromt-delft3dfm

An example in hydromt-delft3dfm is the deriving urban drainage network from osm data, we 1) get osm highway features as graph - street as edges, conneciton as nodes 2) add geometry to the graph nodes and edges 3) add dem, landuse information to the nodes/edge as attributes 3) conduct complex network analysis using the nodes and edges, with attributes as weights 4) convert into geodataframe 5) use this geodataframe to setup delft3dfm model, who represents 1D network as network1d and mesh1d in a Ugrid object (writting is done using hydrolib-core). The use case is demonstrated below:

In the above described workflow, we mainly use graph representation for complex network analysis in networkx. There has been quite some functions implemented to facilitate the add data to graph(https://github.com/Deltares/hydromt_delft3dfm/blob/39-setup-urban-sewer-network-from-global-data-project-overview/hydromt_delft3dfm/workflows/graph.py) and the graph from/to geodataframe convertions(https://github.com/Deltares/hydromt_delft3dfm/blob/39-setup-urban-sewer-network-from-global-data-project-overview/hydromt_delft3dfm/graph_utils.py).

topology functions that have been implemented:

create a network from vector files create_graph_from_openstreetmap create_graph_from_hydrography
query network based on connectivity get_largest component
query network based on nodes/edges attributes query_graph_edges_attributes and query_graph_nodes_attributes

data functions that have been implemented:

add data to network edges/nodes using a raster dataset setup_graph_from_rasterdataset
add data to network edges/nodes using a raster reclass method setup_graph_from_rasterdataset_reclass
add data to network edges/nodes using a geodataframe setup_graph_from_geodataframe

utility functions that have been implemented: network_to_graph graph_to_network read_graph write_graph writing supports: 1). write network geometry as a geodataframe (simple attribute types, like string or float) 2). write network attributes as a graph (more complex attribute types, like list or dictionary) 3). write network data as a ugrid netcdf (time series attribute types, like discharge) --> not yet done

Use cases HybridUrb

HybridUrb is a python package to build and run hybrid models for 1D urban drainage networks. It hybrids a semi-distribute hydrological modelling component based on the graph, and a statistical component based on data at nodes. The former is developing and benefiting from a network object that enables graph. the latter component will be implemented as a vector object. The package is under development (2024).

In Hybridurb, we use both graph representation of network for complex network analysis in networkx, and we use Ugrid representation of network for storing simulation results (which then is imported in Delft-FEWS).

topology functions: a network module to setup/query edge and nodes and attributes in a step by step manner (from previous SO urban nowcasting project 2022): https://github.com/xldeltares/hybridurb/blob/14-construct-the-graph-model-from-geometry-files/hybridurb/networksutils/network.py

note this module is quite old, as can be seen in the hydromt api format. it also has the following methods (uses cases see pictures below) apply partitioning algorithms for graph apply pruning for graph simplification apply shortest path for graph sorting apply contraction algorithms for graph skeleton

In addition, utility functions that convert graph to ugrid 1d: scripts snippet

Use case Ribasim

Over at Ribasim, we want to couple Ribasim to Delwaq, so we can calculate things like salt intrusion in the Dutch national water model. We have an initial coupling working, but would like to generalize the scripts (in)to hydromt(-plugins).

Both Ribasim and Delwaq handle 1d networks, where Delwaq fully support the ugrid format. There are differences however in the node/edge representations. For example, Ribasim has nodes that store water (and thus represent water bodies), but also for pumps and other objects that control flow. Edges link nodes, and thus represent flow, but can also be used for non-flow linking of nodes, such as triggering of pumps when a certain level is reached.

In terms of Network functionality, we only need 1d ugrid support, with filtering of nodes and edges based on their attributes (e.g. to remove pumps). Ideally hydromt would handle make handling the ugrid netcdf a bit easier, especially because we have multiple variables (both static as timeseries) for each of the nodes that are linked to it. The conversion from Ribasim to ugrid would be in a new hydromt-ribasim plugin.

Use cases cross plugin

In addition to the utilities above, we foresee that more generic utility functions for graph manipulation would be beneficial: e.g. representing flowdir as graph (WFLOW); navigate the network upstream and downstream; invert edges and nodes (RIBASIM) ;

Deltares / hydromt