search

ImperialCollegeLondon / SWMManywhere

SWMManywhere is used to derive and simulate a sewer network anywhere in the world
https://imperialcollegelondon.github.io/SWMManywhere/
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

Identify efficiencies in iterate metrics where large calculations are repeatedly performed #101

Open barneydobson opened 6 months ago

barneydobson commented 6 months ago

For example best_outlet_match and dominant_outlet could conceivably be expensive, so it could be wasteful to redo them. Though I'm not sure if these calc times are significant in comparison to the other bottlenecks (e.g., subcatchment derivation and simulation).

Suggestion:

Yeah, it seems a bit wasteful. If the inputs were hashable, you could cache the result, but I don't think that's the case, so we would need to come up with a more creative way of avoiding calculating the same thing again and again.

Originally posted by @dalonsoa in https://github.com/ImperialCollegeLondon/SWMManywhere/pull/100#pullrequestreview-1945312539

barneydobson commented 6 months ago

To give an indication, here is log from one of the paper case studies. So a summary of bottlenecks are:

None of these calculations would benefit from the proposed changed, so I think we can consider this issue a low priority

2024-03-19 10:04:31.191 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: assign_id completed.
2024-03-19 10:04:31.503 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: format_osmnx_lanes completed.
2024-03-19 10:04:31.839 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: double_directed completed.
2024-03-19 10:04:32.332 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: fix_geometries completed.
2024-03-19 10:04:33.987 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: split_long_edges completed.
2024-03-19 10:09:58.822 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: calculate_contributing_area completed.
2024-03-19 10:10:01.239 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: set_elevation completed.
2024-03-19 10:10:02.349 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: set_surface_slope completed.
2024-03-19 10:10:03.386 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: set_chahinian_slope completed.
2024-03-19 10:10:04.556 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: set_chahinian_angle completed.
2024-03-19 10:10:05.734 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: calculate_weights completed.
2024-03-19 10:10:07.945 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: identify_outlets completed.
2024-03-19 10:10:09.322 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: derive_topology completed.
2024-03-19 10:10:43.678 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: pipe_by_pipe completed.
2024-03-19 10:10:44.234 | INFO     | swmmanywhere.graph_utilities:iterate_graphfcns:182 - graphfcn: assign_id completed.
2024-03-19 10:33:06.806 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - nc_deltacon0 completed
2024-03-19 10:33:22.842 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - nc_laplacian_dist completed
2024-03-19 10:33:38.480 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - nc_laplacian_norm_dist completed
2024-03-19 10:33:54.597 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - nc_adjacency_dist completed
2024-03-19 10:33:54.662 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - nc_vertex_edge_distance completed
2024-03-19 10:36:04.074 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - nc_resistance_distance completed
2024-03-19 10:36:08.842 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - bias_flood_depth completed
2024-03-19 10:36:20.928 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - kstest_edge_betweenness completed
2024-03-19 10:37:41.480 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - kstest_betweenness completed
2024-03-19 10:37:44.767 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - outlet_nse_flow completed
2024-03-19 10:37:49.812 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - outlet_nse_flooding completed
2024-03-19 10:37:51.073 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - outlet_kstest_diameters completed
2024-03-19 10:37:52.262 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - outlet_pbias_length completed
2024-03-19 10:37:53.423 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - outlet_pbias_nmanholes completed
2024-03-19 10:37:54.726 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - outlet_pbias_npipes completed
2024-03-19 10:38:00.344 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - subcatchment_nse_flooding completed
2024-03-19 10:38:06.065 | INFO     | swmmanywhere.metric_utilities:wrapper:29 - grid_nse_flooding completed
barneydobson commented 2 months ago

Note - tests on larger graphs have increased this issues importance