search

NOAA-OWP / hydrovis

Other
12 stars 9 forks source link

Performance improvement - change crs on hand data loads and not each forecast rebuild #930

Closed RobHanna-NOAA closed 1 month ago

RobHanna-NOAA commented 1 month ago

Currently, we are re-projecting the data crs on every feature for each forecast update. If we change the projection to 3857 when hand data is loaded in the first place, we can drop the re-project for each forecast. This could make a big impact on calcs for forecasts.

Note: hand is coming in as epsg:5070 and hand Alaska is coming in as epsg:3338. When we data load, we can ensure all huc starting with "19" (which is Alaska), then set the crs as 3338 when the object is loaded before re-projecting. Note. knowing the right incoming projection is super important before reprojecting.

Check the Alaska versus Conus for catchments as well. Is ras2fim covered as well to convert to 3857 at data load? I think it is, but need to check forecast loads to ensure they are also 3857 as well.

RobHanna-NOAA commented 1 month ago

See (Issue 932)[https://github.com/NOAA-OWP/hydrovis/pull/932] which changed Core/LAMBDA/viz_functions/image_based/viz_hand_fim_processing/lambda_function.py. That script is called every time forecast data is loaded and it reprojects all HUCs each time.

Change the initial fim data load so it does the reprojected there instead of each forecast. This means one reproject for any release and not per forecast. We are basically copy/pasting part of this lambda code into the initial hand data load. Watch for catchments too for this.

RobHanna-NOAA commented 1 month ago

Put this optimization on hold for now. It might a similar fix for rems and catchments. Research required for current catchments and rem load and handling for crs, even for hand data. in the lambda above, it was thought the hand data was coming in as ESRI:102039 but for conus it really was EPSG:5070. In 932, it fixed both CONUS and Alaska.

RobHanna-NOAA commented 1 month ago

deemed to big and part of a much bigger picture of optimizing catchments, REMS and hydrotables