Closed FariborzDaneshvar-NOAA closed 6 months ago
@FariborzDaneshvar-NOAA this is related to not having internet access again! This happens in these parts of the plotting code:
The storm is passed to this code as path to the track file created here: https://github.com/oceanmodeling/ondemand-storm-workflow/blob/269f744ebf05a831f15ed7851c57caf75ecc2dbe/singularity/prep/files/analyze_ensemble.py#L131
The strange thing is when I try running this manually using the singularity container on Hercules, I don't get either of the two connection time-outs. Above errors shows one timeout for getting naturalearth_lowres dataset and another for getting the name of the storm.
The natural earth is cached in the container when I try to use it, and the storm name is correctly returned from the track file, so I don't know why we actually get into the request part of the error!
From my console on Hercules compute node (no internet) and with singularity container I get:
>>> import stormevents
>>> st = stormevents.nhc.track.VortexTrack.from_file('./original.22')
>>> st.name
'SANDY'
And for the shape dataset:
>>> gpd.datasets.get_path('naturalearth_lowres')
'/opt/conda/envs/prep/lib/python3.9/site-packages/geopandas/datasets/naturalearth_lowres/naturalearth_lowres.shp'
>>> gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
pop_est ... geometry
0 889953.0 ... MULTIPOLYGON (((180.00000 -16.06713, 180.00000...
1 58005463.0 ... POLYGON ((33.90371 -0.95000, 34.07262 -1.05982...
2 603253.0 ... POLYGON ((-8.66559 27.65643, -8.66512 27.58948...
3 37589262.0 ... MULTIPOLYGON (((-122.84000 49.00000, -122.9742...
4 328239523.0 ... MULTIPOLYGON (((-122.84000 49.00000, -120.0000...
.. ... ... ...
172 6944975.0 ... POLYGON ((18.82982 45.90887, 18.82984 45.90888...
173 622137.0 ... POLYGON ((20.07070 42.58863, 19.80161 42.50009...
174 1794248.0 ... POLYGON ((20.59025 41.85541, 20.52295 42.21787...
175 1394973.0 ... POLYGON ((-61.68000 10.76000, -61.10500 10.890...
176 11062113.0 ... POLYGON ((30.83385 3.50917, 29.95350 4.17370, ...
[177 rows x 6 columns]
in both cases I removed any warning messages!
Just for the sake of documentation:
geopandas
in the log message above is not related to the final error, but it's just a warning about using databases
in gepandas
.original.22
. The file doesn't have the storm name. That's probably either an ensembleperturbations
issue or something in the container script.I'll look into the no name track file issue to resolve this ticket.
Official tracks don't have name, so we can just put the name in our prep script. For the storms you already ran, you can add the storm name to the track files manually. Please list all completed runs you have that needs combining, so that I can fix the track files, then I'll fix the setup script too to take care of that automatically.
Great! thanks. You might want to test it with Michael which has smaller files compared to Sandy. Here are paths to some completed runs:
/work2/noaa/nos-surge/shared/nhc_hurricanes/michael_2018_043acde3-28ab-4fb0-b1bc-ad8d6e36946e/
/work2/noaa/nos-surge/shared/nhc_hurricanes/michael_2018_385e85c2-71fc-4f06-bf9b-c317c7a43473/
/work2/noaa/nos-surge/shared/nhc_hurricanes/michael_2018_38821e69-74ca-4dab-8da8-1a05ad077ba5/
/work2/noaa/nos-surge/shared/nhc_hurricanes/michael_2018_6221a00d-f888-40c8-9cb3-aeeeb39b7220/
/work2/noaa/nos-surge/shared/nhc_hurricanes/michael_2018_ce448226-69d6-41d6-9675-524629d9672d/
I added a fix in bugfix/minor
branch. Since we work off of that branch, it's OK to close this if the fix works
@FariborzDaneshvar-NOAA if this is resolved please close this ticket, thanks
@SorooshMani-NOAA for some cases, it was still getting stuck in the middle of the process or got memory error (even on the compute node)! However a new run with updated workflow went through and completed the processes that did not complete with the first approach in most cases, but there are still some cases that their post-processes have failed while calculating sensitivities, percentiles, or probabilities!
Thank you @FariborzDaneshvar-NOAA. As we discussed, let's keep this open then until we figure out the failed cases. Hopefully exclusive nodes will resolve this. Then we'll create another ticket to write an optimized combine and analyze scripts from scratch to address the post bottleneck.
@SorooshMani-NOAA uncompleted analysis for high leadtimes of Sandy and Dorian also completed with dask and exclusive flag! Thanks for your help! feel free to close this issue.
OK, thanks for trying different things to make it work. We have to start addressing the bottlenecks in post sometime soon. I'll close this for now. Thanks again!
@SorooshMani-NOAA The post-processing step of the workflow on Hercules is failing at the sensitivity plot section with timeout error (see below) Should increasing the time will resolve this issue, or do you have any other recommendation? Thanks