Closed FariborzDaneshvar-NOAA closed 1 year ago
@FariborzDaneshvar-NOAA thanks for bringing this up. I think I'd forgotten to update the info.sif
image on the PW platform. Let me do that and then we should be able to run a BEST track
@FariborzDaneshvar-NOAA the issue should be resolved now:
[Soroosh.Mani@nhccolab2-3 singularity]$ ./scripts/workflow.sh florence 2018
/lustre/hurricanes/florence_2018_cee53420-57bc-4f5b-99a1-a360fd0c6f6e
/scripts/hurricane_data.py:17: DeprecationWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas still uses PyGEOS by default. However, starting with version 0.14, the default will switch to Shapely. To force to use Shapely 2.0 now, you can either uninstall PyGEOS or set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os
os.environ['USE_PYGEOS'] = '0'
import geopandas
In the next release, GeoPandas will switch to using Shapely by default, even if PyGEOS is installed. If you only have PyGEOS installed to get speed-ups, this switch should be smooth. However, if you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html).
import geopandas as gpd
/scripts/hurricane_data.py:49: FutureWarning: The geopandas.dataset module is deprecated and will be removed in GeoPandas 1.0. You can get the original 'naturalearth_lowres' data from https://www.naturalearthdata.com/downloads/110m-cultural-vectors/.
ne_low = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
2023-07-27:12:36:54,94 INFO [hurricane_data.py:52] Fetching hurricane info...
/opt/conda/envs/info/lib/python3.9/site-packages/stormevents/nhc/storms.py:64: FutureWarning: The argument 'date_parser' is deprecated and will be removed in a future version. Please use 'date_format' instead, or read your data in as 'object' dtype and then call 'to_datetime'.
storms = pandas.read_csv(
2023-07-27:12:36:54,997 INFO [hurricane_data.py:59] Fetching a-deck track info...
2023-07-27:12:36:54,998 INFO [hurricane_data.py:158] Fetching b-deck track info...
2023-07-27:12:36:54,998 INFO [hurricane_data.py:161] Fetching BEST windswath...
2023-07-27:12:36:55,722 INFO [hurricane_data.py:186] Fetching water level measurements from COOPS stations...
2023-07-27:12:37:09,496 INFO [hurricane_data.py:194] Writing relevant data to files...
146
Download necessary data...
Setting up the model...
Launching runs
152
[Soroosh.Mani@nhccolab2-3 singularity]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
148 compute_c schism.s Soroosh. CF 2:26 3 sorooshmani-nhccolab2-00003-1-[0001-0003]
149 compute_c schism.s Soroosh. PD 0:00 3 (Dependency)
150 compute_c schism.s Soroosh. PD 0:00 3 (Dependency)
151 compute_c schism.s Soroosh. PD 0:00 3 (Dependency)
152 compute_c post.sba Soroosh. PD 0:00 1 (Dependency)
Thanks @SorooshMani-NOAA, it's been fixed. Here are example run outputs:
/lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles/
@SorooshMani-NOAA I'm getting this IndexError
again for the BEST
track run on the restarted NHC_Colb_2 cluster!
Here is the full message copied from slurm-20.out
in /home/Fariborz.Daneshvar/ondemand-storm-workflow/singularity/scripts/
+ singularity run --bind /lustre /lustre/singularity_images//prep.sif setup_ensemble --track-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/nhc_track/hurricane-track.dat --output-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/ensemble.dir/ --num-perturbations 10 --mesh-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/mesh/ --sample-from-distribution --sample-rule korobov --date-range-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/dates.csv --nwm-file /lustre/static_data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /lustre/static_data/tpxo florence 2018
/opt/conda/envs/prep/lib/python3.9/site-packages/stormevents/nhc/track.py:173: UserWarning: It is recommended to specify the file_deck and/or advisories when reading from file
warnings.warn(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
return self._inv_point(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
return self._inv_point(
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
return fn(*args, **kwargs)
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
dataframe = variable.perturb(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
[unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 133, in _function_wrapper
raise sys.exc_info()[0](traceback.format_exc())
IndexError: Traceback (most recent call last):
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
return fn(*args, **kwargs)
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
dataframe = variable.perturb(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
[unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/scripts/setup_ensemble.py", line 276, in <module>
main(parse_arguments())
File "/scripts/setup_ensemble.py", line 111, in main
perturbations = perturb_tracks(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1959, in perturb_tracks
filenames += perturber.write(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1467, in write
output_filenames = [
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1468, in <listcomp>
completed_future.result()
File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
IndexError: Traceback (most recent call last):
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
return fn(*args, **kwargs)
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
dataframe = variable.perturb(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
[unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1
ERROR conda.cli.main_run:execute(47): `conda run python -m setup_ensemble --track-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/nhc_track/hurricane-track.dat --output-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/ensemble.dir/ --num-perturbations 10 --mesh-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/mesh/ --sample-from-distribution --sample-rule korobov --date-range-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/dates.csv --nwm-file /lustre/static_data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /lustre/static_data/tpxo florence 2018` failed. (See above for error)
@FariborzDaneshvar-NOAA as we discussed this seems to be due to outdated singularity images. I updated them now and it seems to be working. Please close this ticket again once you can confirm it works for you as well. Thanks!
It's working! thanks for fixing it!
A test singularity run with the
--past-forecast
flag removed, failed during thesetting up the model...
step with the following error message:working_directory:
/lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles