search

oceanmodeling / ondemand-storm-workflow

Other
2 stars 1 forks source link

failed to set up the model for the `BEST` track #19

Closed FariborzDaneshvar-NOAA closed 1 year ago

FariborzDaneshvar-NOAA commented 1 year ago

A test singularity run with the --past-forecast flag removed, failed during the setting up the model... step with the following error message:

+ singularity run --bind /lustre /lustre/singularity_images//prep.sif setup_ensemble --track-file /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/nhc_track/hurricane-track.dat --output-directory /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/setup/ensemble.dir/ --num-perturbations 15 --mesh-directory /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/mesh/ --sample-from-distribution --sample-rule korobov --date-range-file /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/setup/dates.csv --nwm-file /lustre/static_data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /lustre/static_data/tpxo florence 2018
/opt/conda/envs/prep/lib/python3.9/site-packages/stormevents/nhc/track.py:173: UserWarning: It is recommended to specify the file_deck and/or advisories when reading from file
  warnings.warn(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
  return self._inv_point(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
  return self._inv_point(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
  return self._inv_point(
concurrent.futures.process._RemoteTraceback:                                                                                                                                                     """
Traceback (most recent call last):
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
      return fn(*args, **kwargs)
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
      dataframe = variable.perturb(
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
      [unique_points[1] - unique_points[0]], max_interpolated_points, axis=0 
IndexError: index 1 is out of bounds for axis 0 with size 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
      r = call_item.fn(*call_item.args, **call_item.kwargs)
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 133, in _function_wrapper 
      raise sys.exc_info()[0](traceback.format_exc())
IndexError: Traceback (most recent call last): 
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
      return fn(*args, **kwargs)
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
      dataframe = variable.perturb(
   File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
      [unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 197, in _run_module_as_main
   return _run_code(code, main_globals, None,
File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 87, in _run_code 
   exec(code, run_globals)
File "/scripts/setup_ensemble.py", line 276, in <module>
   main(parse_arguments())
File "/scripts/setup_ensemble.py", line 111, in main
   perturbations = perturb_tracks(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1959, in perturb_tracks
   filenames += perturber.write(
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1467, in write
   output_filenames = [
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1468, in <listcomp>
   completed_future.result()
File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/_base.py", line 439, in result
   return self.__get_result()
File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result 
   raise self._exception
IndexError: Traceback (most recent call last): 
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
   return fn(*args, **kwargs) 
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
   dataframe = variable.perturb( 
File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
   [unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1 

ERROR conda.cli.main_run:execute(46): `conda run python -m setup_ensemble --track-file /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/nhc_track/hurricane-track.dat --output-directory /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/setup/ensemble.dir/ --num-perturbations 15 --mesh-directory /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/mesh/ --sample-from-distribution --sample-rule korobov --date-range-file /lustre/hurricanes/florence_2018_0d524134-e4de-47c1-abb3-7154cbe8b323/setup/dates.csv --nwm-file /lustre/static_data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /lustre/static_data/tpxo florence 2018` failed. (See above for error)

working_directory: /lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles

SorooshMani-NOAA commented 1 year ago

@FariborzDaneshvar-NOAA thanks for bringing this up. I think I'd forgotten to update the info.sif image on the PW platform. Let me do that and then we should be able to run a BEST track

SorooshMani-NOAA commented 1 year ago

@FariborzDaneshvar-NOAA the issue should be resolved now:

[Soroosh.Mani@nhccolab2-3 singularity]$ ./scripts/workflow.sh florence 2018
/lustre/hurricanes/florence_2018_cee53420-57bc-4f5b-99a1-a360fd0c6f6e
/scripts/hurricane_data.py:17: DeprecationWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas still uses PyGEOS by default. However, starting with version 0.14, the default will switch to Shapely. To force to use Shapely 2.0 now, you can either uninstall PyGEOS or set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:

import os
os.environ['USE_PYGEOS'] = '0'
import geopandas

In the next release, GeoPandas will switch to using Shapely by default, even if PyGEOS is installed. If you only have PyGEOS installed to get speed-ups, this switch should be smooth. However, if you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html).
  import geopandas as gpd
/scripts/hurricane_data.py:49: FutureWarning: The geopandas.dataset module is deprecated and will be removed in GeoPandas 1.0. You can get the original 'naturalearth_lowres' data from https://www.naturalearthdata.com/downloads/110m-cultural-vectors/.
  ne_low = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
2023-07-27:12:36:54,94 INFO     [hurricane_data.py:52] Fetching hurricane info...
/opt/conda/envs/info/lib/python3.9/site-packages/stormevents/nhc/storms.py:64: FutureWarning: The argument 'date_parser' is deprecated and will be removed in a future version. Please use 'date_format' instead, or read your data in as 'object' dtype and then call 'to_datetime'.
  storms = pandas.read_csv(
2023-07-27:12:36:54,997 INFO     [hurricane_data.py:59] Fetching a-deck track info...
2023-07-27:12:36:54,998 INFO     [hurricane_data.py:158] Fetching b-deck track info...
2023-07-27:12:36:54,998 INFO     [hurricane_data.py:161] Fetching BEST windswath...
2023-07-27:12:36:55,722 INFO     [hurricane_data.py:186] Fetching water level measurements from COOPS stations...
2023-07-27:12:37:09,496 INFO     [hurricane_data.py:194] Writing relevant data to files...
146
Download necessary data...
Setting up the model...
Launching runs
152
[Soroosh.Mani@nhccolab2-3 singularity]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               148 compute_c schism.s Soroosh. CF       2:26      3 sorooshmani-nhccolab2-00003-1-[0001-0003]
               149 compute_c schism.s Soroosh. PD       0:00      3 (Dependency)
               150 compute_c schism.s Soroosh. PD       0:00      3 (Dependency)
               151 compute_c schism.s Soroosh. PD       0:00      3 (Dependency)
               152 compute_c post.sba Soroosh. PD       0:00      1 (Dependency)
FariborzDaneshvar-NOAA commented 1 year ago

Thanks @SorooshMani-NOAA, it's been fixed. Here are example run outputs: /lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles/

FariborzDaneshvar-NOAA commented 1 year ago

@SorooshMani-NOAA I'm getting this IndexError again for the BEST track run on the restarted NHC_Colb_2 cluster!

Here is the full message copied from slurm-20.out in /home/Fariborz.Daneshvar/ondemand-storm-workflow/singularity/scripts/

+ singularity run --bind /lustre /lustre/singularity_images//prep.sif setup_ensemble --track-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/nhc_track/hurricane-track.dat --output-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/ensemble.dir/ --num-perturbations 10 --mesh-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/mesh/ --sample-from-distribution --sample-rule korobov --date-range-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/dates.csv --nwm-file /lustre/static_data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /lustre/static_data/tpxo florence 2018
/opt/conda/envs/prep/lib/python3.9/site-packages/stormevents/nhc/track.py:173: UserWarning: It is recommended to specify the file_deck and/or advisories when reading from file
  warnings.warn(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
  return self._inv_point(
/opt/conda/envs/prep/lib/python3.9/site-packages/pyproj/geod.py:404: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(ser.iloc[0]) instead
  return self._inv_point(
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
    dataframe = variable.perturb(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
    [unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 133, in _function_wrapper
    raise sys.exc_info()[0](traceback.format_exc())
IndexError: Traceback (most recent call last):
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
    dataframe = variable.perturb(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
    [unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/scripts/setup_ensemble.py", line 276, in <module>
    main(parse_arguments())
  File "/scripts/setup_ensemble.py", line 111, in main
    perturbations = perturb_tracks(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1959, in perturb_tracks
    filenames += perturber.write(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1467, in write
    output_filenames = [
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1468, in <listcomp>
    completed_future.result()
  File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/opt/conda/envs/prep/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
IndexError: Traceback (most recent call last):
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/utilities.py", line 130, in _function_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 1594, in write_perturbed_track
    dataframe = variable.perturb(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/perturbation/atcf.py", line 933, in perturb
    [unique_points[1] - unique_points[0]], max_interpolated_points, axis=0
IndexError: index 1 is out of bounds for axis 0 with size 1

ERROR conda.cli.main_run:execute(47): `conda run python -m setup_ensemble --track-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/nhc_track/hurricane-track.dat --output-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/ensemble.dir/ --num-perturbations 10 --mesh-directory /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/mesh/ --sample-from-distribution --sample-rule korobov --date-range-file /lustre/hurricanes/florence_2018_7a786ede-9b33-4508-86ab-932259a2d7bc/setup/dates.csv --nwm-file /lustre/static_data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /lustre/static_data/tpxo florence 2018` failed. (See above for error)
SorooshMani-NOAA commented 1 year ago

@FariborzDaneshvar-NOAA as we discussed this seems to be due to outdated singularity images. I updated them now and it seems to be working. Please close this ticket again once you can confirm it works for you as well. Thanks!

FariborzDaneshvar-NOAA commented 1 year ago

It's working! thanks for fixing it!