search

oceanmodeling / ondemand-storm-workflow

Other
2 stars 1 forks source link

failed post-processing step with `ValueError` #23

Open FariborzDaneshvar-NOAA opened 1 year ago

FariborzDaneshvar-NOAA commented 1 year ago

Post-processing step of a test run for hurricane sandy with 5 ensembles failed with the following error message:

[2023-08-15 21:35:23,638] klpc_wetonly    INFO    : dividing 70/30% for training/testing the model
Traceback (most recent call last):
  File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/prep/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/scripts/analyze_ensemble.py", line 392, in <module>
    main(parser.parse_args())
  File "/scripts/analyze_ensemble.py", line 48, in main
    analyze(tracks_dir, ensemble_dir/'analyze')
  File "/scripts/analyze_ensemble.py", line 56, in analyze
    _analyze(tracks_dir, analyze_dir, mann_coef)
  File "/scripts/analyze_ensemble.py", line 176, in _analyze
    plot_perturbations(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/plotting/perturbation.py", line 150, in plot_perturbations
    plot_perturbed_variables_1d(
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/ensembleperturbation/plotting/perturbation.py", line 107, in plot_perturbed_variables_1d
    min_val = perturbed_var.values.min().round(3)
  File "/opt/conda/envs/prep/lib/python3.9/site-packages/numpy/core/_methods.py", line 45, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation minimum which has no identity
ERROR conda.cli.main_run:execute(49): `conda run python -m analyze_ensemble --ensemble-dir /lustre/hurricanes/sandy_2012_4f939ee2-1ef4-4e38-b3dc-6dd62ff0fc8b/setup/ensemble.dir/ --tracks-dir /lustre/hurricanes/sandy_2012_4f939ee2-1ef4-4e38-b3dc-6dd62ff0fc8b/setup/ensemble.dir//track_files` failed. (See above for error)

Path to the slurm_out file on NHC_COLAB_2 cluster: /contrib/Fariborz.Daneshvar/home/workflow_run_logs\sandy_5_ensembles_failed_post_process.out

It should be noted that perturbation.nc file in the analyze directory has values! Path to the analyze directory on NHC_COLAB_2 cluster: /lustre/hurricanes/sandy_2012_Fariborz_OFCL_v2_5/setup/ensemble.dir/analyze/

FariborzDaneshvar-NOAA commented 1 year ago

Update! After discussing the issue with @SorooshMani-NOAA, one guess was that it might be due to few number of ensembles. But similar test runs (with 5 ensembles) for hurricane florence completed! so the number of ensembles (i.e. 5) might not be the reason.