Closed FariborzDaneshvar-NOAA closed 11 months ago
part 2 (cont.):
5: ABORT: Illegal bnd node 368218 -1 0
6: ABORT: Illegal bnd node 368218 -1 0
7: ABORT: Illegal bnd node 368218 -1 0
8: ABORT: Illegal bnd node 368218 -1 0
9: ABORT: Illegal bnd node 368218 -1 0
10: ABORT: Illegal bnd node 368218 -1 0
11: ABORT: Illegal bnd node 368218 -1 0
12: ABORT: Illegal bnd node 368218 -1 0
13: ABORT: Illegal bnd node 368218 -1 0
14: ABORT: Illegal bnd node 368218 -1 0
15: ABORT: Illegal bnd node 368218 -1 0
16: ABORT: Illegal bnd node 368218 -1 0
17: ABORT: Illegal bnd node 368218 -1 0
18: ABORT: Illegal bnd node 368218 -1 0
19: ABORT: Illegal bnd node 368218 -1 0
20: ABORT: Illegal bnd node 368218 -1 0
21: ABORT: Illegal bnd node 368218 -1 0
22: ABORT: Illegal bnd node 368218 -1 0
23: ABORT: Illegal bnd node 368218 -1 0
24: ABORT: Illegal bnd node 368218 -1 0
25: ABORT: Illegal bnd node 368218 -1 0
26: ABORT: Illegal bnd node 368218 -1 0
27: ABORT: Illegal bnd node 368218 -1 0
28: ABORT: Illegal bnd node 368218 -1 0
29: ABORT: Illegal bnd node 368218 -1 0
30: ABORT: Illegal bnd node 368218 -1 0
31: ABORT: Illegal bnd node 368218 -1 0
0: ABORT: Illegal bnd node 368218 -1 0
4: ABORT: Illegal bnd node 368218 -1 0
[sorooshmani-nhccolab2-00011-1-0005:17146] 31 more processes have sent help message help-mpi-api.txt / mpi-abort
[sorooshmani-nhccolab2-00011-1-0005:17146] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
+ '[' 0 -eq 0 ']'
+ echo 'Combining outputs...'
Combining outputs...
+ date
Thu Sep 21 17:50:00 UTC 2023
+ pushd outputs
/lustre/hurricanes/irene_2011_05f1c18a-5bd7-471f-bfd2-66b66655cb8b/setup/ensemble.dir/spinup/outputs /lustre/hurricanes/irene_2011_05f1c18a-5bd7-471f-bfd2-66b66655cb8b/setup/ensemble.dir/spinup ~/ondemand-storm-workflow/singularity/scripts
+ ls 'hotstart*'
+ popd
/lustre/hurricanes/irene_2011_05f1c18a-5bd7-471f-bfd2-66b66655cb8b/setup/ensemble.dir/spinup ~/ondemand-storm-workflow/singularity/scripts
+ singularity exec --bind /lustre /lustre/singularity_images//solve.sif expect -f /scripts/combine_gr3.exp maxelev 1
spawn combine_gr3
Input file name (e.g.: maxelev):
maxelev
Input # of scalar fields:
1
At line 56 of file /schism/src/Utility/Combining_Scripts/combine_gr3.f90 (unit = 10)
Fortran runtime error: Cannot open file 'outputs/maxelev_000000': No such file or directory
Error termination. Backtrace:
#0 0x2b130e026ad0 in ???
#1 0x2b130e027649 in ???
#2 0x2b130e2771f6 in ???
#3 0x55c33fdcc712 in MAIN__
#4 0x55c33fdcc1ce in main
+ singularity exec --bind /lustre /lustre/singularity_images//solve.sif expect -f /scripts/combine_gr3.exp maxdahv 3
spawn combine_gr3
Input file name (e.g.: maxelev):
maxdahv
Input # of scalar fields:
3
At line 56 of file /schism/src/Utility/Combining_Scripts/combine_gr3.f90 (unit = 10)
Fortran runtime error: Cannot open file 'outputs/maxdahv_000000': No such file or directory
Error termination. Backtrace:
#0 0x2b5843e68ad0 in ???
#1 0x2b5843e69649 in ???
#2 0x2b58440b91f6 in ???
#3 0x5611f4b3e712 in MAIN__
#4 0x5611f4b3e1ce in main
+ mv maxdahv.gr3 maxelev.gr3 -t outputs
mv: cannot stat 'maxdahv.gr3': No such file or directory
mv: cannot stat 'maxelev.gr3': No such file or directory
@FariborzDaneshvar-NOAA after the latest mesh fix I reran Irene and it is already fixed by the fix for Florence. I'm running a 2 member ensemble (up to now 8 hours for the members!) Since it takes time, instead of running your own version, I suggest inspecting my last night's run and close the ticket if mesh looks good. Thanks!
Thanks @SorooshMani-NOAA!
@SorooshMani-NOAA here is the error message I got for the spinup run of
irene
with theBEST
track (run directory onNHC_COLAB_2
:/lustre/hurricanes/irene_2011_05f1c18a-5bd7-471f-bfd2-66b66655cb8b
):part 1: