Closed FariborzDaneshvar-NOAA closed 6 months ago
Workflow is running for other leadtimes of ike_2008 without any issue!
I also ran it several times (in two different days) and every time random number of runs failed: | run directory | No. of failed runs |
---|---|---|
ike_2008_fdc63f00-2b10-4848-92af-7d64335e249a | 13 (5,6,7,12,13,19,21,22,25,27,28,29,30) | |
ike_2008_bc56cd29-7a6a-494a-babe-d82ea0636b8e | spinup | |
ike_2008_7d722a89-0f3e-43fe-b2aa-da1f8c571456 | spinup | |
ike_2008_3b071f55-2544-4bde-8992-dddd4a1a83f2 | spinup | |
ike_2008_759ad86f-b237-4fef-9437-1cdb3f458b13 | 25 (1,2,3,4,5,6,8,9,12,13,14,15,16,17,18,19,20,21,23,25,26,27,28,29,30) | |
ike_2008_c440364a-b6a5-42c9-942b-b20796bfd951 | 25 (2,3,4,5,6,7,8,9,10,12,13,14,15,16,17,18,19,20,21,22,24,25,26,28,30) |
I also ran one of failed SCHISM runs separately, but it failed with the same error!
@pvelissariou1 have you seen this error before? It seems to be related to parmetis during initialization. Does it point to an issue on the compute node libraries or is it some issue with mesh in your opinion?
I checked your input files in /work2/noaa/nos-surge/shared/nhc_hurricanes/ike_2008_bc56cd29-7a6a-494a-babe-d82ea0636b8e/setup/ensemble.dir/spinup and all seem consistent except of course the perturber hurricane tracks. I don't think that these random crashes are related with the SCHISM code itself. Maybe the issue is with the way the simulation is being run within singularity. I don't know how your singularity environment is configured, e.g., memory, network interface, mpi channel, etc. In your slurm job submission script you may add:
ulimit -l unlimited
ulimit -s unlimited (I think this one is already set in the script)
export I_MPI_DEBUG=10 (for debugging purposes. can be deleted afterwards)
export KMP_STACKSIZE=20480000000 (2 GB)
export KMP_AFFINITY=verbose,granularity=core,compact,1,0 (comment/uncomment to check its usefulness)
@FariborzDaneshvar-NOAA Can you try the above to see if the issue is resolved?
In singularity run the command: ulimit -a
first to see how the limits are set (watch for the -l, -s options in ulimit output)
@pvelissariou1 Thanks for your input. I updated schism.sbatch
file and added the following lines (as you suggested) before srun $BIN_DIR/pschism_HERCULES_PAHM_BLD_STANDALONE_TVD-VL 4
command
ulimit -l unlimited
ulimit -s unlimited
export I_MPI_DEBUG=10
export KMP_STACKSIZE=20480000000
export KMP_AFFINITY=verbose,granularity=core,compact,1,0
But then got this error message!
Shell debugging restarted
+ unset __lmod_sh_dbg
+ return 0
+ export MV2_ENABLE_AFFINITY=0
+ MV2_ENABLE_AFFINITY=0
+ ulimit -l unlimited
+ ulimit -s unlimited
+ export I_MPI_DEBUG=10
+ I_MPI_DEBUG=10
+ export KMP_STACKSIZE=20480000000
+ KMP_STACKSIZE=20480000000
+ export KMP_AFFINITY=verbose,granularity=core,compact,1,0
+ KMP_AFFINITY=verbose,granularity=core,compact,1,0
+ srun /work2/noaa/nos-surge/smani/bin//pschism_HERCULES_PAHM_BLD_STANDALONE_TVD-VL 4
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /opt/slurm/lib/libpmi2.so
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /opt/slurm/lib/libpmi2.so
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /opt/slurm/lib/libpmi2.so
...
per ulimit -a
:
max locked memory (kbytes, -l) unlimited
stack size (kbytes, -s) 16384
@Fariborz, @SorooshMani-NOAA Then the calls are directed to the pmi channel. Let me check on that. Keep the ulimit -s unlimited
and ulimit -l unlimited
in the script. Can you share the full log file with me?
@pvelissariou1 Here is the slurm-*.out file: slurm-568745.txt
Thanks
@FariborzDaneshvar-NOAA Fariborz can you replace srun
by srun --mpi=pmi2
and resubmit to see what happens? Leave all other definitions in the script as they are. Can you increase: I_MPI_DEBUG=100
. Last we need to recompile SCHISM with the option: a) cmake -DDEBUG=ON, b) make -DUSE_DEBUG to actually get the lines/files where the problem is located.
@FariborzDaneshvar-NOAA Could you please share your slurm submission script?
@FariborzDaneshvar-NOAA Could you please share your slurm submission script?
Here it is: schism.txt
I used schism.sbatch
from @SorooshMani-NOAA directory and it solved the problem! ✅
Thanks @pvelissariou1 and @SorooshMani-NOAA for your help and looking into this!
@FariborzDaneshvar-NOAA Could you please share the new slurm batch file? Thanks
@pvelissariou1 here it is: schism.txt
As part of running workflow for 6 leadtimes of 25 storms recommended by the NHC team, random SCHISM runs for Ike 2008 (with 24 hr leadtimes) failed with
segmentation fault
. Here is one example error message for one of 13 failed runs (run#5) out of 31 ensemble:/work2/noaa/nos-surge/shared/nhc_hurricanes/ike_2008_fdc63f00-2b10-4848-92af-7d64335e249a