Closed johodges closed 6 months ago
I'll take a look.
The plot thickens. I submitted the same job (on a better node) and it was able to initialize and ran 80 time steps but then died with a bad termination and no other warnings/errors.
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 63 PID 39127 RUNNING AT n0016 = KILLED BY SIGNAL: 9 (Killed)
Run it in debug model
I think it was a memory issue. The node has ~50gb of memory and the job was trying to allocate more than that. I submitted it to a node with more memory and the job is using ~150gb of memory. That seems like a lot of memory to me for ~900k cells with SOLID_PHASE_ONLY=T. Does the CELL_SIZE need to be 0.0025 for this verification case to accomplish the objective?
The case is a big sphere with a 3D heat conduction calculation, which requires a 1-D grid at each surface cell, and each interior node stores info about the other 2 coordinate directions. So it is a memory hog. This is part of a convergence study, so the objective is to demonstrate converge at fine resolution.
You need a bigger boat.
The original error message needs to be fixed. The MPI timeout did not involve particles, but rather surface/solid phase info.
Thanks for the info. I can modify the version of qfds I am using to specify the memory required by the node to make sure it gets assigned to one of our larger nodes.
Or you can get a bigger boat ;)
Describe the bug I was running firebot on our cluster last night and had a clean run other than an error in stage5 (release verification). I am running the case on a single node with a lot of oversubscription (8 cores for the 64 processes). Interestingly it failed before the first time step, but it was able to complete 1 time step during the debug verification.
I was also a bit surprised to see it flag a timeout in particles since I do not see any particles defined in the input file. I can increase the MPI_TIMEOUT as suggested here https://github.com/firemodels/fds/issues/11322 which may fix the issue, but I was curious if the particles aspect was pointing to an underlying issue.
Fire Dynamics Simulator
Current Date : February 27, 2024 04:36:17 Revision : FDS-6.8.0-1658-g66a99ae-jh-firebot Revision Date : Mon Feb 26 18:56:56 2024 -0500 Compiler : Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.11.1 Build 20231117_000000 Compilation Date : Feb 26, 2024 18:33:56
Number of MPI Processes: 64
MPI version: 3.1 MPI library version: Intel(R) MPI Library 2021.11 for Linux* OS
Job TITLE : Heat transfer in solid sphere Job ID string : ht3d_sphere_96
ERROR: MPI exchange of particles timed out for MPI process 11 running on n0009. FDS will abort.
Desktop (please complete the following information):