NGEET / fates

repository for the Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
Other
105 stars 92 forks source link

Add graceful failure to seed dispersal mode when max dispersal distance parameter is too high #1078

Open glemieux opened 1 year ago

glemieux commented 1 year ago

Running regression tests with seed dispersal on and the fates_seed_dispersal_max_dist set to 2.5E6 resulted in tests with grid resolutions of f19_g17 and smaller to fail with either of the the following errors:

1659 MPT ERROR: STDOUT or STDERR buffer exhausted.
1660         Set MPI_UNBUFFERED_STDIO=true or output no more than about 8192
1661         characters between newlines.
1662         (HPE MPT 2.25  08/14/21 03:06:24)

or

MPT: shepherd terminated: r11i1n11.ib0.cheyenne.ucar.edu - job aborting

My suspicion after conducting some research into the failure mechanism is that this is possibly due to there being too many neighbors for the MPI passing routines to handle. Running with a maximum distance that was approximately an order of magnitude smaller, results in the tests running successfully. As such, there should be some method of determining what a maximum limit on the dispersal maximum based on the grid resolution. Using this we could then gracefully fail and let the user know that the distance is too high.