geodynamics / axisem

AxiSEM is a parallel spectral-element method to solve 3D wave propagation in a sphere with axisymmetric or spherically symmetric visco-elastic, acoustic, anisotropic structures.
65 stars 31 forks source link

recursive I/O error #15

Open sstaehler opened 10 years ago

sstaehler commented 10 years ago

For very coarse (and therefore fast, >100s) meshes, I sometimes get solver crashes with "recursive I/O error" when compiling with ifort.

forrtl: severe(40): recursive I/O operation, unit 6, file unknown

It happens after line 444 in nc_routines.F90 and is very elusive (happens at random in 1 out of 5 runs). Web search did not produce conclusive results.

Workaround: Set verbosity to 0 in inparam_advanced, which solved the problem for me.

martinvandriel commented 10 years ago

was this fixed in https://github.com/geodynamics/axisem/commit/d8d73f3cf4be400f1851257d1eb250d285c94665 ?

sstaehler commented 10 years ago

No, the error occurs in nc_routines.F90. Maybe we should ask Dr. Fortran, he has answered dozens of questions about this error https://software.intel.com/en-us/forums/topic/270127 The error occurs when two instances try to write into one IO-unit at the same time. Might happen at this part of the code, who knows. However, I find it difficult to generate a simplified version of this routine that replicates this bug

My suspicion is that it only occurs for very long periods, where the strain dumping takes no time so that the dumping thread writes to unit 6 while the main thread does as well. Never happened before for normal periods (anything less than 50s)

martinvandriel commented 9 years ago

Is this still the case? I think we fixed some concurrent file accesses...