Ocean irreproducible on Cori-KNL/Intel

MPAS-Dev / MPAS

Repository for private MPAS development prior to the MPAS v6.0 release.

Other

4 stars 0 forks source link

Ocean irreproducible on Cori-KNL/Intel #1519

Open philipwjones opened 6 years ago

philipwjones commented 6 years ago

During a QU240 test case on Cori-KNL, MPAS ocean model was found to be irreproducible in a pure MPI configuration when built using Intel at -O2 or higher. Namely, two identical 8-core jobs submitted one after the other generated non-b4b differences in the output file.

srun -n 8 ./ocean_model mv output.nc orig.nc srun -n 8 .ocean_model
nccompare output.nc orig.nc finds significant differences but should be b4b

philipwjones commented 6 years ago

Further diagnosis by @greenrongreen pointed to a data alignment issue. Am testing and will submit a PR soon with a fix to compiler options.

philipwjones commented 6 years ago

Added -fp-model consistent flag fixes this and is in PR #1520 Can also fix using -align array64byte but above flag appears more general.

philipwjones commented 6 years ago

Replaced by https://github.com/MPAS-Dev/MPAS-Model/pull/37 that addresses issue above and reports in new repo

philipwjones commented 6 years ago

Sorry, meant to close related PR not issue