search

E3SM-Project / transport_se

Atmosphere transport mini app
1 stars 1 forks source link

CMAKE machine file for Edison doesn't enable openmp #9

Closed mt5555 closed 9 years ago

mt5555 commented 9 years ago

Need to update cmake/machines/edison.cmake to allow for openMP testing on Edison

mt5555 commented 9 years ago

when running with more than 1 openMP thread, I get:

Rank 356 [Sun Jun 28 12:03:05 2015] [c5-3c0s10n3] Fatal error in MPI_Allreduce: Invalid datatype, error stack\ : MPI_Allreduce(1007): MPI_Allreduce(sbuf=0x2e247f0, rbuf=0x7ffffffef630, count=1, INVALID DATATYPE, MPI_MAX, M\ PI_COMM_WORLD) failed

mt5555 commented 9 years ago

Fixed one bug causing segfaults with openMP: it looks like dcmip_wrapper.F90 code is not thread safe. It is using global/static variables to communicate between subroutines. So when multiple threads call these routines, they will clobber each other's global data.

Temporary fix: put call calls to these routines inside $OMP CRITICAL blocks. Longer term: could copy data directly in elem%state, instead of calling a routine to copy into global variables and another routine to copy from the globals into elem%state? (or I may not be fully understanding this code)

With this temp fix, and setting OMP_STACKSIZE to 64M, ne120 perf test case is running.