MOM6 OpenMP support not working (ocean_only benchmark)

[puneet336]

Hi, I had compiled MOM6 by setting OPENMP = on in the config template files using intel compilers (2019u5). while trying the launch config of 28 ranks x 2 threads per rank i noticed that the threads are not being launched (monitored processes for entire duration of run using top), here is the snapshot of what i observed -

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
872167 user1    20   0 4916068 496352 135928 R 100.0   0.3   1:19.43 MOM6
872168 user1    20   0 4910696 485280 134636 R 100.0   0.2   1:19.45 MOM6
872171 user1    20   0 4920744 494296 137556 R 100.0   0.3   1:19.52 MOM6
872174 user1    20   0 4922224 497292 137500 R 100.0   0.3   1:19.51 MOM6
872183 user1    20   0 4904688 489716 134632 R 100.0   0.2   1:19.55 MOM6
872184 user1    20   0 4916528 493556 134332 R 100.0   0.3   1:19.48 MOM6
872162 user1    20   0 4993096 569064 207580 R  99.7   0.3   1:16.86 MOM6
872163 user1    20   0 4930392 497904 136572 R  99.7   0.3   1:19.50 MOM6
872164 user1    20   0 4944180 496488 135184 R  99.7   0.3   1:19.42 MOM6
872165 user1    20   0 4936960 490484 135308 R  99.7   0.2   1:19.45 MOM6
872166 user1    20   0 4915968 497968 135960 R  99.7   0.3   1:19.46 MOM6
872169 user1    20   0 4920508 493160 137136 R  99.7   0.3   1:19.44 MOM6
872172 user1    20   0 4917964 491684 136776 R  99.7   0.3   1:19.52 MOM6
872173 user1    20   0 4920084 498832 138652 R  99.7   0.3   1:19.45 MOM6
872175 user1    20   0 4920440 497424 135300 R  99.7   0.3   1:19.46 MOM6
872177 user1    20   0 4920572 498236 135900 R  99.7   0.3   1:19.54 MOM6
872178 user1    20   0 4920744 494464 136588 R  99.7   0.3   1:19.54 MOM6
872179 user1    20   0 4899664 484792 135700 R  99.7   0.2   1:19.52 MOM6
872180 user1    20   0 4918696 490188 134212 R  99.7   0.2   1:19.47 MOM6
872181 user1    20   0 4922620 495444 137224 R  99.7   0.3   1:19.53 MOM6
872182 user1    20   0 4920572 495668 133368 R  99.7   0.3   1:19.57 MOM6
872185 user1    20   0 4919600 494836 136044 R  99.7   0.3   1:19.54 MOM6
872186 user1    20   0 4902864 480148 134468 R  99.7   0.2   1:19.53 MOM6
872187 user1    20   0 4919600 491800 134604 R  99.7   0.3   1:19.53 MOM6
872188 user1    20   0 4922808 495000 134556 R  99.7   0.3   1:19.52 MOM6
872189 user1    20   0 4902192 486496 134724 R  99.7   0.2   1:19.49 MOM6
872170 user1    20   0 4924668 501076 136768 R  99.3   0.3   1:19.46 MOM6
872176 user1    20   0 4920176 499420 137552 R  99.3   0.3   1:19.46 MOM6

I am able to see following in the stdout, so i assume that OpenMP support has been enabled in the code -

NOTE from PE     0: fms_affinity_set: OCEAN affinity assumes hyper-threading hardware disabled
 MOM_domains_mod OMPthreading           22           0           1

Do i need to carry out any modification in the input file to get the openmp threads to work ? please advice. slurm.omp.txt

[marshallward]

There is some support in MOM6 for OpenMP parallelization, but I do not know of any configuration which benefits from OpenMP threads vs MPI ranks.

Having said that, they ought to at least work. But we should probably resolve #1527 before looking into this too carefully.

[marshallward]

I apologize for dropping this issue, #1527 ended up crowding out this one.

While others could weigh in more, the short answer is that MOM6 currently only provides very limited support for OpenMP. Current implementation is localized to individual loops or modules. It doesn't surprise me that certain tests have no evidence of threading nor any speedup.

I will close this, given how much time has passed, but feel free to re-open or open a discussion tab if you want to revisit this issue.