ACME build fails on Cetus

[jayeshkrishna]

The nightly builds fail on Cetus (and Mira). @wlin7 independently saw the same issue and discussed it in #595 . The build fails with undefined references to the SMP libraries.

cam_history.F90:4433: undefined reference to `_xlsmpParSelf'

The relevant discussions from #595 is included below.

[jayeshkrishna]

@wlin7 : There is one other cetus/mira problem: standard script in current master would fail to link to create executable. The error is about undefined reference to xlsmp library, such as cam_history.F90:4433: undefined reference to `_xlsmpParSelf'.

Don't know if the problem has been reported and fixed in a branch. For now, I just applied a workaround in Macros by adding the following.

ifeq ($(SUPPORTS_CXX), TRUE) LDFLAGS += $(CXX_LIBS) endif

Is it supposed to work like this? Can you please take a look and take care if it? BTW, setting SUPPORTS_CXX = FALSE does not disable the use of CXX_LIBS, which includes libxlsmp.

[jayeshkrishna]

@jayeshkrishna : Looks like an issue with mixing threaded (OMP) flags with a non-threaded build. When specifying the flags in the makefile, the threaded (openmp) flags should not be specified for single threaded builds. Is this issue on master (What is the case that I can use to reproduce the problem)?

[jayeshkrishna]

@wlin7 :

I feel like it is not limited to non-threaded build. It happens to builds with single or multiple threads. For a test, you may try any typical configuration, for example with "-compset FC5 -res ne30_ne30", and check the Macros.

Regarding threading flag, -qsmp=omp is always used for compilation for multi-threaded configuration. But -qsmp=omp is no more included in LDFLAGS in the current master for cetus/mira, while it is used in slightly earlier master (merged up to late November). Don't know if -qsmp=omp really needed for LDFLAGS.

With CXX_LIBS, which include -lxlsmp, the build does succeed. Not sure if it is the proper way, though. The earlier version of master does not involve CXX_LIBS (and not seem to need libxlsmp, at least not explicitly thru LDFLAGS)

[jayeshkrishna]

@jayeshkrishna : Yes, this (using CXX_LIBS instead of -qsmp=omp) was a change made in c2165ec and this case was definitely not covered (tested) in the commit. I am working on a possible fix and let you know how it goes.

[jayeshkrishna]

@wlin7 : Thanks. A formal fix should be nice. For now, I am ok with my approach by adding the few lines in Macros. I have also posted this issue on https://acme-climate.atlassian.net/wiki/display/ATM/Problems+running+post-cime+v1-alpha+model in case some users happened to be using the current version on mira/cetus.

[jayeshkrishna]

@agsalin : How can I perform an ACME build with Trilinos on Mira (create_newcase args etc)?

[jayeshkrishna]

@matthewhoffman / @agsalin : I saw information in the confluence - https://acme-climate.atlassian.net/wiki/display/OCNICE/Land+Ice+-+Building+MPASLI+with+Albany+external+libraries+in+ACME - regarding MPAS LI + Albany support. Can you provide me the details of create_newcase command () that works for MPAS LI + Albany on Mira/Cetus? Would the following work?

./create_newcase -case MPASLIALB_ONLY_f09_g16_a_cetus_test -compset MPASLIALB_ONLY -res f09_g16_a -mach cetus -compiler ibm

[jayeshkrishna]

@agsalin : The ALBANY installation (/home/agsalin/Albany/build/install) on Cetus/Mira is not currently accessible to others.

[matthewhoffman]

@jayeshkrishna , Your create_newcase command looks like it should work if/once Albany is available. I have not run on Cetus myself.

[jayeshkrishna]

I get input file missing errors for the case above (-compset MPASLIALB_ONLY -res f09_g16_a) on Mira.

[agsalin]

@jayeshkrishna : Sorry, that is obviously a temporary location. Is there project space where I have write permissions? I think there is an issue at ALCF that there are different Projects(allocations) with different permissions. I am on ClimateEnergy but not the HighRes one.

Also: I don't seem to be able to log in to Cetus or Mira right now.

[jayeshkrishna]

Can you contact ALCF support regarding the login issue (Meanwhile your home dir is only accessible to you)?

[jayeshkrishna]

So, my fix currently works for single threaded and multi threaded builds. Waiting for input from andy to test it for the albany cases before creating a PR.