Get HOMMEXX building in CAM

[mfdeakin-sandia]

There are a few things which need to be done before this can happen

• [X] Add preqx_cxx to the list of valid values for CAM_TARGET
• [X] Add the list of directories containing the sources to the write_filepath function in CAM's configuration when dyn_target is preqx_cxx
• [X] Add Kokkos' build information to the configuration settings
• [X] Enable building the C++ without Hommexx_config.h
• [X] Ensure CAM calls the C++ in the main timestepping loop
• [X] Ensure all the defines needed are specified (QSIZE_D, PLEV, etc)
• [X] Ensure vectorization settings are specified correctly (ie, -xCORE_AVX512)
• [X] Ensure HOMMEXX picks the correct explicit vector type
• [ ] Ensure the Kokkos forcing method implementations are called
• [ ] Ensure Kokkos initialization methods are called
• [ ] Ensure Kokkos finalization methods are called
• [ ] Ensure the MPI communicator is set correctly
• [X] Ensure all data needed is copied - this should be done somewhere in the prim_run_subcycle subroutine's code path
• [ ] Other issue?

[mfdeakin-sandia]

Here's how I test this:

1. Checkout the mfdeakin-sandia/cam/hommexx branch in the E3SM repository

2. Clone (or just sym-link) hommexx in the components directory.

3. Ensure the compiler entry cime/config/e3sm/machines/config_compilers.xml for the machine being built on has the KOKKOS_PATH entry pointing to the installation of Kokkos.

4. If you're not building on an already supported machine, adapt the entries below to their respective files for the machine you're building on. Note that I have a branch with entries for white and blake in mfdeakin-sandia/testbeds.

5. Run create test: cime/scripts/create_test SMS.ne4_ne4.FC5AQUAP.snl-blake_intel.allactive-hommexx

These are the configuration entries needed to build on blake in step 4: cime/config/e3sm/machines/config_machines.xml:

<machine MACH="snl-blake">
    <DESC>Skylake Testbed machine</DESC>
    <NODENAME_REGEX>blake</NODENAME_REGEX>
    <BATCH_SYSTEM>slurm</BATCH_SYSTEM>
    <OS>LINUX</OS>
    <TESTS>e3sm_developer</TESTS>
    <COMPILERS>intel18</COMPILERS>
    <MPILIBS>openmpi</MPILIBS>
    <RUNDIR>$ENV{HOME}/projects/e3sm/scratch/$CASE/run</RUNDIR>
    <EXEROOT>$ENV{HOME}/projects/e3sm/scratch/$CASE/bld</EXEROOT>
    <DIN_LOC_ROOT>$ENV{HOME}/projects/e3sm/cesm-inputdata</DIN_LOC_ROOT>
    <DIN_LOC_ROOT_CLMFORC>$ENV{HOME}/projects/e3sm/ptclm-data</DIN_LOC_ROOT_CLMFORC>
    <DOUT_S_ROOT>$ENV{HOME}/projects/e3sm/scratch/archive/$CASE</DOUT_S_ROOT>
    <DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
    <CIME_OUTPUT_ROOT>$ENV{HOME}/projects/e3sm/scratch</CIME_OUTPUT_ROOT>
    <BASELINE_ROOT>$ENV{HOME}/projects/e3sm/baselines/$COMPILER</BASELINE_ROOT>
    <CCSM_CPRNC>$CCSMROOT/tools/cprnc/build/cprnc</CCSM_CPRNC>
    <SUPPORTED_BY>mdeakin at sandia dot gov</SUPPORTED_BY>
    <GMAKE_J>48</GMAKE_J>
    <MAX_TASKS_PER_NODE>48</MAX_TASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE>48</MAX_MPITASKS_PER_NODE>
    <module_system type="module" allow_error="true">
      <init_path lang="sh">/usr/share/Modules/init/sh</init_path>
      <init_path lang="python">/usr/share/Modules/init/python.py</init_path>
      <cmd_path lang="sh">module</cmd_path>
      <cmd_path lang="python">module</cmd_path>
      <modules>
    <command name="load">zlib/1.2.11</command>
    <command name="load">intel/compilers/18.1.163</command>
    <command name="load">openmpi/2.1.2/intel/18.1.163</command>
    <command name="load">hdf5/1.10.1/openmpi/2.1.2/intel/18.1.163</command>
    <command name="load">netcdf-exo/4.4.1.1/openmpi/2.1.2/intel/18.1.163</command>
      </modules>
    </module_system>
    <environment_variables>
      <env name="NETCDF_C_PATH">$ENV{NETCDF_ROOT}</env>
      <env name="NETCDF_FORTRAN_PATH">$ENV{NETCDFF_ROOT}</env>
    </environment_variables>
    <mpirun mpilib="default">
      <executable>mpirun</executable>
      <arguments></arguments>
    </mpirun>
</machine>

cime/config/e3sm/machines/config_compilers.xml:

<compiler MACH="snl-blake">
  <KOKKOS_PATH>/ascldap/users/mdeakin/blake/kokkos-release</KOKKOS_PATH>
</compiler>

cime/config/e3sm/machines/config_batch.xml:

   <batch_system MACH="snl-blake" type="slurm" >
     <queues>
       <queue walltimemax="02:00" default="true">blake</queue>
     </queues>
   </batch_system>