search

lanl / vpic

Vector Particle-In-Cell (VPIC) Project
Other
152 stars 76 forks source link

support vpic config using a cmake cache file #69

Closed chuckcranor closed 4 years ago

chuckcranor commented 5 years ago

The current set of config scripts in arch configure vpic by running cmake with the config specified on the cmake command line using cmake cache variables (i.e. the "-D" flag). The lanl scripts also force the module config and run make. There are two problems with this. First, users no longer have access to the cmake command line (e.g. so that they can specify the installation prefix for "make install" with -DCMAKE_INSTALL_PREFIX=...). Second, the config processes on lanl systems differs from non-lanl systems.

This patch adds a new optional way to configure vpic with cmake that works around these issues. When an arch script is invoked with the optional arg "initcache" it generates a file called "vpic-init-cache.cmake" and stops. The "vpic-init-cache.cmake" file contains all the cache variables that the arch script would have normally put on the command line (with the "-D" flag).

This allows the user (or automated build script) to run cmake themselves with additional flags by using the "-C" flag to load the "vpic-init-cache.cmake" file.

For example, to build a test vpic install in /tmp/try one can do:

    % ls
    % ../arch/gcc/v4-sse initcache
    ./vpic-init-cache.cmake created.
    Now you can run "cmake -C ./vpic-init-cache.cmake src_dir" to configure.
    %
    % ls
vpic-init-cache.cmake
    %
    % cat vpic-init-cache.cmake
    set(CMAKE_BUILD_TYPE "Release" CACHE STRING "Initial cache" FORCE)
    set(ENABLE_INTEGRATED_TESTS "ON" CACHE BOOL "Initial cache" FORCE)
    set(USE_V4_SSE "ON" CACHE BOOL "Initial cache" FORCE)
    set(CMAKE_C_FLAGS "-rdynamic -fno-strict-aliasing" CACHE STRING "Initial cache" FORCE)
    set(CMAKE_CXX_FLAGS "-rdynamic -fno-strict-aliasing" CACHE STRING "Initial cache" FORCE)
    %
    % cmake -C ./vpic-init-cache.cmake -DCMAKE_INSTALL_PREFIX=/tmp/try ..
    loading initial cache file ./vpic-init-cache.cmake
    -- The C compiler identification is GNU 7.4.0
    -- The CXX compiler identification is GNU 7.4.0
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Found MPI_C: /usr/lib/x86_64-linux-gnu/libmpich.so (found version "3.1")
    -- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/libmpichcxx.so (found version "3.1")
    -- Found MPI: TRUE (found version "3.1")
    -- Looking for pthread.h
    -- Looking for pthread.h - found
    -- Looking for pthread_create
    -- Looking for pthread_create - not found
    -- Looking for pthread_create in pthreads
    -- Looking for pthread_create in pthreads - not found
    -- Looking for pthread_create in pthread
    -- Looking for pthread_create in pthread - found
    -- Found Threads: TRUE
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /proj/TableFS/data/chuck/src/lanl/devel/vpic/b
    %

then a "make install" would build and install vpic in CMAKE_INSTALL_PREFIX ("/tmp/try" in the example above).

To use this with LANL-style builds, we decouple the module configuration from vpic configuration. We assume LANL users will configure all their modules the way they want them prior to trying to configure vpic. We do not change the module config. Instead, we look into the environment to determine the current module configuration and use that. To support this mode of operation we introduce a new script "arch/lanl-current-mods" that combines ATS1 and CTS1 configs into a single script named "lanl-current-mods" ... this is used in the same way as noted above, but internally it includes extra code to examine and sanity check the currently loaded set of modules. Here is an example from GR:

    [ccranor@gr-fe2 b]$ ../arch/lanl-current-mods initcache
    ERROR: No modules loaded.  A compiler, MPI, and cmake are required.
    [ccranor@gr-fe2 b]$
    [ccranor@gr-fe2 b]$ module load gcc openmpi cmake
    [ccranor@gr-fe2 b]$
    [ccranor@gr-fe2 b]$ ../arch/lanl-current-mods initcache
    Compiler loaded: GNU version 7.4.0
    MPI loaded: OMPI version 2.1.2
    CPU target: broadwell

    Currently Loaded Modules:
      1) gcc/7.4.0   2) openmpi/2.1.2   3) cmake/3.14.6

    ./vpic-init-cache.cmake created.
    Now you can run "cmake -C ./vpic-init-cache.cmake src_dir" to configure.
    [ccranor@gr-fe2 b]$
    [ccranor@gr-fe2 b]$ cat vpic-init-cache.cmake
    set(CMAKE_BUILD_TYPE "Release" CACHE STRING "Initial cache" FORCE)
    set(ENABLE_INTEGRATED_TESTS "OFF" CACHE BOOL "Initial cache" FORCE)
    set(ENABLE_UNIT_TESTS "OFF" CACHE BOOL "Initial cache" FORCE)
    set(ENABLE_OPENSSL "OFF" CACHE BOOL "Initial cache" FORCE)
    set(DISABLE_DYNAMIC_RESIZING "OFF" CACHE BOOL "Initial cache" FORCE)
    set(SET_MIN_NUM_PARTICLES "128" CACHE STRING "Initial cache" FORCE)
    set(USE_LEGACY_SORT "ON" CACHE BOOL "Initial cache" FORCE)
    set(USE_V4_PORTABLE "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_V4_SSE "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_V4_AVX "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_V4_AVX2 "ON" CACHE BOOL "Initial cache" FORCE)
    set(USE_V8_PORTABLE "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_V8_AVX "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_V8_AVX2 "ON" CACHE BOOL "Initial cache" FORCE)
    set(USE_V16_PORTABLE "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_V16_AVX512 "OFF" CACHE BOOL "Initial cache" FORCE)
    set(VPIC_PRINT_MORE_DIGITS "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_OPENMP "OFF" CACHE BOOL "Initial cache" FORCE)
    set(USE_PTHREADS "ON" CACHE BOOL "Initial cache" FORCE)
    set(BUILD_SHARED_LIBS "OFF" CACHE BOOL "Initial cache" FORCE)
    set(CMAKE_C_COMPILER "mpicc" CACHE STRING "Initial cache" FORCE)
    set(CMAKE_CXX_COMPILER "mpicxx" CACHE STRING "Initial cache" FORCE)
    set(CMAKE_C_FLAGS "-g -O2 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -fno-strict-aliasing -Winline -rdynamic -march=broadwell" CACHE STRING "Initial cache" FORCE)
    set(CMAKE_CXX_FLAGS "-g -O2 -ffast-math -fno-unsafe-math-optimizations -fomit-frame-pointer -fno-strict-aliasing -Winline -rdynamic -march=broadwell" CACHE STRING "Initial cache" FORCE)
    [ccranor@gr-fe2 b]$

Now you can simply do:

    % cmake -C ./vpic-init-cache.cmake -DCMAKE_INSTALL_PREFIX=/tmp/try ..

in the same way as the non-lanl example above and then do "make install" to build.

To make this work, we introduce additional shell functions in arch/cmake-fns.sh (to create and append to the "./vpic-init-cache.cmake" file) and arch/module-fns.sh (to examine + sanity check the currently loaded modules and use that into to set VCOM, VMPI, and VCPU).

The new "arch/lanl-current-mods" script is a combined version of lanl-ats1-hsw, lanl-ats1-knl, and lanl-cts1 adapted to work with arch/module-fns.sh.

The arch/gcc/* files were also adapted to use "arch/cmake-fns.sh" and support the "initcache" arg.

codecov[bot] commented 5 years ago

Codecov Report

Merging #69 into devel will decrease coverage by 0.01%. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##            devel      #69      +/-   ##
==========================================
- Coverage   82.37%   82.35%   -0.02%     
==========================================
  Files         114      114              
  Lines        6904     6904              
  Branches     1074     1074              
==========================================
- Hits         5687     5686       -1     
  Misses        768      768              
- Partials      449      450       +1
Impacted Files Coverage Δ
src/util/pipelines/pipelines_thread.cc 60.74% <0%> (-0.94%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8d60cd8...eae8d51. Read the comment docs.

rfbird commented 5 years ago

@chuckcranor maybe we should briefly chat about this -- what's the end goal here? I see that this commit lets you have more control and support more flexible automated builds..but then what?

chuckcranor commented 5 years ago

The end goal is to be able to build master vpic with cmake's ExternalProject module using the ExternalProject_Add() call. Something like:

#
# create vpic target
#
ExternalProject_Add (vpic GIT_REPOSITORY ${VPIC_REPO} GIT_TAG ${VPIC_TAG}
    CMAKE_ARGS -C ./vpic-init-cache.cmake -DBUILD_SHARED_LIBS=ON
    CMAKE_CACHE_ARGS ${UMBRELLA_CMAKECACHE}
    UPDATE_COMMAND ""
)

#
# add extra cache config step
#
ExternalProject_Add_Step (vpic prepare
    COMMAND /arch/${VPIC_CFG} initcache
    COMMENT "generating initial cmake cache for configure ${VPIC_CFG}"
    DEPENDEES update
    DEPENDERS configure
    WORKING_DIRECTORY )

This is how we build a suite of components for our DeltaFS project. For example, if you wanted to build a test install of DeltaFS in /tmp/dd on GR-fe, you can just do this:

mkdir -p /tmp/dd/src
cd /tmp/dd/src
git clone https://github.com/pdlfs/deltafs-umbrella
mkdir deltafs-umbrella/build
cd deltafs-umbrella/build
cmake -DCMAKE_INSTALL_PREFIX=/tmp/dd -DMERCURY_NA_INITIALLY_ON="sm;ofi" -DUMBRELLA_SKIP_TESTS=ON  ..
make

Assuming you have your web proxy set to the LANL proxy, this will download, build, and install in /tmp/dd all the parts of DeltaFS including mercury RPC, OFI libfabric, our DeltaFS code, and vpic407 (which is what we have been using for our SC, PDSW, and cluster papers that use vpic... but we want to update from 407 to master).

I guess this is similar to something you might do with a container or spack or some other package manager, but it is a simple and pure cmake-based system and does not require "modules" (so we can build it locally where we don't use modules and it also works at LANL where there are modules and sometimes you are on a cray).

Since we run in both LANL (GR, TR, and TT) and non-LANL environments I'm trying to get the newer vpic to build the same way in all those environments so our build is fully portable...

chuckcranor commented 5 years ago

Oh, in the above deltafs-umbrella example, when you configure the overall project that ${VPIC_CFG} cmake variable allows you to specify the vpic config script to use from the higher-level meta project (we call it "umbrella"). So, after you clone deltafs-umbrella, when you run cmake you'd do something like "-DVPIC_CFG=gcc/v4-see" or "-DVPIC_CFG=lanl-current-mods" to choose how you want to compile vpic (i.e. the selection of which arch script to run is passed in from above through the ExternalProject_Add() call (seemed like a cool thing to do!)).