improve adcirc patchset patch application "audit trail"

[wwlwpd]

@jasonfleming - let me know what you think we need in addition to the ability to:

1. cd into adcirc source dir and run git diff
2. cat the .out file that lists the paths of all patches applied

[asgs@qbc2 asgs]$ cat ../adcirc-cg-v53release-intel/v53release-applied.out 
09/13/21 02:58:39 CDT /ddnA/work/asgs/asgs/patches/ADCIRC/v53release/01-v53release-qbc.patch
09/13/21 02:58:39 CDT /ddnA/work/asgs/asgs/patches/ADCIRC/v53release/02-netcdfhome-cleanup.patch
09/13/21 02:58:39 CDT /ddnA/work/asgs/asgs/patches/ADCIRC/v53release/03-issue-551.patch

Acceptance Criteria:

• [ ] add env $ADCIRC_BUILD_SCRIPT which would give easy access in the asgsh to $ADCIRCBASE/asgs-profile.sh; this in turn would be able to be passed on to run.properties
• [ ] capture in run.properties relevant information regarding the adcirc build

[wwlwpd]

@jasonfleming is the above information sufficient for the audit trail? Let me know what questions you have, but I think it's already doing what you're wanting.

[jasonfleming]

Hey @wwlwpd I like this. Would it be possible to add the git SHA to document what version of each patch was applied? Or would there be some other way of definitively knowing that? Would also like to document which macros.inc file was used to build SWAN.

[wwlwpd]

I am sure I can improve the audit info, I'll think more about it.

[wwlwpd]

@jasonfleming - had some ideas...

fwiw, if you go into the ADCIRC src directory, you can see the effective changes from the patches via git diff since it is a git repo and changes are tracked.

One thing I thought of was to copy the .patch files to the ADCIRC directory, then apply them there so the actual files that were applied are physically located there also;

Yet another option is to do a git commit to the ADCIRC repo each time a patch is applied; the commit message could contain the information (though obviously the committed changes are for local info only).

Any of those sound like a winner to you?

[jasonfleming]

I think we want to make a distinction between documenting the state of the source code and the executable that was used to generate a certain set of numerical results. We want to have provenance that allows us to start with a certain set of output data, and ascertain the executable that generated it, all the compiler flags, library versions, modules, etc that were used to compile it, as well as the source code version and all patches that were applied to the source. I think this info needs to go into the run.properties files, and eventually into the netCDF metadata in each of the netCDF files.

[wwlwpd]

Ah, ok. I think we're on the right track...check this out - this might be a good start. This is everything that is captured in an ADCIRC profile that results from a build (e.g., *~/.adcirc-meta/v53release-intel):

export ASGS_HOME=/home1/0000/asgs
export ASGS_MACHINE_NAME=frontera
export NETCDFHOME=/work2/0000/asgs/frontera/opt/default
export ADCIRCBASE=/work2/0000/asgs/frontera/adcirc-cg-v53release-intel
export ADCIRCDIR=/work2/0000/asgs/frontera/adcirc-cg-v53release-intel/work
export SWANDIR='/work2/0000/asgs/frontera/adcirc-cg-v53release-intel/swan'
export ADCIRC_COMPILER=intel
export ADCIRC_GIT_BRANCH=v53release
export ADCIRC_GIT_URL=git@github.com:adcirc
export ADCIRC_GIT_REPO=adcirc-cg
export ASGS_MAKEJOBS=1
export ADCIRC_MAKE_CMD='make padcirc adcirc adcprep hstime aswip compiler=intel NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/0000/asgs/frontera/opt/default NETCDFROOT=/work2/0000/asgs/frontera/opt/default MACHINENAME=frontera'
export SWAN_UTIL_BINS_MAKE_CMD='make unhcat compiler=intel NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/0000/asgs/frontera/opt/default NETCDFROOT=/work2/0000/asgs/frontera/opt/default MACHINENAME=frontera'
export ADCSWAN_MAKE_CMD='make adcswan padcswan SWAN=enable compiler=intel MACHINENAME=frontera'
export ADCIRC_PROFILE_NAME=v53release-intel
export ADCIRC_BINS='padcirc adcirc adcprep hstime aswip'
export ADCSWAN_BINS='adcswan padcswan'
export SWAN_UTIL_BINS='unhcat.exe'

Based on what you said, what information is missing? You said information on the patchset - what information would you like, the list of files? patchset name?

All of this is written via cloud/general/init-adcirc.sh and is in the asgsh profile environment so it can be easily grabbed during a run for inclusion in run.properties.

[wwlwpd]

@jasonfleming, the current PR (draft) writes a file in $ADCIRCBASE/asgs-build.sh that contains:

• paths of patches applied (behind comment sigils)
• actual build commands

We can add additional information, just let me know what. What do you think?

e.g.,

# 10/04/21 01:15:37 CDT /work2/06482/asgs/stampede2/asgs/patches/ADCIRC/v53release/01-v53release-qbc.patch
# 10/04/21 01:15:37 CDT /work2/06482/asgs/stampede2/asgs/patches/ADCIRC/v53release/02-netcdfhome-cleanup.patch
# 10/04/21 01:15:37 CDT /work2/06482/asgs/stampede2/asgs/patches/ADCIRC/v53release/03-issue-551.patch
# 10/04/21 01:15:37 CDT /work2/06482/asgs/stampede2/asgs/patches/ADCIRC/v53release/04-issue-549.patch
# 10/04/21 01:15:37 CDT /work2/06482/asgs/stampede2/asgs/patches/ADCIRC/v53release/05-issue-550.patch
cd /work2/06482/asgs/stampede2/adcirc-cg-fart/swan && \
   make unhcat compiler=intel MACHINENAME=stampede2 && \
cd /work2/06482/asgs/stampede2/adcirc-cg-fart/work && \
   make padcirc adcirc adcprep hstime aswip SWAN=enable compiler=intel NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/06482/asgs/stampede2/opt/default MACHINENAME=stampede2 && \
   make adcswan padcswan compiler=intel MACHINENAME=stampede2 NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/06482/asgs/stampede2/opt/default

[wwlwpd]

Basically we're dealing with 2 places to put the audit/meta info:

1. $ADCIRCBASE/asgs-build.sh
2. $HOME/.adcirc-meta/<adcirc-profile-name>

I think one nice thing would to be to add a variable to the adcirc profile meta env, $ADCIRC_BUILD_SCRIPT which would give easy access in the asgs environment to the file 1 above.

[wwlwpd]

@jasonfleming what "meta" data about ADCIRC would you like to capture in run.properties; this is going to be a two part issue:

First, identify the information we want to make available to asgs_main.sh via environment.

Second, add this information to run.properties.

What I need from you, @jasonfleming, is the adcirc related meta data we need to capture, e.g.,

• version
• ASGS patchset
• path to build script
• ...what else?

[jasonfleming]

Hey @wwlwpd sorry for the delay, I know I am the bottleneck here. :-) I am not very familiar with the subsystem that builds ADCIRC in ASGS, so I will need to dig into this some today before I can provide anything useful. I am looking at this now.

[jasonfleming]

Hey @wwlwpd I went through and made sort of a wish list of what I would like to see in an automatically generated adcirc.executables.provenance.json file. The suggested example below takes everything you suggested, adds a few additional things and formats it as json. I've added blank lines to set of the main part of what you suggested (in the center) with my additions at the top and bottom.

This file could be generated by init-adcirc.sh and kept in the directory with the ADCIRC build, then copied into the scenario directories and made available for posting to opendap.

One other detail to mention: the SWAN=enable command line option to make is only needed for compiling adcprep ... it does not have an effect when compiling adcswan or padcswan. I have adjusted the make command lines in the json below to reflect this. Also the SWAN make unhcat command line does not need any other information, it gets everything it needs from the SWAN macros.inc file, as long as the right macros.inc file is being used. In fact, we might want to pull out some key lines from that file to show as metadata in this new json file.

{
    "time.adcirc.executables.built" : "2021-Oct-04-T08:43:29+0000",
    "adcirc.source.commit" : "41be52943927bc2af0f4a894e0417134f373a119",
    "time.adcirc.source.commit" : "2020-Sep-01-T12:00:00+0000",
    "time.adcirc.source.clone" : "2021-Oct-04-T07:00:00+0000",        
    "adcirc.source.tag" : "v55release",
    "adcirc.source.asgs.patches.applied" : [ { "path" : "/work2/06482/asgs/stampede2/asgs/patches/ADCIRC/v53release/01-v53release-qbc.patch", "commit" : "1234562943927bc2af0f4a894e0417134f373a100" } ],

    "env.adcirc.build.ASGS_HOME" : "/home1/0000/asgs",
    "env.adcirc.build.ASGS_MACHINE_NAME" : "frontera",
    "env.adcirc.build.NETCDFHOME" : "/work2/0000/asgs/frontera/opt/default",
    "env.adcirc.build.ADCIRCBASE" : "/work2/0000/asgs/frontera/adcirc-cg-v53release-intel",
    "env.adcirc.build.ADCIRCDIR" : "/work2/0000/asgs/frontera/adcirc-cg-v53release-intel/work",
    "env.adcirc.build.SWANDIR" : "/work2/0000/asgs/frontera/adcirc-cg-v53release-intel/swan",
    "env.adcirc.build.ADCIRC_COMPILER" : "intel",
    "env.adcirc.build.ADCIRC_GIT_BRANCH" : "v53release",
    "env.adcirc.build.ADCIRC_GIT_URL" : "git@github.com:adcirc",
    "env.adcirc.build.ADCIRC_GIT_REPO" : "adcirc-cg",
    "env.adcirc.build.ASGS_MAKEJOBS" : "1",
    "env.adcirc.build.ADCIRC_MAKE_CMD" : "make padcirc adcirc adcprep hstime aswip SWAN=enable compiler=intel NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/0000/asgs/frontera/opt/default
 MACHINENAME=frontera",
    "env.adcirc.build.SWAN_UTIL_BINS_MAKE_CMD" : "make unhcat",
    "env.adcirc.build.ADCSWAN_MAKE_CMD" : "make adcswan padcswan compiler=intel MACHINENAME=frontera compiler=intel NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/0000/asgs/frontera/opt/default MACHINENAME=frontera",        
    "env.adcirc.build.ADCIRC_PROFILE_NAME" : "v53release-intel",
    "env.adcirc.build.ADCIRC_BINS" : "padcirc adcirc adcprep hstime aswip",
    "env.adcirc.build.ADCSWAN_BINS" : "adcswan padcswan",
    "env.adcirc.build.SWAN_UTIL_BINS" : "unhcat.exe",

    "env.adcirc.build.PATH" : "/opt/apps/xalt/xalt/bin:/opt/apps/hwloc/1.11.12/bin:/opt/apps/pmix/3.1.4/bin:/opt/apps/cmake/3.20.3/bin:/opt/apps/intel19/python3/3.7.0/bin:/opt/apps/autotools/1.2/bin:/opt/apps/git/2.24.1/bin:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin:/opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64:/opt/apps/gcc/8.3.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/ddn/ime/bin:/opt/dell/srvadmin/bin:.:/home1/00976/jgflemin/bin",
    "env.adcirc.build.LD_LIBRARY_PATH" : "/opt/apps/hwloc/1.11.12/lib:/opt/apps/pmix/3.1.4/lib:/opt/apps/intel19/python3/3.7.0/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/libfabric/lib:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/release:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib:/opt/intel/debugger_2020/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2020.1.217/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2020.1.217/linux/tbb/lib/intel64_lin/gcc4.8:/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2020.1.217/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2020.1.217/linux/compiler/lib/intel64_lin:/opt/apps/gcc/8.3.0/lib64:/opt/apps/gcc/8.3.0/lib",
    "adcirc.build.fortran.compiler.version" : "19.1.1.217 20200306",
    "adcirc.build.c.compiler.version" : "19.1.1.217 20200306",    
    "adcirc.build.swan.compiler" : "intel",
    "adcirc.build.swan.macros.inc" : "macros.inc.frontera",
    "adcirc.build.modules.loaded" : "intel/19.1.1 impi/19.0.9 git/2.24.1 autotools/1.2 python3/3.7.0 cmake/3.20.3 pmix/3.1.4 hwloc/1.11.12 xalt/2.10.27 TACC"
}

[wwlwpd]

This is exactly what I needed - thank you.

[wwlwpd]

Mind is I strike the word provenance and use something like:

• adcirc.executables.source.json, or
• adcirc.executables.origin.json ?

... just the simplify it.

[wwlwpd]

How do you think this is shaping up, 100% dynamic at this point.

  {
    "adcirc.build.swan.macros-inc"             : "/work2/06482/$USER/stampede2/adcirc-cg-swan/swan/macros.inc",
    "adcirc.build.swan.macros-inc.F90_SER"     : "gfortran",
    "adcirc.build.swan.macros-inc.F90_MPI"     : "mpif90",
    "adcirc.build.swan.macros-inc.F90_OMP"     : "gfortran",
    "adcirc.build.swan.macros-inc.FLAGS_OPT"   : "-O",
    "adcirc.build.swan.macros-inc.FLAGS_MSC"   : "-w -fno-second-underscore -ffree-line-length-none",
    "adcirc.build.swan.macros-inc.FLAGS90_MSC" : "$(FLAGS_MSC) -ffree-line-length-none",
    "adcirc.build.swan.macros-inc.FLAGS_SER"   : "",
    "adcirc.build.swan.macros-inc.FLAGS_MPI"   : "",
    "adcirc.build.swan.macros-inc.FLAGS_OMP"   : "-fopenmp",
    "time.adcirc.executables.built"      : "2021-Oct-06-T13:50:13-0500",
    "adcirc.source.commit"               : "4f3bb7e0be8abf2c2679d516e1ec35c0da2d9b98",
    "adcirc.source.asgs.patches.set      : "v53release",
    "adcirc.source.asgs.patches.applied" : [
      "/work2/06482/$USER/stampede2/asgs/patches/ADCIRC/v53release/01-v53release-qbc.patch",
      "/work2/06482/$USER/stampede2/asgs/patches/ADCIRC/v53release/02-netcdfhome-cleanup.patch",
      "/work2/06482/$USER/stampede2/asgs/patches/ADCIRC/v53release/03-issue-551.patch",
      "/work2/06482/$USER/stampede2/asgs/patches/ADCIRC/v53release/04-issue-549.patch",
      "/work2/06482/$USER/stampede2/asgs/patches/ADCIRC/v53release/05-issue-550.patch"
    ],
    "adcirc.source.branch-base"          : "v53release",
    "env.adcirc.build.ASGS_HOME"         : "/home1/06482/$USER",
    "env.adcirc.build.ASGS_MACHINE_NAME" : "stampede2",
    "env.adcirc.build.NETCDFHOME"        : "/work2/06482/$USER/stampede2/opt/default",
    "env.adcirc.build.ADCIRCBASE"        : "/work2/06482/$USER/stampede2/adcirc-cg-swan",
    "env.adcirc.build.ADCIRCDIR"         : "/work2/06482/$USER/stampede2/adcirc-cg-swan/work",
    "env.adcirc.build.SWANDIR"           : "/work2/06482/$USER/stampede2/adcirc-cg-swan/swan",
    "env.adcirc.build.ADCIRC_COMPILER"   : "intel",
    "env.adcirc.build.ADCIRC_GIT_BRANCH" : "v53release",
    "env.adcirc.build.ADCIRC_GIT_URL"    : "git@github.com:adcirc",
    "env.adcirc.build.ADCIRC_GIT_REPO"   : "adcirc-cg",
    "env.adcirc.build.ASGS_MAKEJOBS"     : "1",
    "env.adcirc.build.ADCIRC_MAKE_CMD"   : "make padcirc adcirc adcprep hstime aswip SWAN=enable compiler=intel NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/06482/$USER/stampede2/opt/default MACHINENAME=stampede2",
    "env.adcirc.build.SWAN_UTIL_BINS_MAKE_CMD" : "make unhcat compiler=intel MACHINENAME=stampede2",
    "env.adcirc.build.ADCSWAN_MAKE_CMD"        : "make adcswan padcswan compiler=intel MACHINENAME=stampede2 NETCDF=enable NETCDF4=enable NETCDF4_COMPRESSION=enable NETCDFHOME=/work2/06482/$USER/stampede2/opt/default",
    "env.adcirc.build.ADCIRC_PROFILE_NAME"     : "swan",
   "env.adcirc.build.ADCIRC_BINS"       : "padcirc adcirc adcprep hstime aswip",
    "env.adcirc.build.ADCSWAN_BINS"      : "adcswan padcswan",
    "env.adcirc.build.SWAN_UTIL_BINS"    : "unhcat.exe",
    "env.adcirc.build.PATH"              : "/work2/06482/$USER/stampede2/opt/default/python-2.7.18/bin:/home1/06482/$USER/.local/bin:/work2/06482/$USER/stampede2/opt/default/perl5/bin:/work2/06482/$USER/stampede2/opt/default/perl5/perls/perl-5.32.0/bin:/work2/06482/$USER/stampede2/opt/default/bin:/work2/06482/$USER/stampede2/asgs/cloud/general:/work2/06482/$USER/stampede2/asgs/bin:/work2/06482/$USER/stampede2/asgs/my-bin:/work2/06482/$USER/stampede2/asgs/cloud/general:/work2/06482/$USER/stampede2/asgs/config:/work2/06482/$USER/stampede2/asgs/config/tests:/work2/06482/$USER/stampede2/asgs/input:/work2/06482/$USER/stampede2/asgs/input/data_assimilation:/work2/06482/$USER/stampede2/asgs/monitoring:/work2/06482/$USER/stampede2/asgs/output:/work2/06482/$USER/stampede2/asgs/output/cera_contour:/work2/06482/$USER/stampede2/asgs/output/cpra_postproc:/work2/06482/$USER/stampede2/asgs/output/cpra_postproc/MEX:/work2/06482/$USER/stampede2/asgs/output/cpra_postproc/old/tools:/work2/06482/$USER/stampede2/asgs/output/Cuba_post:/work2/06482/$USER/stampede2/asgs/output/FG49:/work2/06482/$USER/stampede2/asgs/output/NGOM_post:/work2/06482/$USER/stampede2/asgs/output/PartTrack:/work2/06482/$USER/stampede2/asgs/output/postProcessFlux:/work2/06482/$USER/stampede2/asgs/output/POSTPROC_KMZGIS:/work2/06482/$USER/stampede2/asgs/output/POSTPROC_KMZGIS/FigGen:/work2/06482/$USER/stampede2/asgs/output/POSTPROC_KMZGIS/RenciGETools-10/src:/work2/06482/$USER/stampede2/asgs/output/test:/work2/06482/$USER/stampede2/asgs/output/TRACKING_FILES:/work2/06482/$USER/stampede2/asgs/output/validation:/work2/06482/$USER/stampede2/asgs/tides:/work2/06482/$USER/stampede2/asgs/util:/work2/06482/$USER/stampede2/asgs/util/admin:/work2/06482/$USER/stampede2/asgs/util/input:/work2/06482/$USER/stampede2/asgs/util/input/mesh:/work2/06482/$USER/stampede2/asgs/util/input/nodalattr:/work2/06482/$USER/stampede2/asgs/util/output:/work2/06482/$USER/stampede2/asgs/util/troubleshooting:/home1/06482/$USER/opt/bin:/home1/06482/$USER/bin:/home1/06482/$USER/local/bin:/home1/06482/$USER/opt/bin:/home1/06482/$USER/bin:/home1/06482/$USER/local/bin:/opt/apps/xalt/xalt/bin:/opt/apps/cmake/3.16.1/bin:/opt/apps/intel18/python2/2.7.15/bin:/opt/apps/autotools/1.1/bin:/opt/apps/git/2.24.1/bin:/opt/apps/libfabric/1.7.0/bin:/opt/apps/intel18/impi/18.0.2/bin:/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/bin:/opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64:/opt/apps/gcc/6.3.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:.",
    "env.adcirc.build.LD_LIBRARY_PATH"   : "/work2/06482/$USER/stampede2/opt/default/lib:/home1/06482/$USER/local/lib:/home1/06482/$USER/local/lib:/opt/apps/intel18/python2/2.7.15/lib:/opt/apps/libfabric/1.7.0/lib:/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/lib:/opt/intel/debugger_2018/libipt/intel64/lib:/opt/intel/debugger_2018/iga/lib:/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2018.2.199/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64:/opt/apps/gcc/6.3.0/lib64:/opt/apps/gcc/6.3.0/lib",
    "env.adcirc.build.LD_INCLUDE_PATH"   : "/work2/06482/$USER/stampede2/opt/default/include",
    "adcirc.build.fortran.mpif90"        : "/opt/apps/intel18/impi/18.0.2/bin/mpif90",
    "adcirc.build.fortran.compiler"      : "/opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort",
    "adcirc.build.fortran.compiler.version" : "ifort (IFORT) 18.0.2 20180210",
    "adcirc.build.c.compiler"            : "/opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc",
    "adcirc.build.c.compiler.version"    : "icc (ICC) 18.0.2 20180210",
    "adcirc.build.modules.loaded"        : "  1) intel/18.0.2   2) libfabric/1.7.0   3) impi/18.0.2   4) git/2.24.1   5) autotools/1.1   6) python2/2.7.15   7) cmake/3.16.1   8) xalt/2.10.2   9) TACC"
  }

[jasonfleming]

Hey @wwlwpd sorry for the delay, this is beautiful, thank you.