search

NOAA-EMC / GDASApp

Global Data Assimilation System Application
GNU Lesser General Public License v2.1
15 stars 31 forks source link

Use install option in GDASApp build #1302

Open RussTreadon-NOAA opened 5 days ago

RussTreadon-NOAA commented 5 days ago

EE2 does not require executable to be copied to the job run directory $DATA. Executables can be referenced from $HOMEmodel/exec.

This issue is opened to use the cmake install option to copy GDASApp executables, modules, and libraries in directories more aligned with the EE2 requirement.

RussTreadon-NOAA commented 5 days ago

This issue is an EE2 compliance issue being tracked on GDASApp issue #1254

RussTreadon-NOAA commented 5 days ago

As a test make the following modification to sorc/build_gdas.sh in a working copy of g-w branch feature/radbcor (see g-w PR #2875)

@@ -24,6 +24,6 @@ shift $((OPTIND-1))
 # shellcheck disable=SC2086
 BUILD_JOBS="${BUILD_JOBS:-8}" \
 WORKFLOW_BUILD="ON" \
-./gdas.cd/build.sh ${_opts} -f
+./gdas.cd/build.sh ${_opts} -f -p /work/noaa/da/rtreadon/git/global-workflow/radbcor/bin

 exit

Currently executing build_gdas.sh on Hercules. Install is proceeding very slowly. Is this expected @danholdaway ? Did you simply add -p $INSTALL_PATH to build_gdas.sh in your test?

danholdaway commented 5 days ago

@RussTreadon-NOAA in my test I modified the build script as I believe the logic is flawed when you choose install. Here is the logic:

# Build
echo "Building ..."
set -x
if [[ $BUILD_JCSDA == 'YES' ]]; then
  make -j ${BUILD_JOBS:-6} VERBOSE=$BUILD_VERBOSE
else
  builddirs="gdas iodaconv land-imsproc land-jediincr gdas-utils bufr-query"
  for b in $builddirs; do
    cd $b
    make -j ${BUILD_JOBS:-6} VERBOSE=$BUILD_VERBOSE
    cd ../
  done
fi

# Install
if [[ -n ${INSTALL_PREFIX:-} ]]; then
  echo "Installing ..."
  set -x
  make install
  set +x
fi

Note that the script loops over the builddirs and makes all the code associated with these packages. But once it's done with that there is still a very large amount that remains unbuilt, e.g. ufo tests, saber tests, oops toy model etc. That final make install will make all that remaining code but it will only do it with one processor. If you want to do install you have to make absolutely everything, there is no two ways about it. Otherwise you won't copy the libraries to the install path.

For quicker build with install it needs to be something like:

# Build
echo "Building ..."
set -x
if [[ -z ${INSTALL_PREFIX:-} ]]; then
  if [[ $BUILD_JCSDA == 'YES' ]]; then
    make -j ${BUILD_JOBS:-6} VERBOSE=$BUILD_VERBOSE
  else
    builddirs="gdas iodaconv land-imsproc land-jediincr gdas-utils bufr-query"
    for b in $builddirs; do
      cd $b
      make -j ${BUILD_JOBS:-6} VERBOSE=$BUILD_VERBOSE
      cd ../
    done
  fi
fi
set +x

# Install
if [[ -n ${INSTALL_PREFIX:-} ]]; then
  echo "Installing ..."
  set -x
  make -j ${BUILD_JOBS:-6} install
  set +x
fi

I.e. skip the builddirs since they will be built anyway and let CMake figure out an optimal parallel strategy for building them. However note that it will always be a lot slower than the normal way of building because 1000s of tests will be built.

danholdaway commented 5 days ago

Relooking at it you could combine BUILD_JCSDA with INSTALL_PREFIX. Perhaps that was the original intention?

RussTreadon-NOAA commented 5 days ago

Thank you @danholdaway for explaining why install is taking so much time. I'm executing faulty logic. Let me try your suggestions. Another item on our TODO list is turning off all the JEDI tests (GDASApp issue #1269

RussTreadon-NOAA commented 5 days ago

Tested the suggested change to build.sh. Build with INSTALL_PREFIX set is progressing very slowly (now approaching 3 hours). Will revisit next week.

danholdaway commented 2 days ago

In that case it might be a prerequisite that the building of the JEDI tests can be turned off. We could proceed with this relatively easily but it's a feature that has met resistance in the past. I think some of that resistance comes from a difference in working styles. Those opposed argue that the cost of building JEDI is in the noise of the time it takes to run an experiment so why introduce the extra flag and complicating of the build infrastructure, which is already complicated.

One idea I was playing around with before I went on leave is whether we could try to eliminate the need to build JEDI for most people. Then the issue might go away. I think it would have to be done in tandem with implementing more of a process for updating the JEDI hashes. Perhaps we could brainstorm this week?

RussTreadon-NOAA commented 2 days ago

The Hercules build with INSTALL_PREFIX set eventually failed with

-- Installing: /work/noaa/da/rtreadon/git/global-workflow/radbcor/bin/share/soca/testdata/72x35x25/INPUT/ocean_hgrid.nc
-- Installing: /work/noaa/da/rtreadon/git/global-workflow/radbcor/bin/share/soca/testdata/72x35x25/INPUT/ocean_topog.nc
CMake Error at soca/test/cmake_install.cmake:65 (file):
  file INSTALL cannot find
  "/work/noaa/da/rtreadon/git/global-workflow/radbcor/sorc/gdas.cd/bundle/soca/test/Data/rossrad.dat":
  No such file or directory.
Call Stack (most recent call first):
  soca/cmake_install.cmake:84 (include)
  cmake_install.cmake:122 (include)

make: *** [Makefile:130: install] Error 1

File rossrad.nc replaced rossrad.dat. However, rossrad.dat is still referenced in three soca files.

First, gdas.cd/sorc/soca/test/CMakeLists.txt has

set( soca_install_data
  Data/rossrad.dat
  Data/godas_sst_bgerr.nc )
install(FILES ${soca_install_data}
        DESTINATION ${INSTALL_DATA_DIR}/testdata/ )

The two other files that reference rossrad.dat are

gdas.cd/sorc/soca/tutorial/tutorial_tools.sh:    ln -sf $datadir/Data/rossrad.dat .
gdas.cd/sorc/soca/.gitattributes:test/Data/rossrad.dat filter=lfs diff=lfs merge=lfs -text
RussTreadon-NOAA commented 2 days ago

If the install option for the GDASApp build satisfies the EE2 executable requirement, then we should try to find a way to speed up the build with install.

We don't need JEDI ctests when building and installing GDASApp for use in operations. If turning off JEDI ctests speeds up the build and install for operations, we should figure out a way to make this happen.

RussTreadon-NOAA commented 2 days ago

@danholdaway , I agree with your brainstorming idea. The core GDASApp infrastructure team needs to develop a plan to work through the various items from the bi-weekly JEDI workflow sync meeting with EIB.

RussTreadon-NOAA commented 1 day ago

Work for this issue will be done in feature/install

RussTreadon-NOAA commented 1 day ago

@danholdaway recommended making the following change in CMakeLists.txt for the various JEDI submodules in sorc/

option( ENABLE_JEDI_CTESTS "Build JEDI ctests" ON )
if( ENABLE_JEDI_CTESTS )
  add_subdirectory( test )
endif()

The default behavior is for JEDI ctests to be active (ON). Users can add -DENABLE_JEDI_CTESTS=OFF to the cmake configure to turn off JEDI ctests.

feature/install was cloned on Hercules in /work/noaa/da/rtreadon/git/GDASApp/install/. The above scripting was added to CMakeLists.txt in the following JEDI submodules

        modified:   sorc/bufr-query (modified content)
        modified:   sorc/crtm (modified content)
        modified:   sorc/fv3-jedi (modified content)
        modified:   sorc/gsibec (modified content)
        modified:   sorc/ioda (modified content)
        modified:   sorc/iodaconv (modified content)
        modified:   sorc/saber (modified content)
        modified:   sorc/soca (modified content)
        modified:   sorc/ufo (modified content)
        modified:   sorc/vader (modified content)

build.sh was modified as follows

@@ -24,6 +24,7 @@ usage() {
   echo "  -f  force a clean build             DEFAULT: NO"
   echo "  -d  include JCSDA ctest data        DEFAULT: NO"
   echo "  -a  build everything in bundle      DEFAULT: NO"
+  echo "  -j  build with JEDI ctests          DEFAULT: OFF"
   echo "  -h  display this message and quit"
   echo
   exit 1
@@ -39,6 +40,7 @@ BUILD_VERBOSE="NO"
 CLONE_JCSDADATA="NO"
 CLEAN_BUILD="NO"
 BUILD_JCSDA="NO"
+ENABLE_JEDI_CTESTS="OFF"
 COMPILER="${COMPILER:-intel}"

 while getopts "p:t:c:hvdfa" opt; do
@@ -64,6 +66,9 @@ while getopts "p:t:c:hvdfa" opt; do
     a)
       BUILD_JCSDA=YES
       ;;
+    j)
+      ENABLE_JEDI_CTESTS=ON
+      ;;
     h|\?|:)
       usage
       ;;
@@ -98,6 +103,10 @@ mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
 # If INSTALL_PREFIX is not empty; install at INSTALL_PREFIX
 [[ -n "${INSTALL_PREFIX:-}" ]] && CMAKE_OPTS+=" -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX}"

+# Activate JEDI ctests if requested
+ENABLE_JEDI_CTESTS=${ENABLE_JEDI_CTESTS:-"OFF"}
+CMAKE_OPTS+=" -DENABLE_JEDI_CTESTS=${ENABLE_JEDI_CTESTS}"
+
 # activate tests based on if this is cloned within the global-workflow
 WORKFLOW_BUILD=${WORKFLOW_BUILD:-"OFF"}
 CMAKE_OPTS+=" -DWORKFLOW_TESTS=${WORKFLOW_BUILD}"

./build.sh was then executed. ctest -N was executed in build/ upon completion. 238 tests remain. Prior to this change there were 1952 ctests. Below is the list of remaining ctests

Test project /work/noaa/da/rtreadon/git/GDASApp/install/build
  Test   #1: gsw_poly_check
  Test   #2: gsw_check_functions
  Test   #3: bufr_query_coding_norms
  Test   #4: oops_coding_norms
  Test   #5: test_oops_base_dummy_run_one
  Test   #6: test_oops_base_dummy_run_no_validate
  Test   #7: test_oops_base_dummy_run_validate_zero
  Test   #8: test_oops_base_dummy_run_bad_arg_zero
  Test   #9: test_oops_base_dummy_run_bad_arg_one
  Test  #10: test_oops_base_dummy_run_bad_arg_two
  Test  #11: test_oops_base_dummy_run_bad_arg_three
  Test  #12: test_oops_base_dummy_run_help
  Test  #13: test_oops_base_dummy_run_h
  Test  #14: test_oops_base_variables
  Test  #15: test_oops_base_obsvariables
  Test  #16: test_base_posttimer
  Test  #17: test_util_signal_trap_fpe_div_by_zero
  Test  #18: test_util_signal_trap_fpe_invalid_op
  Test  #19: test_util_signal_trap_fpe_valid_op
  Test  #20: test_util_stacktrace
  Test  #21: test_util_random
  Test  #22: test_util_pushstringvector
  Test  #23: test_util_parameters
  Test  #24: test_generic_atlas_interpolator
  Test  #25: test_generic_unstructured_interpolator
  Test  #26: test_generic_atlas_global_interpolator
  Test  #27: test_generic_unstructured_global_interpolator
  Test  #28: test_generic_unstructured_global_interpolator_parallel
  Test  #29: test_generic_gc99
  Test  #30: test_generic_soar
  Test  #31: test_coupled_splitvariables
  Test  #32: test_util_isanypointinvolumeinterior
  Test  #33: test_util_partialdatetime
  Test  #34: test_util_datetime
  Test  #35: test_util_duration
  Test  #36: test_util_intset_parser
  Test  #37: test_util_scalarormap
  Test  #38: test_util_floatcompare
  Test  #39: test_util_compositepath
  Test  #40: test_util_stringfunctions
  Test  #41: test_util_testreference
  Test  #42: test_util_range
  Test  #43: test_mpi_mpi
  Test  #44: test_fft_multiple
  Test  #45: test_util_algorithms
  Test  #46: test_util_comparenvectors
  Test  #47: test_util_missingvalues
  Test  #48: test_util_associativecontainers
  Test  #49: test_util_propertiesofnvectors
  Test  #50: test_util_localenvironment
  Test  #51: test_util_typetraits
  Test  #52: test_util_wildcard
  Test  #53: test_util_configfunctions
  Test  #54: test_util_confighelpers
  Test  #55: test_util_timewindow
  Test  #56: test_util_arrayutil
  Test  #57: test_base_fieldsets
  Test  #58: test_util_fieldset_helpers_and_operations
  Test  #59: test_util_fieldset_subcommunicators
  Test  #60: test_util_functionspace_helpers
  Test  #61: test_util_functionspace_helpers_p2
  Test  #62: test_util_functionspace_helpers_p4
  Test  #63: test_assimilation_fullgmres
  Test  #64: test_assimilation_rotmat
  Test  #65: test_assimilation_solvematrixequation
  Test  #66: test_assimilation_spectrallmp
  Test  #67: test_assimilation_testvector3d
  Test  #68: test_assimilation_tridiagsolve
  Test  #69: vader_coding_norms
  Test  #70: saber_coding_norms_src
  Test  #71: saber_coding_norms_quench
  Test  #72: ioda_coding_norms
  Test  #73: test_ioda-collective-functions-h5file
  Test  #74: test_ioda-collective-functions-h5mem
  Test  #75: test_ioda-engines_complex_objects_strings-default
  Test  #76: test_ioda-engines_complex_objects_strings-h5file
  Test  #77: test_ioda-engines_complex_objects_strings-h5mem
  Test  #78: test_ioda-engines_complex_objects_strings-ObsStore
  Test  #79: test_ioda-chunks_and_filters-default
  Test  #80: test_ioda-chunks_and_filters-h5file
  Test  #81: test_ioda-chunks_and_filters-h5mem
  Test  #82: test_ioda-chunks_and_filters-ObsStore
  Test  #83: test_ioda-engines_data-selections-default
  Test  #84: test_ioda-engines_data-selections-h5file
  Test  #85: test_ioda-engines_data-selections-h5mem
  Test  #86: test_ioda-engines_data-selections-ObsStore
  Test  #87: test_ioda-engines_dim-selectors-default
  Test  #88: test_ioda-engines_dim-selectors-h5file
  Test  #89: test_ioda-engines_dim-selectors-h5mem
  Test  #90: test_ioda-engines_dim-selectors-ObsStore
  Test  #91: test_ioda-engines_exception
  Test  #92: test_ioda-fillvalues-default
  Test  #93: test_ioda-fillvalues-h5file
  Test  #94: test_ioda-fillvalues-h5mem
  Test  #95: test_ioda-fillvalues-ObsStore
  Test  #96: test_ioda-engines_io_templated_tests-default
  Test  #97: test_ioda-engines_io_templated_tests-h5file
  Test  #98: test_ioda-engines_io_templated_tests-h5mem
  Test  #99: test_ioda-engines_io_templated_tests-ObsStore
  Test #100: test_ioda-engines_hier_paths-default
  Test #101: test_ioda-engines_hier_paths-h5file
  Test #102: test_ioda-engines_hier_paths-h5mem
  Test #103: test_ioda-engines_hier_paths-ObsStore
  Test #104: test_ioda-engines_layouts_layoutobsgroupodb
  Test #105: test_ioda-engines_layouts_layoutobsgroup
  Test #106: test_ioda-engines_obsgroup-default
  Test #107: test_ioda-engines_obsgroup-h5file
  Test #108: test_ioda-engines_obsgroup-h5mem
  Test #109: test_ioda-engines_obsgroup-ObsStore
  Test #110: test_ioda-engines_obsgroup_append
  Test #111: test_ioda-engines_obsgroup_append_function
  Test #112: test_ioda-engines_sfuncs_concatstringvectors
  Test #113: test_ioda-engines_sfuncs_convertv1pathtov2path
  Test #114: test_ioda-engines_persist-default
  Test #115: test_ioda-engines_persist-h5file
  Test #116: test_ioda-engines_persist-h5mem
  Test #117: test_ioda-engines_persist-ObsStore
  Test #118: test_ioda-engines_list_objects-default
  Test #119: test_ioda-engines_list_objects-h5file
  Test #120: test_ioda-engines_list_objects-h5mem
  Test #121: test_ioda-engines_list_objects-ObsStore
  Test #122: test_ioda-engines_hasvariables_stitchcomplementaryvars
  Test #123: test_ioda-engines_hasvariables_convertvariableunits
  Test #124: ioda-python
  Test #125: ioda-obsspace-python
  Test #126: test_ioda-engines_examples_prep_data
  Test #127: test_ioda-engines-01-default
  Test #128: test_ioda-engines-01-h5file
  Test #129: test_ioda-engines-01-h5mem
  Test #130: test_ioda-engines-01-obsstore
  Test #131: test_ioda-engines-02-default
  Test #132: test_ioda-engines-02-h5file
  Test #133: test_ioda-engines-02-h5mem
  Test #134: test_ioda-engines-02-obsstore
  Test #135: test_ioda-engines-03-default
  Test #136: test_ioda-engines-03-h5file
  Test #137: test_ioda-engines-03-h5mem
  Test #138: test_ioda-engines-03-obsstore
  Test #139: test_ioda-engines-04-default
  Test #140: test_ioda-engines-04-h5file
  Test #141: test_ioda-engines-04-h5mem
  Test #142: test_ioda-engines-04-obsstore
  Test #143: test_ioda-engines-05a-default
  Test #144: test_ioda-engines-05a-h5file
  Test #145: test_ioda-engines-05a-h5mem
  Test #146: test_ioda-engines-05a-obsstore
  Test #147: test_ioda-engines-05b-default
  Test #148: test_ioda-engines-05b-h5file
  Test #149: test_ioda-engines-05b-h5mem
  Test #150: test_ioda-engines-05b-obsstore
  Test #151: test_ioda-engines-00-Strings-F
  Test #152: test_ioda-engines-00-VecStrings-F
  Test #153: test_ioda-engines-01-GroupsAndObsSpaces-F
  Test #154: test_ioda-engines-02-Attributes-F
  Test #155: test_ioda-engines-03-Variables-F
  Test #156: test_ioda-engines-01-Py
  Test #157: test_ioda-engines-02-Py
  Test #158: test_ioda-engines-03-Py
  Test #159: test_ioda-engines-04-Py
  Test #160: test_ioda-engines-05-Py
  Test #161: test_ioda-engines-06-Py
  Test #162: test_ioda-engines-07a-Py-ObsSpaceClass
  Test #163: test_ioda-engines-07b-Py-ObsSpaceClassDataTypes
  Test #164: test_ioda-engines-chrono-Py
  Test #165: test_ioda-engines_chrono-default
  Test #166: test_ioda-engines_chrono-h5file
  Test #167: test_ioda-engines_chrono-h5mem
  Test #168: test_ioda-engines_complex_objects_array_from_struct-default
  Test #169: test_ioda-engines_complex_objects_array_from_struct-h5file
  Test #170: test_ioda-engines_complex_objects_array_from_struct-h5mem
  Test #171: test_ioda-engines_complex_objects_array_from_struct-ObsStore
  Test #172: test_ioda-engines_fixed_length_strings-default
  Test #173: test_ioda-engines_fixed_length_strings-h5file
  Test #174: test_ioda-engines_fixed_length_strings-h5mem
  Test #175: test_ioda-engines_fixed_length_strings_client-default
  Test #176: test_ioda-engines_fixed_length_strings_client-h5file
  Test #177: test_ioda-engines_fixed_length_strings_client-h5mem
  Test #178: test_ioda-engines_named_types-default
  Test #179: test_ioda-engines_named_types-h5file
  Test #180: test_ioda-engines_named_types-h5mem
  Test #181: test_ioda-engines_units
  Test #182: test_ioda-engines_basic_math
  Test #183: test_ioda-engines_variables_math
  Test #184: ioda_pyiodautils_coding_norms
  Test #185: ufo_coding_norms
  Test #186: test_ufo_opr_autogenerated
  Test #187: test_autogeneratedfilter
  Test #188: test_femps_csgrid
  Test #189: fv3jedi_test_tier1_coding_norms
  Test #190: soca_coding_norms
  Test #191: test_gdasapp_util_coding_norms
  Test #192: test_gdasapp_util_ioda_example
  Test #193: test_gdasapp_util_prepdata
  Test #194: test_gdasapp_util_rads2ioda
  Test #195: test_gdasapp_util_ghrsst2ioda
  Test #196: test_gdasapp_util_rtofstmp
  Test #197: test_gdasapp_util_rtofssal
  Test #198: test_gdasapp_util_smap2ioda
  Test #199: test_gdasapp_util_smos2ioda
  Test #200: test_gdasapp_util_viirsaod2ioda
  Test #201: test_gdasapp_util_icecamsr2ioda
  Test #202: test_gdasapp_util_icecmirs2ioda
  Test #203: test_gdasapp_util_icecjpssrr2ioda
  Test #204: test_dautils_ioda_example
  Test #205: iodaconv_compo_coding_norms
  Test #206: iodaconv_gsi_ncdiag_coding_norms
  Test #207: iodaconv_goes_coding_norms
  Test #208: iodaconv_hdf5_coding_norms
  Test #209: iodaconv_land_coding_norms
  Test #210: iodaconv_lib-python_coding_norms
  Test #211: iodaconv_marine_coding_norms
  Test #212: iodaconv_conventional_coding_norms
  Test #213: iodaconv_ncep_coding_norms
  Test #214: iodaconv_ssec_coding_norms
  Test #215: iodaconv_wrfda_ncdiag_coding_norms
  Test #216: iodaconv_singleob_coding_norms
  Test #217: iodaconv_mrms_coding_norms
  Test #218: iodaconv_gnssro_coding_norms
  Test #219: iodaconv_bufr_coding_norms
  Test #220: iodaconv_satbias_py_coding_norms
  Test #221: iodaconv_gsi_varbc_coding_norms
  Test #222: test_gdasapp_check_python_norms
  Test #223: test_gdasapp_check_yaml_keys
  Test #224: test_gdasapp_jedi_increment_to_fv3
  Test #225: test_gdasapp_fv3jedi_fv3inc
  Test #226: test_gdasapp_snow_create_ens
  Test #227: test_gdasapp_snow_imsproc
  Test #228: test_gdasapp_snow_apply_jediincr
  Test #229: test_gdasapp_snow_letkfoi_snowda
  Test #230: test_gdasapp_convert_bufr_adpsfc_snow
  Test #231: test_gdasapp_convert_bufr_adpsfc
  Test #232: test_gdasapp_convert_gsi_satbias
  Test #233: test_bufr2ioda_insitu_profile_argo
  Test #234: test_bufr2ioda_insitu_profile_bathy
  Test #235: test_bufr2ioda_insitu_profile_glider
  Test #236: test_bufr2ioda_insitu_profile_tesac
  Test #237: test_bufr2ioda_insitu_profile_xbtctd
  Test #238: test_bufr2ioda_insitu_surface_trkob

Total Tests: 238
CoryMartin-NOAA commented 1 day ago

Just chining in here since I saw this thread, I think we should basically require JCSDA core to accept PRs that make building the ctests optional. There is no reason why the default can't be the current behavior and we can add some extra CMake logic to skip these thousands of tests. If JEDI is supposed to be flexible, and used by all, then this is something that needs to be added as an option.

RussTreadon-NOAA commented 1 day ago

Add ENABLE_JEDI_CTESTS to a few more CMakeLists.txt. Down to 168 tests

(gdasapp) hercules-login-2:/work/noaa/da/rtreadon/git/GDASApp/install/build$ ctest -N
Test project /work/noaa/da/rtreadon/git/GDASApp/install/build
  Test   #1: bufr_query_coding_norms
  Test   #2: oops_coding_norms
  Test   #3: test_oops_base_dummy_run_one
  Test   #4: test_oops_base_dummy_run_no_validate
  Test   #5: test_oops_base_dummy_run_validate_zero
  Test   #6: test_oops_base_dummy_run_bad_arg_zero
  Test   #7: test_oops_base_dummy_run_bad_arg_one
  Test   #8: test_oops_base_dummy_run_bad_arg_two
  Test   #9: test_oops_base_dummy_run_bad_arg_three
  Test  #10: test_oops_base_dummy_run_help
  Test  #11: test_oops_base_dummy_run_h
  Test  #12: test_oops_base_variables
  Test  #13: test_oops_base_obsvariables
  Test  #14: test_base_posttimer
  Test  #15: test_util_signal_trap_fpe_div_by_zero
  Test  #16: test_util_signal_trap_fpe_invalid_op
  Test  #17: test_util_signal_trap_fpe_valid_op
  Test  #18: test_util_stacktrace
  Test  #19: test_util_random
  Test  #20: test_util_pushstringvector
  Test  #21: test_util_parameters
  Test  #22: test_generic_atlas_interpolator
  Test  #23: test_generic_unstructured_interpolator
  Test  #24: test_generic_atlas_global_interpolator
  Test  #25: test_generic_unstructured_global_interpolator
  Test  #26: test_generic_unstructured_global_interpolator_parallel
  Test  #27: test_generic_gc99
  Test  #28: test_generic_soar
  Test  #29: test_coupled_splitvariables
  Test  #30: test_util_isanypointinvolumeinterior
  Test  #31: test_util_partialdatetime
  Test  #32: test_util_datetime
  Test  #33: test_util_duration
  Test  #34: test_util_intset_parser
  Test  #35: test_util_scalarormap
  Test  #36: test_util_floatcompare
  Test  #37: test_util_compositepath
  Test  #38: test_util_stringfunctions
  Test  #39: test_util_testreference
  Test  #40: test_util_range
  Test  #41: test_mpi_mpi
  Test  #42: test_fft_multiple
  Test  #43: test_util_algorithms
  Test  #44: test_util_comparenvectors
  Test  #45: test_util_missingvalues
  Test  #46: test_util_associativecontainers
  Test  #47: test_util_propertiesofnvectors
  Test  #48: test_util_localenvironment
  Test  #49: test_util_typetraits
  Test  #50: test_util_wildcard
  Test  #51: test_util_configfunctions
  Test  #52: test_util_confighelpers
  Test  #53: test_util_timewindow
  Test  #54: test_util_arrayutil
  Test  #55: test_base_fieldsets
  Test  #56: test_util_fieldset_helpers_and_operations
  Test  #57: test_util_fieldset_subcommunicators
  Test  #58: test_util_functionspace_helpers
  Test  #59: test_util_functionspace_helpers_p2
  Test  #60: test_util_functionspace_helpers_p4
  Test  #61: test_assimilation_fullgmres
  Test  #62: test_assimilation_rotmat
  Test  #63: test_assimilation_solvematrixequation
  Test  #64: test_assimilation_spectrallmp
  Test  #65: test_assimilation_testvector3d
  Test  #66: test_assimilation_tridiagsolve
  Test  #67: vader_coding_norms
  Test  #68: saber_coding_norms_src
  Test  #69: saber_coding_norms_quench
  Test  #70: ioda_coding_norms
  Test  #71: test_ioda-engines_examples_prep_data
  Test  #72: test_ioda-engines-01-default
  Test  #73: test_ioda-engines-01-h5file
  Test  #74: test_ioda-engines-01-h5mem
  Test  #75: test_ioda-engines-01-obsstore
  Test  #76: test_ioda-engines-02-default
  Test  #77: test_ioda-engines-02-h5file
  Test  #78: test_ioda-engines-02-h5mem
  Test  #79: test_ioda-engines-02-obsstore
  Test  #80: test_ioda-engines-03-default
  Test  #81: test_ioda-engines-03-h5file
  Test  #82: test_ioda-engines-03-h5mem
  Test  #83: test_ioda-engines-03-obsstore
  Test  #84: test_ioda-engines-04-default
  Test  #85: test_ioda-engines-04-h5file
  Test  #86: test_ioda-engines-04-h5mem
  Test  #87: test_ioda-engines-04-obsstore
  Test  #88: test_ioda-engines-05a-default
  Test  #89: test_ioda-engines-05a-h5file
  Test  #90: test_ioda-engines-05a-h5mem
  Test  #91: test_ioda-engines-05a-obsstore
  Test  #92: test_ioda-engines-05b-default
  Test  #93: test_ioda-engines-05b-h5file
  Test  #94: test_ioda-engines-05b-h5mem
  Test  #95: test_ioda-engines-05b-obsstore
  Test  #96: test_ioda-engines-00-Strings-F
  Test  #97: test_ioda-engines-00-VecStrings-F
  Test  #98: test_ioda-engines-01-GroupsAndObsSpaces-F
  Test  #99: test_ioda-engines-02-Attributes-F
  Test #100: test_ioda-engines-03-Variables-F
  Test #101: test_ioda-engines-01-Py
  Test #102: test_ioda-engines-02-Py
  Test #103: test_ioda-engines-03-Py
  Test #104: test_ioda-engines-04-Py
  Test #105: test_ioda-engines-05-Py
  Test #106: test_ioda-engines-06-Py
  Test #107: test_ioda-engines-07a-Py-ObsSpaceClass
  Test #108: test_ioda-engines-07b-Py-ObsSpaceClassDataTypes
  Test #109: test_ioda-engines-chrono-Py
  Test #110: test_ioda-engines_chrono-default
  Test #111: test_ioda-engines_chrono-h5file
  Test #112: test_ioda-engines_chrono-h5mem
  Test #113: test_ioda-engines_complex_objects_array_from_struct-default
  Test #114: test_ioda-engines_complex_objects_array_from_struct-h5file
  Test #115: test_ioda-engines_complex_objects_array_from_struct-h5mem
  Test #116: test_ioda-engines_complex_objects_array_from_struct-ObsStore
  Test #117: test_ioda-engines_fixed_length_strings-default
  Test #118: test_ioda-engines_fixed_length_strings-h5file
  Test #119: test_ioda-engines_fixed_length_strings-h5mem
  Test #120: test_ioda-engines_fixed_length_strings_client-default
  Test #121: test_ioda-engines_fixed_length_strings_client-h5file
  Test #122: test_ioda-engines_fixed_length_strings_client-h5mem
  Test #123: test_ioda-engines_named_types-default
  Test #124: test_ioda-engines_named_types-h5file
  Test #125: test_ioda-engines_named_types-h5mem
  Test #126: test_ioda-engines_units
  Test #127: test_ioda-engines_basic_math
  Test #128: test_ioda-engines_variables_math
  Test #129: ioda_pyiodautils_coding_norms
  Test #130: ufo_coding_norms
  Test #131: test_femps_csgrid
  Test #132: fv3jedi_test_tier1_coding_norms
  Test #133: soca_coding_norms
  Test #134: test_dautils_ioda_example
  Test #135: iodaconv_compo_coding_norms
  Test #136: iodaconv_gsi_ncdiag_coding_norms
  Test #137: iodaconv_goes_coding_norms
  Test #138: iodaconv_hdf5_coding_norms
  Test #139: iodaconv_land_coding_norms
  Test #140: iodaconv_lib-python_coding_norms
  Test #141: iodaconv_marine_coding_norms
  Test #142: iodaconv_conventional_coding_norms
  Test #143: iodaconv_ncep_coding_norms
  Test #144: iodaconv_ssec_coding_norms
  Test #145: iodaconv_wrfda_ncdiag_coding_norms
  Test #146: iodaconv_singleob_coding_norms
  Test #147: iodaconv_mrms_coding_norms
  Test #148: iodaconv_gnssro_coding_norms
  Test #149: iodaconv_bufr_coding_norms
  Test #150: iodaconv_satbias_py_coding_norms
  Test #151: iodaconv_gsi_varbc_coding_norms
  Test #152: test_gdasapp_check_python_norms
  Test #153: test_gdasapp_check_yaml_keys
  Test #154: test_gdasapp_jedi_increment_to_fv3
  Test #155: test_gdasapp_fv3jedi_fv3inc
  Test #156: test_gdasapp_snow_create_ens
  Test #157: test_gdasapp_snow_imsproc
  Test #158: test_gdasapp_snow_apply_jediincr
  Test #159: test_gdasapp_snow_letkfoi_snowda
  Test #160: test_gdasapp_convert_bufr_adpsfc_snow
  Test #161: test_gdasapp_convert_bufr_adpsfc
  Test #162: test_gdasapp_convert_gsi_satbias
  Test #163: test_bufr2ioda_insitu_profile_argo
  Test #164: test_bufr2ioda_insitu_profile_bathy
  Test #165: test_bufr2ioda_insitu_profile_glider
  Test #166: test_bufr2ioda_insitu_profile_tesac
  Test #167: test_bufr2ioda_insitu_profile_xbtctd
  Test #168: test_bufr2ioda_insitu_surface_trkob

Total Tests: 168

Trying to figure out the source for the following tests

test_util_
test_ioda-engines_
iodaconv_
test_bufr2ioda_
danholdaway commented 1 day ago

The usual command for adding a test is ecbuild_add_test so you could try to grep for that in every CMakeLists.txt across the source code directories.

RussTreadon-NOAA commented 1 day ago

Tedious process but down to 104 ctests returned by ctest -N.

danholdaway commented 11 hours ago

What's the make time at this point? Perhaps we can have a few tests being built. If the changes become convoluted tests are likely to creep back in with future code changes anyway.

RussTreadon-NOAA commented 6 hours ago

What's the make time at this point? Perhaps we can have a few tests being built. If the changes become convoluted tests are likely to creep back in with future code changes anyway.

The most recent build (configure & compile) on Hercules with 104 ctests took 36:50 (minutes:seconds). develop with 1899 ctests took 37:39 to build on Hercules. It seems wrong that the two timings are basically the same.