Coordinate testing fixes

[jgfouca]

The dashboard has looked pretty bad for a while. I'd like to make a task list to get us back on track. Please see the list below for the list of problems and checkboxes for potential solutions, I'd like volunteers for all checkboxes

• Weaver internal compiler errors
  • [x] Make a list of problematic source files and break them into smaller compilation units (this is not necessary if the compiler upgrade works).
  • [x] Upgrade compiler/TPL toolchain on weaver @bartgol
• cuda-memcheck fails : the last cdash report for scream_unit_tests_cuda_mem_check shows 6 tests with memcheck fails
  • [x] Fix homme_shoc_cld_p3_rrtmgp_ut_np1_omp1 cuda-memcheck fail @bartgol
  • [x] Fix p3_run_and_cmp_cxx_ut_np1_omp1 cuda-memcheck fail @bartgol
  • [x] Fix rrtmgp_standalone_ut_np1_omp1 cuda-memcheck fail @bartgol
  • [x] Fix sc_ut_ut_np1_omp1 cuda-memcheck fail @bartgol
  • [x] Fix shoc_cld_p3_rrtmgp_ut_np1_omp1 cuda-memcheck fail @bartgol
  • [x] Fix shoc_run_and_cmp_cxx_ut_np1_omp1 cuda-memcheck fail @bartgol
  • [x] Why are cuda-memcheck builds not being submitted successfully to cdash most nights? Timeout issues? @bartgol I checked the Jenkins logs for the gap (no report from the 18th till the 26th) and I saw lots of ICEs during baselines generation. If test-all-scream craps out during baselines gen, we don't get a cdash submission (which is performed during the ctest call, during the branch test phase)
• syrah and quartz are not reporting to cdash this appears to be a problem with test-launcher, which has never worked on these machines
  • [x] Fix it @jgfouca
• mappy valgrind fails
  • [x] Fix dyn_grid_io_ut_np1_omp1 valgrind error @bartgol
  • [x] Fix io_test_ut_np1_omp1 valgrind error @bartgol
  • [x] Fix p3_standalone_ut_np1_omp1 valgrind error
  • [x] Why are valgrind builds not being submitted successfully to cdash most nights? Timeout issues? @bartgol Looks like they are submitted successfully, at least since the end of September. There was a ~10 days gap, and in those days valgrind took up to 2h for some tests (I believe that's after we started using mpirun for all tests, and before we fixed everything with test-launcher). That said, since Oct 8th, build time went up from <5min to 90-100min. I believe that's when @jgfouca fixed our test invoke line from valgrind test-launcher -e <exec> to test-launcher -e valgrind <exec> (before we were valgrind-ing test-launcher, rather than our execs). So It hink the current 90-100min build time are likely to stay. If that's too much, we can probably add a 'DEV' test profile, where we run just 1 time step (SHORT runs 2 time steps), which might cut our long tests runtime by 40-50%.
• mappy CIME failure to report
  • [x] We seem to be missing some reports from mappy's CIME run @jgfouca
• Lassen fails , looks like all the errors are Error: Remote JSM server is not responding on host lassen71010-25-2021 00:57:53:454 74949 main: Error initializing RM connection. Exiting. There appear to also be issues with taskset , failed to set pid 0's affinity: Invalid argument
  • [x] Investigate. Did something change on Lassen side or SCREAM side to cause break? @AaronDonahue
  • [x] Fix the taskset issue, the system reserves cores 0-8 so the user cannot taskset to those @jgfouca
  • [x] Fix the JSM server issue
• cori-knl CIME build errors
  • [x] Fix build error for case SMS_D_Ln5.ne4_ne4.F2000SCREAMv1.cori-knl_intel.scream-3min_ts @jgfouca

[AaronDonahue]

Marking the triage for whats going on with lassen as completed since I think we have a pretty good idea of the issue now.

[bartgol]

Re: gcc 8 toolchain. I ran test-all on PR #1260 (which exhibited an ICE during AT testing on weaver), and was able to reproduce the ICE. I then merged the gcc8 branch into it, and successfully built.

@jgfouca do you want to pull the trigger on switching to gcc8 on weaver? PR #1282 brought in a fix in Homme that was needed to avoid a separate (and much less likely to strike us again) ICE in Homme, so now we could do a PR with just machine_specs.py and weaver.cmake changes. I can package the PR in 5 minutes.

[jgfouca]

@bartgol yes, please upgrade weaver to gcc8

[jgfouca]

All tasks complete. Testing is in much better shape (before mappy maintenance happened). Thanks everyone!