Closed hkershaw-brown closed 8 months ago
The builds that use specific libraries such as rttov will be addressed in a future pull request
In order to add ifx to the list of compilers to build with, we need to update fixsystem to include ifx
will do, thanks Marlee!
I've added ifx. To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests
To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests
^^^ This should be added to the README
And a follow up question @hkershaw-brown - if the submit_jobs.sh script submits a job for each compiler, are we expecting the ifx build to still run and just error out on all branches other than quickbuild_tests?
To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests
^^^ This should be added to the README
Nope don't add to the README. When this pull request is merged into main, mkmf.template.ifx.linux will exist on main. This 'To run' note was just for you for this pull request.
And a follow up question @hkershaw-brown - if the submit_jobs.sh script submits a job for each compiler, are we expecting the ifx build to still run and just error out on all branches other than quickbuild_tests?
Yes because mkmf.template.ifx.linux only exists on the quickbuild_tests branch.
Once quickbuild_tests is merged into main mkmf.template.ifx.linux will exist on main.
Should we add some code for teardown? Currently, consecutive runs will fail with
Directory exists: /glade/derecho/scratch/masmith/build_everything/nvhpc
This is the teardown: https://github.com/NCAR/DART/blob/a20971ad106b382c02056c6d269f57d50cfb423d/developer_tests/build_everything/run_all_quickbuilds.sh#L143
If you don't have a compiler.dateTime directory then the job did not finish.
So I think at that point it is worth manually investigating why the job has failed, rather than rm -rf
directories.
To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests
^^^ This should be added to the README
Nope don't add to the README. When this pull request is merged into main, mkmf.template.ifx.linux will exist on main. This 'To run' note was just for you for this pull request.
And a follow up question @hkershaw-brown - if the submit_jobs.sh script submits a job for each compiler, are we expecting the ifx build to still run and just error out on all branches other than quickbuild_tests?
Yes because mkmf.template.ifx.linux only exists on the quickbuild_tests branch. Once quickbuild_tests is merged into main mkmf.template.ifx.linux will exist on main.
That makes much more sense and is obvious in hindsight. For some reason, I thought you meant that mkmf.template.ifx.linux was not going to be added to main with this PR. Ignore this.
Sometimes Derecho is unable to make a connection to the DART remote repository, causing some of the jobs to fail with this message:
Cloning into 'DART'...
fatal: unable to access 'https://github.com/NCAR/DART.git/': Failed to connect to github.com port 443 after 1 ms: Couldn't connect to server
./run_all_quickbuilds.sh: line 51: cd: DART: No such file or directory
fatal: not a git repository (or any parent up to mount point /glade/derecho)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /glade/derecho)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
unknown branch
I have submitted a support request with CISL Help to address this issue.
Additional failures with cce:
Just making a note of these on this PR, as other failures are noted in the body, but we can create issues for these as well
masmith@derecho1:~/DART/developer_tests/build_everything> grep -a FAILED test-2/build-everything-cce.o2646766
cce RESULT: 0 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/clm/work/ FAILED
cce RESULT: 3 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/noah/work/ FAILED
cce RESULT: 6 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/wrf_hydro/work/ FAILED
cce RESULT: 12 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/wrf/work/ FAILED
CLM:
ftn -O2 -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cce/15.0.1/cuko/include -c /glade/u/home/masmith/DART/models/clm/dart_to_clm.f90
ftn-1569 ftn: WARNING UPDATE_SNOW, File = ../../../../../../u/home/masmith/DART/models/clm/dart_to_clm.f90, Line = 612, Column = 19
A DO loop variable or expression of type default real or double precision real is a deleted feature of the Fortran standard.
ftn-1569 ftn: WARNING UPDATE_SNOW, File = ../../../../../../u/home/masmith/DART/models/clm/dart_to_clm.f90, Line = 628, Column = 19
A DO loop variable or expression of type default real or double precision real is a deleted feature of the Fortran standard.
ftn-319 ftn: ERROR UPDATE_SNOW, File = ../../../../../../u/home/masmith/DART/models/clm/dart_to_clm.f90, Line = 752, Column = 76
A subscript must be a scalar integer expression.
Cray Fortran : Version 15.0.1 (20230120205242_66f7391d6a03cf932f321b9f6b1d8612ef5f362c)
Line in question - https://github.com/NCAR/DART/blob/74b4221e5f4e41e4de2980fc9ff8697ba4540a8b/models/clm/dart_to_clm.f90#L752
snlsno(ncolumn), which is the subscript for gain_dzsno, is a real(r8) - changed to integer (line 418) and it compiles
https://github.com/NCAR/DART/blob/74b4221e5f4e41e4de2980fc9ff8697ba4540a8b/models/clm/dart_to_clm.f90#L418
NOAH/WRF_HYDRO:
ftn -O2 -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cce/15.0.1/cuko/include -c /glade/u/home/masmith/DART/models/wrf_hydro/noah_hydro_mod.f90
Error message :: _expr_type: Invalid table type
Error detected :: File '/home/jenkins/crayftn/pdgcs/v_expr_utl.c', line 7360
Initiated from :: Line 1280 (v_main.c)
Optimizer built :: 2023-01-20 (production)
File :: /glade/u/home/masmith/DART/models/wrf_hydro/noah_hydro_mod.f90
Function :: getchannelgridcoords
at or near line :: 660
Compiler hash :: 66f7391d6a03cf932f321b9f6b1d8612ef5f362c
Target :: x86-milan
ftn-7991 ftn: INTERNAL GETCHANNELGRIDCOORDS, File = ../../../../../../u/home/masmith/DART/models/wrf_hydro/noah_hydro_mod.f90, Line = 660
INTERNAL COMPILER ERROR: "_expr_type: Invalid table type" (/home/jenkins/crayftn/pdgcs/v_expr_utl.c, line 7360, version 66f7391d6a03cf932f321b9f6b1d8612ef5f362c)
Line in question: https://github.com/NCAR/DART/blob/74b4221e5f4e41e4de2980fc9ff8697ba4540a8b/models/wrf_hydro/noah_hydro_mod.f90#L660
WRF:
Building WRF_DART_utilities/add_pert_where_high_refl build 18 of 29
............................................................................................... Makefile is ready.
ftn -O2 -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cce/15.0.1/cuko/include -c /glade/u/home/masmith/DART/models/wrf/WRF_DART_utilities/add_pert_where_high_refl.f90
ftn-292 ftn: ERROR ADD_PERT_WHERE_HIGH_REFL, File = ../../../../../../u/home/masmith/DART/models/wrf/WRF_DART_utilities/add_pert_where_high_refl.f90, Line = 37, Column = 8
"F2KCLI" is specified as the module name on a USE statement, but the compiler cannot find it.
Cray Fortran : Version 15.0.1 (20230120205242_66f7391d6a03cf932f321b9f6b1d8612ef5f362c)
Solution is to remove the following line: use f2kcli https://github.com/NCAR/DART/blob/74b4221e5f4e41e4de2980fc9ff8697ba4540a8b/models/wrf/WRF_DART_utilities/add_pert_where_high_refl.f90#L37 Compiles after this change
Description:
building on Ann's previous pull request https://github.com/NCAR/DART/pull/575 improved build_everything, submits a job on Derecho for each compiler, each job runs every quickbuild.sh
time: ccc ~12minutes gfortran ~2:30 minutes nvhpc ~4:40 minutes ifort ~4:40 minutes
Note several of the converters require external libraries or code to be added before compiling, ignoring these for this pull request (rttov, hdfeos, wrf code, ncep prepbuf code):
Other failures: #592 #352, #594
Types of changes
Documentation changes needed?
Tests
Derecho: build everything (that has a quickbuild.sh) in DART for cce, intel, gcc, nvhpc
Checklist for merging
Checklist for release
Testing Datasets