Add shared physics gwdo

weiwangncar commented 8 months ago

TYPE: enhancement, new feature

KEYWORDS: shared physics, gwdo scheme

SOURCE: internal

DESCRIPTION OF CHANGES: This PR makes use of the GWDO code (gwd_opt=1) from the future MMM-physics repository, which are CCPP-compliant. To do so, we

copied module_bl_gwdo.F from latest MPAS driver. This routine is now a driver to call the CCPP-compliant bl_gwdo.F90 in the shared physics directory: physics_mmm.
added errflg and errmsg in module_pbl_driver.F and the call to the new module_bl_gwdo.F. Both are required by the CCPP-compliant physics. Change DX to DX2D.
modified Makefile to 1) compile bl_gwdo.F in physics_mmm, 2) load bl_gwdo.o from physics_mmm directory to the WRF library: LIBWRFLIB.
added dependecy of module_bl_gwod.o to depend on phys/physics_mmm/bl_gwdo.o, and bl_gwdo.o on ccpp_kind_types.F
Modify arch/preamble and postamble to accommodate compilation for .F90 files.
Modify clean to remove compiled files in phys/physics_mmm/.
Modify wrftladj/module_pbl_driver_ad.F to add error variables and dx2dtmp.

Solution: The CCPP-compliant GWDO code will be residing in phys/physics_mmm/ directory. Currently it is a copy of the code from the future MMM-physics repo.

LIST OF MODIFIED FILES: M arch/postamble M arch/preamble M clean M main/depend.common M phys/Makefile M phys/module_bl_gwdo.F M phys/module_pbl_driver.F A phys/physics_mmm/bl_gwdo.F90 M wrftladj/module_pbl_driver_ad.F

TESTS CONDUCTED:

Option run before and after the change produced bit-for-bit results when compiled with -d option.
The Jenkins tests are all passing.

RELEASE NOTE: Add gwdo (gwd_opt=1) code from the future shared MMM-physics repository. Future change to this option should be made to the MMM-physics repository (to be available soon).

weiwangncar commented 8 months ago

The compilation for DA failed. Will look into it next week.

weiwangncar commented 8 months ago

With the last change, the regression test results are:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None

weiwangncar commented 8 months ago

@liujake Since I modified one file, module_pbl_driver_ad.F, in wrftladj/ directory, can you take a look and see if it is ok? It is the similar change I made for module_pbl_driver.F. Thanks.

weiwangncar commented 8 months ago

The compilation for DA failed. Will look into it next week.

The issue has been resolved.

liujake commented 8 months ago

The change looks good to me.

MicroTed commented 7 months ago

@weiwangncar One issue here is that it breaks things again on a case-preserving but case-insensitive file system (i.e., Macos default filesystem). It happily clobbers the .F90 into the processed .f90 (but still named .F90). Is it possible to change the names in physics_mmm to .F to be consistent with everything else?

weiwangncar commented 7 months ago

@MicroTed This was a decision the CCPP group made. Let me ask around and see if there is a solution to this. Could that system be configured so that it is case sensitive?

weiwangncar commented 7 months ago

@duda @islas @dudhia Any comments?

dudhia commented 7 months ago

calling in @dustinswales of the CCPP group

dustinswales commented 7 months ago

The case does matter. Uppercase file extensions enable the use of pre-processor directives, lowercase does not.

MicroTed commented 7 months ago

In CCPP, the .F90 files are compiled directly by the fortran compiler without the explicitly preprocessed intermediary files. So perhaps the compile process for the physics_mmm codes could be treated that way, too? All the preprocessing does is remove some comment lines

MicroTed commented 7 months ago

@dustinswales Maybe to clarify, the WRF build system is taking, e.g., mp_wsm6.F90 and preprocessing it explicitly with sed and cpp into mp_wsm6.f90 and then compiling that. The problem on the default mac filesystem is that mp_wsm6.F90 is now replaced by the processed version. (i.e., still has the name mp_wsm6.F90 but is actually the processed mp_wsm6.90) It still compiles because the only change is the removal of some commented lines.

sed -e "s/^\!.*'.*//" -e "s/^ *\!.*'.*//" physics_mmm/mp_wsm6.F90 > physics_mmm/mp_wsm6.G
cpp -P -nostdinc -xassembler-with-cpp -I/Users/ted.mansell/develop/wrfgit/WRFV4git/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DDA_CORE=0 -DWRFPLUS=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 -DMACOS -DNONSTANDARD_SYSTEM_FUNC  -DWRF_USE_CLM -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT  -DDM_PARALLEL -DNETCDF -DLANDREAD_STUB=1 -DUSE_ALLOCATABLES -Dwrfmodel -DGRIB1 -DINTIO -DKEEP_INT_AROUND -DLIMIT_ARGS -DBUILD_RRTMG_FAST=0 -DBUILD_RRTMK=0 -DBUILD_SBM_FAST=1 -DSHOW_ALL_VARS_USED=0 -DCONFIG_BUF_LEN=65536 -DMAX_DOMAINS_F=21 -DMAX_HISTORY=25 -DNMM_NEST=0  -I. -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT   physics_mmm/mp_wsm6.G  > physics_mmm/mp_wsm6.bb
/Users/ted.mansell/develop/wrfgit/WRFV4git/tools/standard.exe physics_mmm/mp_wsm6.bb | cpp -P -nostdinc -xassembler-with-cpp -traditional-cpp -DUSE_NETCDF4_FEATURES -DWRFIO_NCD_LARGE_FILE_SUPPORT > physics_mmm/mp_wsm6.f90
rm -f physics_mmm/mp_wsm6.G physics_mmm/mp_wsm6.bb
time mpif90 -f90=ifort -o physics_mmm/mp_wsm6.o -c -O3 -fp-model precise -w -ftz -align all -fno-alias -fno-common -FR -convert big_endian    -I../dyn_em  -I/Users/ted.mansell/develop/wrfgit/WRFV4git/external/esmf_time_f90  -I/Users/ted.mansell/develop/wrfgit/WRFV4git/main -I/Users/ted.mansell/develop/wrfgit/WRFV4git/external/io_netcdf -I/Users/ted.mansell/develop/wrfgit/WRFV4git/external/io_int -I/Users/ted.mansell/develop/wrfgit/WRFV4git/frame -I/Users/ted.mansell/develop/wrfgit/WRFV4git/share -I/Users/ted.mansell/develop/wrfgit/WRFV4git/phys -I/Users/ted.mansell/develop/wrfgit/WRFV4git/wrftladj -I/Users/ted.mansell/develop/wrfgit/WRFV4git/chem -I/Users/ted.mansell/develop/wrfgit/WRFV4git/inc -I/Users/Shared/opt/local/netcdf460ser/include  -real-size `expr 8 \* 4` -i4  physics_mmm/mp_wsm6.f90

dudhia commented 7 months ago

If these are our only F90 files, we just need to adapt our suffix rules?

dustinswales commented 7 months ago

The suffix convention in the CCPP is the same as the fortran standard. That is, F90/.f90 -> free form files .F/.f -> fixed-form source files. My suggestions would be for WRF to adopt using this convention, not the other way around.

MicroTed commented 7 months ago

I tested adding a new rule to phys/Makefile:

.F90.o:
    $(RM) $@
    $(FC) -o $@ -c $(FCFLAGS) $(OMP) $(MODULE_DIRS) $(PROMOTION) $(FCSUFFIX) $*.F90

And that seems to override the rule in configure.wrf

mgduda commented 7 months ago

@duda @islas @dudhia Any comments?

Just FYI, I'm @mgduda!

mgduda commented 7 months ago

I would agree that we should adapt the rule for compiling .F90 files in the physics_mmm directory to let the compiler handle preprocessing, which is what I expect would happen in most other host models.

dudhia commented 7 months ago

@mgduda could we safely just modify it in configure.wrf since there are no other .F90 files and I think it is a new suffix rule there?

mgduda commented 7 months ago

The suffix convention in the CCPP is the same as the fortran standard. That is, F90/.f90 -> free form files .F/.f -> fixed-form source files. My suggestions would be for WRF to adopt using this convention, not the other way around.

Side question: does the Fortran standard actually say anything about file suffixes? I thought that it was simply convention among compilers to assume that .F90/.f90 were free form and .F/.f were fixed-form, and capitalization of the suffix implied the need for preprocessing.

mgduda commented 7 months ago

@mgduda could we safely just modify it in configure.wrf since there are no other .F90 files and I think it is a new suffix rule there?

I think that may be possible. It would be nice if we had a system with a case-insensitive, case-preserving filesystem for testing to make sure there isn't anything that might cause the .f90 files to be compiled into .o files a second time. For now, if we can do so, it may be safer to restrict the scope of the .F90.o suffix rule to just the physics_mmm directory.

dustinswales commented 7 months ago

The suffix convention in the CCPP is the same as the fortran standard. That is, F90/.f90 -> free form files .F/.f -> fixed-form source files. My suggestions would be for WRF to adopt using this convention, not the other way around.

Side question: does the Fortran standard actually say anything about file suffixes? I thought that it was simply convention among compilers to assume that .F90/.f90 were free form and .F/.f were fixed-form, and capitalization of the suffix implied the need for preprocessing.

Not sure if it's just convention or written into the standard

dudhia commented 7 months ago

It was a bit hard to find but it is in arch/postamble where we now have rules for .F90.o and .F90.f90 that get appended in the configure.wrf.

dudhia commented 7 months ago

physics_mmm doesn't have its own Makefile. Should we add one?

MicroTed commented 7 months ago

If the .F90.o and .F90.f90 rules were added just for the physics_mmm, then simply removing the .F90.f90 and replacing .F90.o with something like the rule above (without cpp/sed) ought to work? (And throw in $(ARCHFLAGS) for good measure)

wrf-model / WRF

Add shared physics gwdo #1989