search

DOI-USGS / COAWST

COAWST modeling system git repository
Other
103 stars 48 forks source link

WRF 4D variables saved incorrectly. #296

Open antonioricchi opened 1 month ago

antonioricchi commented 1 month ago

Hi John, I ran the runs with version 3.8 of COAWST. The WRf, WRf+ROMS and WRF+ROMS+SWAN runs ran perfectly and the ground output is consistent. Unfortunately, the output of WRF, of the 4D variables is saved incorrectly. in the picture I add the plt (raw plot...with ncview) of P. As you can see the ranges are very strange, how close I get, the image shows bands. I tried on two servers. Same error. I compiled all the packages with gfortran (don't know if this is a useful infomration), and used the frlags you suggested in May. Do you have any idea if this is a problem with this version? If I can do something about it? thanks for your help

restrictedrage fullrange
jcwarner-usgs commented 1 month ago

Hello Antonio! is this ncview of wrfout or of roms_his file? -j

antonioricchi commented 1 month ago

Hi John! This is ncview of a wrfout, P variable (but the problem is there for all '3d' variables). Other versions of wrf standalone (no coawst), with this grid, and this configuration, work fine.

jcwarner-usgs commented 1 month ago

the only thing i can think of is the quilting. are you using quilting? what if you turn that off?

antonioricchi commented 1 month ago

Dear John Sorry the really stupid question, but what do you mean by “quilting”?

antonioricchi commented 1 month ago

Here is my namelist, I think quilting is off.
Do you see anything else that might cause problems? regards

&time_control run_days = 0, run_hours = 84, run_minutes = 0, run_seconds = 0, start_year = 2020, 2000, 2000, start_month = 09, 01, 01, start_day = 16, 24, 24, start_hour = 00, 12, 12, start_minute = 00, 00, 00, start_second = 00, 00, 00, end_year = 2020, 2000, 2000, end_month = 09, 01, 01, end_day = 19, 25, 25, end_hour = 12, 12, 12, end_minute = 00, 00, 00, end_second = 00, 00, 00, interval_seconds = 21600 input_from_file = .true.,.false.,.false., history_outname = '/data/hpc/CPL/IANOS/OOST55/AOW_OOST55_wrfoutd', history_interval = 15, 60, 60, rst_outname = '/data/hpc/CPL/IANOS/OOST55/AOW_OOST55_wrfrstd', frames_per_outfile = 1, 1000, 1000, restart = .false., restart_interval = 720, io_form_history = 2 io_form_restart = 2 io_form_input = 2 io_form_boundary = 2 io_form_auxinput4 = 2 debug_level = 0 auxinput4_inname = "wrflowinp_d" auxinput4_interval = 360, 360, 360, force_use_old_data = .true. / &domains time_step = 5, time_step_fract_num = 0, time_step_fract_den = 1, max_dom = 1, s_we = 1, 1, 1, e_we = 600, 112, 94, s_sn = 1, 1, 1, e_sn = 720, 97, 91, s_vert = 1, 1, 1, e_vert = 70, 28, 28, num_metgrid_levels = 20 dx = 1800, 10000, 3333, dy = 1800, 10000, 3333, grid_id = 1, 2, 3, parent_id = 0, 1, 2, i_parent_start = 0, 31, 30, j_parent_start = 0, 17, 30, parent_grid_ratio = 1, 3, 3, parent_time_step_ratio = 1, 3, 3, feedback = 1, smooth_option = 0 nproc_x = 18 nproc_y = 10 /

&physics cu_physics = 0, cudt = 5, kfeta_trigger = 1, kf_edrates = 0, mp_physics = 16, mp_zero_out = 0, no_mp_heating = 0, do_radar_ref = 1, shcu_physics = 0, bl_pbl_physics = 5, bldt = 0, grav_settling = 0, topo_wind = 0, ysu_topdown_pblmix = 0, scalar_pblmix = 1, tracer_pblmix = 1, sf_sfclay_physics = 5, !2 usato in altri case isfflx = 1, iz0tlnd = 1, isftcflx = 0, sf_surface_physics = 2, num_land_cat = 21, num_soil_cat = 16, num_soil_layers = 4, surface_input_source = 3, usemonalb = T, rdmaxalb = T, rdlai2d = T, ua_phys = T, sf_surface_mosaic = 0, opt_thcnd = 1, tmn_update = 0, sf_urban_physics = 0, ra_lw_physics = 4, ra_sw_physics = 4, radt = 2, swint_opt = 1, ra_call_offset = 0, swrad_scat = 1, slope_rad = 0, topo_shading = 0, cu_rad_feedback = T, ra_sw_eclipse = 0, icloud = 1, insert_init_cloud = F, sf_ocean_physics = 0, sst_skin = 0, sst_update = 1, fractional_seaice = 0, seaice_albedo_opt = 0, seaice_albedo_default = 0.65, seaice_snowdepth_opt = 0, seaice_snowdepth_min = 0.001, seaice_snowdepth_max = 10., seaice_thickness_opt = 0, seaice_thickness_default = 3.0, sf_lake_physics = 0, lightning_option = 0, hailcast_opt = 0, /

&dynamics non_hydrostatic = T, hybrid_opt = 2, etac = 0.20, gwd_opt = 0, rk_ord = 3, h_mom_adv_order = 5, h_sca_adv_order = 5, v_mom_adv_order = 3, v_sca_adv_order = 3, moist_adv_opt = 1, scalar_adv_opt = 1, momentum_adv_opt = 1, tke_adv_opt = 1, diff_opt = 1, km_opt = 4, w_damping = 1, diff_6th_opt = 2, diff_6th_factor = 0.12, diff_6th_slopeopt = 1, diff_6th_thresh = 0.10, zadvect_implicit = 0, w_crit_cfl = 1.2, phi_adv_z = 1, moist_mix6_off = F, scalar_mix6_off = F, tke_mix6_off = F, damp_opt = 3, zdamp = 5000., dampcoef = 0.3, !0.2 time_step_sound = 0, epssm = 0.6, use_theta_m = 1, use_q_diabatic = 0, /

&bdy_control spec_bdy_width = 5, spec_zone = 1, relax_zone = 4, specified = .true., .false.,.false., nested = .false., .true., .true., /

&grib2 /

&namelist_quilt nio_tasks_per_group = 0, nio_groups = 0, /

jcwarner-usgs commented 1 month ago

for Sandy i had

&namelist_quilt nio_tasks_per_group = 0, nio_groups = 1, /

not sure if that will help.

antonioricchi commented 1 month ago

unfortunately does not solve it. I have observe that this problem only exists on the vertical level 0

fs-jbzambon commented 1 month ago

Throwing myself in here because we've tried quilting in the past with COAWST-WRF and were not successful.

See this paper on WRF I/O performance which defines "Quilt Servers: A third technique for writes that uses I/O (or quilt) servers that deal exclusively with I/O, enabling the compute PEs to continue with their work without waiting for data to be written to disk before proceeding."

Basically with nio_tasks_per_group >0, you're allocating a set of processors to handle just the I/O, leaving the other processors to handle the actual model computation. IIRC, COAWST cannot handle the additional processor allocation. John and I messed around with this for a while before I settled on using io_history_form=102 and using the joiner code to stitch the individual tiles' output together as a post-processing step.

If you're not running a time-sensitive case, e.g. a forecast, then I wouldn't worry about any of this and like John said, use 

&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/

Hope that helps!

antonioricchi commented 4 weeks ago

Dear John e Joe, thanks for the suggestions. I have carried out some experiments. 1) I ran the same simulation with WRF standalone in COAWST. Same problem 2) I ran a simulation of another case study, WRF standalone in COAWST, and I get the same problem. 3) I ran the simulation on another server, in standard WRF, and the result is ok, and in standalone WRF COAWST, and the result is wrong.

The namelist is the same. I use
&namelist_quilt nio_tasks_per_group = 0, nio_groups = 1, /

At this point, I suppose I have some problem with the WRF compilation. This is the first time I have had this problem, even though I have run dozens of simulations with COAWST. It's the first time I've used it compiled with gfortran (I can't compile it with intel on this cluster). How can I proceed?

jcwarner-usgs commented 4 weeks ago

can u try the Sandy case we distribute, use WRF + ROMS, and see if that has the problem.

jcwarner-usgs commented 4 weeks ago

Is this problem with WRF by itself? Then i might need to update WRF. It would be a problem in that model. What version of WRF did you try and it worked?

antonioricchi commented 3 weeks ago

Hi John

1) Sandy : it works perfectly, as do all test cases. This is really strange indeed. 2) The problem is with any approach I use. WRF, WRF+ROMS etc, using my grids and my WRF case. I am using the ‘same grid’ approach, but I guess it doesn't depend on that, but on something in WRF. I have tried numerous test cases, all very (with many grid points), but the result is always very bad. Am I doing something wrong, or is there something going on in COAWST with very large grids.

I also ran Sandy, with many cores, and it works perfectly.

I enclose the script I use to compile, which William Hataway sent me and which I modified to use on my cluster.

In any case, even some indication as to why this error might occur would be very helpful.

PS, the WRF files are not generated with cowast's WPS, but with external wps. Could this be a critical issue?

!/bin/bash

start=$(date) START=$(date +"%s")

/home/ricchi/setup1.sh

############################### Version Numbers ##########################

For Ease of updating

########################################################################## export METPLUS_Version=5.1.0 export MET_Version=11.1.0 export MET_VERSION=11.1 export METPLUS_DATA=5.1

export HDF5_Version=1.14.4 export HDF5_Sub_Version=3 export Zlib_Version=1.3.1 export Netcdf_C_Version=4.9.2 export Netcdf_Fortran_Version=4.6.1 export Netcdf_CXX_Version=4.3.1 export Mpich_Version=4.2.1 export Libpng_Version=1.6.39 export Jasper_Version=1.900.1

export WRF_VERSION=4.6.0 export WPS_VERSION=4.5

echo $PASSWD | sudo -S apt -y update

echo $PASSWD | sudo -S apt -y upgrade && sudo -S apt -y autoremove

echo $PASSWD | sudo -S apt -y install autoconf automake bison build-essential byacc cmake csh curl default-jdk default-jre emacs flex g++ gawk gcc gfortran git ksh libcurl4-openssl-dev libjpeg-dev libncurses5 libncurses6 libpixman-1-dev libpng-dev libtool libxml2 libxml2-dev m4 make mlocate ncview okular openbox pipenv pkg-config python3 python3-dev python3-pip tcsh unzip xauth xorg time

module load pkgconf/1.9.5-broadwell-gcc_12.2.0 module load libiconv/1.17-broadwell-gcc_12.2.0 module load xz/5.4.1-broadwell-gcc_12.2.0 module load zlib-ng/2.1.4-broadwell-gcc_12.2.0 module load libxml2/2.10.3-broadwell-gcc_12.2.0 module load berkeley-db/18.1.40-broadwell-gcc_12.2.0 module load bzip2/1.0.8-broadwell-gcc_12.2.0 module load ncurses/6.4-broadwell-gcc_12.2.0 module load readline/8.2-broadwell-gcc_12.2.0 module load gdbm/1.23-broadwell-gcc_12.2.0 module load perl/5.38.0-broadwell-gcc_12.2.0 module load zstd/1.5.5-broadwell-gcc_12.2.0 module load openjpeg/2.3.1-broadwell-gcc_12.2.0 module load eccodes/2.25.0-broadwell-gcc_12.2.0 module load libjpeg-turbo/3.0.0-broadwell-intel_2021.9.0

    echo " "
    ##############################Directory Listing############################
    export HOME=$(
            cd
            pwd
    )
    mkdir $HOME/COAWST
    export COAWST_FOLDER=$HOME/COAWST
    cd "${COAWST_FOLDER}"
    mkdir Downloads
    mkdir WRFPLUS
    mkdir WRFDA
    mkdir Libs
    export DIR="${COAWST_FOLDER}"/Libs
    mkdir Libs/grib2
    mkdir Libs/NETCDF
    mkdir Libs/MPICH
    mkdir -p Tests/Environment
    mkdir -p Tests/Compatibility

    echo " "
    #############################Core Management####################################

    export CPU_CORE=$(nproc) # number of available threads on system
    export CPU_6CORE="6"
    export CPU_HALF=$(($CPU_CORE / 2))                    #half of availble cores on system
    export CPU_HALF_EVEN=$(($CPU_HALF - ($CPU_HALF % 2))) #Forces CPU cores to even number to avoid partial core export. ie 7 cores would be 3.5 cores.

    if [ $CPU_CORE -le $CPU_6CORE ]; then #If statement for low core systems.  Forces computers to only use 1 core if there are 4 cores or less on the system.
            export CPU_HALF_EVEN="2"
    else
            export CPU_HALF_EVEN=$(($CPU_HALF - ($CPU_HALF % 2)))
    fi

    echo "##########################################"
    echo "Number of Threads being used $CPU_HALF_EVEN"
    echo "##########################################"

    echo " "
    ##############################Downloading Libraries############################
    #Force use of ipv4 with -4
    cd Downloads
    wget -c https://github.com/madler/zlib/archive/refs/tags/v$Zlib_Version.tar.gz
    wget -c https://github.com/HDFGroup/hdf5/releases/download/hdf5_$HDF5_Version.$HDF5_Sub_Version/hdf5-$HDF5_Version-$HDF5_Sub_Version.tar.gz
    wget -c https://github.com/Unidata/netcdf-c/archive/refs/tags/v$Netcdf_C_Version.tar.gz
    wget -c https://github.com/Unidata/netcdf-fortran/archive/refs/tags/v$Netcdf_Fortran_Version.tar.gz
    wget -c https://download.sourceforge.net/libpng/libpng-$Libpng_Version.tar.gz
    wget -c https://www.ece.uvic.ca/~frodo/jasper/software/jasper-$Jasper_Version.zip
    wget -c https://github.com/pmodels/mpich/releases/download/v$Mpich_Version/mpich-$Mpich_Version.tar.gz
    wget -c https://github.com/Unidata/netcdf-cxx4/archive/refs/tags/v$Netcdf_CXX_Version.tar.gz

    echo " "
    ####################################Compilers#####################################
    export CC=gcc
    export CXX=g++
    export FC=gfortran
    export F77=gfortran
    export CFLAGS="-fPIC -fPIE -O3"

    #IF statement for GNU compiler issue
    export GCC_VERSION=$(/usr/bin/gcc -dumpfullversion | awk '{print$1}')
    export GFORTRAN_VERSION=$(/usr/bin/gfortran -dumpfullversion | awk '{print$1}')
    export GPLUSPLUS_VERSION=$(/usr/bin/g++ -dumpfullversion | awk '{print$1}')

    export GCC_VERSION_MAJOR_VERSION=$(echo $GCC_VERSION | awk -F. '{print $1}')
    export GFORTRAN_VERSION_MAJOR_VERSION=$(echo $GFORTRAN_VERSION | awk -F. '{print $1}')
    export GPLUSPLUS_VERSION_MAJOR_VERSION=$(echo $GPLUSPLUS_VERSION | awk -F. '{print $1}')

    export version_10="1230"

    if [ $GCC_VERSION_MAJOR_VERSION -ge $version_10 ] || [ $GFORTRAN_VERSION_MAJOR_VERSION -ge $version_10 ] || [ $GPLUSPLUS_VERSION_MAJOR_VERSION -ge $version_10 ]; then
            export fallow_argument=-fallow-argument-mismatch
            export boz_argument=-fallow-invalid-boz
    else
            export fallow_argument=
            export boz_argument=
    fi

    export FFLAGS="$fallow_argument -m64"
    export FCFLAGS="$fallow_argument -m64"

    echo "##########################################"
    echo "FFLAGS = $FFLAGS"
    echo "FCFLAGS = $FCFLAGS"
    echo "CFLAGS = $CFLAGS"
    echo "##########################################"

    echo " "
    #############################zlib############################
    #Uncalling compilers due to comfigure issue with zlib$Zlib_Version
    #With CC & CXX definied ./configure uses different compiler Flags

    cd "${COAWST_FOLDER}"/Downloads
    tar -xvzf v$Zlib_Version.tar.gz
    cd zlib-$Zlib_Version/
    autoreconf -i -f 2>&1 | tee autoreconf.log
    ./configure --prefix=$DIR/grib2 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check

    echo " "
    ##############################MPICH############################
    #F90= due to compiler issues with mpich install
    cd "${COAWST_FOLDER}"/Downloads
    tar -xvzf mpich-$Mpich_Version.tar.gz
    cd mpich-$Mpich_Version/
    autoreconf -i -f 2>&1 | tee autoreconf.log

    F90= ./configure --prefix=$DIR/MPICH --with-device=ch3 FFLAGS=$fallow_argument FCFLAGS=$fallow_argument 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    # make check

    export PATH=$DIR/MPICH/bin:$PATH

    export MPIFC=$DIR/MPICH/bin/mpifort
    export MPIF77=$DIR/MPICH/bin/mpifort
    export MPIF90=$DIR/MPICH/bin/mpifort
    export MPICC=$DIR/MPICH/bin/mpicc
    export MPICXX=$DIR/MPICH/bin/mpicxx

    echo " "
    #############################libpng############################
    cd "${COAWST_FOLDER}"/Downloads
    export LDFLAGS=-L$DIR/grib2/lib
    export CPPFLAGS=-I$DIR/grib2/include
    tar -xvzf libpng-$Libpng_Version.tar.gz
    cd libpng-$Libpng_Version/
    autoreconf -i -f 2>&1 | tee autoreconf.log

    CC=$MPICC FC=$MPIFC F77=$MPIF77 F90=$MPIF90 CXX=$MPICXX CFLAGS=$CFLAGS ./configure --prefix=$DIR/grib2 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check
    echo " "
    #############################JasPer############################
    cd "${COAWST_FOLDER}"/Downloads
    unzip jasper-$Jasper_Version.zip
    cd jasper-$Jasper_Version/
    autoreconf -i -f 2>&1 | tee autoreconf.log

    ./configure --prefix=$DIR/grib2
    CC=$MPICC FC=$MPIFC F77=$MPIF77 F90=$MPIF90 CXX=$MPICXX CFLAGS=$CFLAGS ./configure --prefix=$DIR/grib2 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check

    export JASPERLIB=$DIR/grib2/lib
    export JASPERINC=$DIR/grib2/include

    echo " "
    #############################hdf5 library for netcdf4 functionality############################
    cd "${COAWST_FOLDER}"/Downloads
    tar -xvzf hdf5-$HDF5_Version-$HDF5_Sub_Version.tar.gz
    cd hdf5-$HDF5_Version-$HDF5_Sub_Version
    autoreconf -i -f 2>&1 | tee autoreconf.log

    CC=$MPICC FC=$MPIFC F77=$MPIF77 F90=$MPIF90 CXX=$MPICXX CFLAGS=$CFLAGS ./configure --prefix=$DIR/grib2 --with-zlib=$DIR/grib2 --enable-hl --enable-fortran --enable-parallel 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check

    export HDF5=$DIR/grib2
    export HDF5_INCDIR=$DIR/grib2/include
    export HDF5_LIBDIR=$DIR/grib2/lib
    export PHDF5=$DIR/grib2
    export PHDF5_INCDIR=$DIR/grib2/include
    export PHDF5_LIBDIR=$DIR/grib2/lib
    export LD_LIBRARY_PATH=$DIR/grib2/lib:$LD_LIBRARY_PATH

    echo " "

    ##############################Install NETCDF C Library############################
    cd "${COAWST_FOLDER}"/Downloads
    tar -xzvf v$Netcdf_C_Version.tar.gz
    cd netcdf-c-$Netcdf_C_Version/
    export CPPFLAGS=-I$DIR/grib2/include
    export LDFLAGS=-L$DIR/grib2/lib
    export LIBS="-lhdf5_hl -lhdf5 -lz -lcurl -lgfortran -lgcc -lm -ldl "
    autoreconf -i -f 2>&1 | tee autoreconf.log

    CC=$MPICC FC=$MPIFC CXX=$MPICXX F90=$MPIF90 F77=$MPIF77 CFLAGS=$CFLAGS ./configure --prefix=$DIR/NETCDF --disable-dap --enable-netcdf-4 --enable-netcdf4 --enable-shared --enable-static --enable-cdf5 --enable-parallel-tests 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check

    export PATH=$DIR/NETCDF/bin:$PATH
    export NETCDF=$DIR/NETCDF
    export NETCDF_INC=$NETCDF/include
    export NETCDF_LIB=$NETCDF/lib
    export NETCDF_INCDIR=$NETCDF/include
    export NETCDF_LIBDIR=$NETCDF/lib
    export NETCDFPATH=$NETCDF
    echo " "
    ##############################NetCDF fortran library############################
    cd "${COAWST_FOLDER}"/Downloads
    tar -xvzf v$Netcdf_Fortran_Version.tar.gz
    cd netcdf-fortran-$Netcdf_Fortran_Version/
    export LD_LIBRARY_PATH=$DIR/NETCDF/lib:$LD_LIBRARY_PATH
    export CPPFLAGS="-I$DIR/NETCDF/include -I$DIR/grib2/include"
    export LDFLAGS="-L$DIR/NETCDF/lib -L$DIR/grib2/lib"
    export LIBS="-lnetcdf  -lcurl -lhdf5_hl -lhdf5 -lz -lm -ldl -lgcc -lgfortran"
    autoreconf -i -f 2>&1 | tee autoreconf.log

    CC=$MPICC FC=$MPIFC CXX=$MPICXX F90=$MPIF90 F77=$MPIF77 CFLAGS=$CFLAGS ./configure --prefix=$DIR/NETCDF --enable-netcdf-4 --enable-netcdf4 --enable-shared --enable-static --enable-parallel-tests --enable-hdf5 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check

    echo " "

    ############################NetCDF CXX library############################
    cd "${COAWST_FOLDER}"/Downloads
    tar -xvzf v$Netcdf_CXX_Version.tar.gz
    cd netcdf-cxx4-$Netcdf_CXX_Version/
    export LD_LIBRARY_PATH=$DIR/NETCDF/lib:$LD_LIBRARY_PATH
    export CPPFLAGS="-I$DIR/NETCDF/include -I$DIR/grib2/include"
    export LDFLAGS="-L$DIR/NETCDF/lib -L$DIR/grib2/lib"
    export LIBS="-lnetcdff -lnetcdf  -lcurl -lhdf5_hl -lhdf5 -lz -lm -ldl -lgcc -lgfortran"
    autoreconf -i -f 2>&1 | tee autoreconf.log

    CC=$MPICC FC=$MPIFC CXX=$MPICXX F90=$MPIF90 F77=$MPIF77 CFLAGS=$CFLAGS ./configure --prefix=$DIR/NETCDF --enable-netcdf-4 --enable-netcdf4 --enable-shared --enable-static --enable-parallel-tests --enable-hdf5 2>&1 | tee configure.log
    automake -a -f 2>&1 | tee automake.log
    make -j $CPU_HALF_EVEN 2>&1 | tee make.log
    make -j $CPU_HALF_EVEN install 2>&1 | tee make.install.log
    #make check

    echo " "

    #################################### System Environment Tests ##############

    cd "${COAWST_FOLDER}"/Downloads
    wget -c https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_NETCDF_MPI_tests.tar
    wget -c https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_tests.tar

    tar -xvf Fortran_C_tests.tar -C "${COAWST_FOLDER}"/Tests/Environment
    tar -xvf Fortran_C_NETCDF_MPI_tests.tar -C "${COAWST_FOLDER}"/Tests/Compatibility

    export one="1"
    echo " "
    ############## Testing Environment #####

    cd "${COAWST_FOLDER}"/Tests/Environment

    cp ${NETCDF}/include/netcdf.inc .
    echo " "
    echo " "
    echo "Environment Testing "
    echo "Test 1"
    $FC TEST_1_fortran_only_fixed.f
    ./a.out | tee env_test1.txt
    export TEST_PASS=$(grep -w -o -c "SUCCESS" env_test1.txt | awk '{print$1}')
    if [ $TEST_PASS -ge 1 ]; then
            echo "Enviroment Test 1 Passed"
    else
            echo "Environment Compiler Test 1 Failed"
            exit
    fi
    read -r -t 3 -p "I am going to wait for 3 seconds only ..."

    echo " "
    echo "Test 2"
    $FC TEST_2_fortran_only_free.f90
    ./a.out | tee env_test2.txt
    export TEST_PASS=$(grep -w -o -c "SUCCESS" env_test2.txt | awk '{print$1}')
    if [ $TEST_PASS -ge 1 ]; then
            echo "Enviroment Test 2 Passed"
    else
            echo "Environment Compiler Test 2 Failed"
            exit
    fi
    echo " "
    read -r -t 3 -p "I am going to wait for 3 seconds only ..."

    echo " "
    echo "Test 3"
    $CC TEST_3_c_only.c
    ./a.out | tee env_test3.txt
    export TEST_PASS=$(grep -w -o -c "SUCCESS" env_test3.txt | awk '{print$1}')
    if [ $TEST_PASS -ge 1 ]; then
            echo "Enviroment Test 3 Passed"
    else
            echo "Environment Compiler Test 3 Failed"
            exit
    fi
    echo " "
    read -r -t 3 -p "I am going to wait for 3 seconds only ..."

    echo " "
    echo "Test 4"
    $CC -c -m64 TEST_4_fortran+c_c.c
    $FC -c -m64 TEST_4_fortran+c_f.f90
    $FC -m64 TEST_4_fortran+c_f.o TEST_4_fortran+c_c.o
    ./a.out | tee env_test4.txt
    export TEST_PASS=$(grep -w -o -c "SUCCESS" env_test4.txt | awk '{print$1}')
    if [ $TEST_PASS -ge 1 ]; then

           echo "Enviroment Test 4 Passed"
    else
            echo "Environment Compiler Test 4 Failed"
            exit
    fi
    echo " "
    read -r -t 3 -p "I am going to wait for 3 seconds only ..."

    echo " "
    ############## Testing Environment #####

    cd "${COAWST_FOLDER}"/Tests/Compatibility

    cp ${NETCDF}/include/netcdf.inc .

    echo " "
    echo " "
    echo "Library Compatibility Tests "
    echo "Test 1"
    $FC -c 01_fortran+c+netcdf_f.f
    $CC -c 01_fortran+c+netcdf_c.c
    $FC 01_fortran+c+netcdf_f.o 01_fortran+c+netcdf_c.o \
            -L${NETCDF}/lib -lnetcdff -lnetcdf

    ./a.out | tee comp_test1.txt
    export TEST_PASS=$(grep -w -o -c "SUCCESS" comp_test1.txt | awk '{print$1}')
    if [ $TEST_PASS -ge 1 ]; then
            echo "Compatibility Test 1 Passed"
    else
            echo "Compatibility Compiler Test 1 Failed"
            exit
    fi
    echo " "
    read -r -t 3 -p "I am going to wait for 3 seconds only ..."

    echo " "

    echo "Test 2"
    $MPIFC -c 02_fortran+c+netcdf+mpi_f.f
    $MPICC -c 02_fortran+c+netcdf+mpi_c.c
    $MPIFC 02_fortran+c+netcdf+mpi_f.o \
            02_fortran+c+netcdf+mpi_c.o \
            -L${NETCDF}/lib -lnetcdff -lnetcdf

    $DIR/MPICH/bin/mpirun ./a.out | tee comp_test2.txt
    export TEST_PASS=$(grep -w -o -c "SUCCESS" comp_test2.txt | awk '{print$1}')
    if [ $TEST_PASS -ge 1 ]; then
            echo "Compatibility Test 2 Passed"
    else
            echo "Compatibility Compiler Test 2 Failed"
            exit
    fi
    echo " "
    read -r -t 3 -p "I am going to wait for 3 seconds only ..."
    echo " "

    echo " All tests completed and passed"
    echo " "
    ########################COAWST 3.8###################################
    cd "${COAWST_FOLDER}"
    git clone https://github.com/DOI-USGS/COAWST.git

    # Compile Model Coupling Toolkit (MCT)
    cd "${COAWST_FOLDER}"/COAWST/Lib/MCT
    MPIHEADER="-I$DIR/MPICH/include" ./configure --prefix="${COAWST_FOLDER}"/Libs/MCT  2>&1 | tee coawst.configure.log

    make 2>&1 | tee coawst.make.log
    make install 2>&1 | tee coawst.make.install.log

    export MCT_INCDIR="${COAWST_FOLDER}"/Libs/MCT/include
    export MCT_LIBDIR="${COAWST_FOLDER}"/Libs/MCT/lib

    cd "${COAWST_FOLDER}"/COAWST

    #modifying build_coawst to for paths
    sed -i '133s/INLET_TEST/SANDY/g' build_coawst.sh
    sed -i '141s|/cygdrive/e/data/models/COAWST|"${COAWST_FOLDER}"/COAWST/ |g' build_coawst.sh
    sed -i '209s|export|#export|g' build_coawst.sh
    sed -i '210s|#export|export|g' build_coawst.sh
    sed -i '251s|#export|export|g' build_coawst.sh
    sed -i '305s|Inlet_test/Coupled|Sandy|g' build_coawst.sh
    sed -i '306s|Inlet_test/Coupled|Sandy|g' build_coawst.sh

    cd "${COAWST_FOLDER}"/COAWST/Compilers
    sed -i '158s|-assume byterecl||g' Linux-gfortran.mk

    #this removes the arch prompt for gnu compilers
    cd "${COAWST_FOLDER}"/COAWST/WRF/arch/

    sed -i '433s|.*|  $response = "34 \\n";|g' Config.pl
    sed -i '898s|.*|  $response = "1 \\n";|g' Config.pl

    cd "${COAWST_FOLDER}"/COAWST/

    chmod 775 *.sh

    ./build_coawst.sh -j 4 2>&1 | tee build_coawst.log

    if [ $? -eq 0 ] && [ -e "coawstM" ]; then
    echo "Script ran successfully and coawstM exists."
    else
    echo "Script did not run successfully or coawstM does not exist."
    fi

    #cd "${COAWST_FOLDER}"/COAWST
    #ln -sf Projects/Sandy/namelist.input .
    #ln -sf Projects/Sandy/wrf* .
    #cp Projects/Sandy/coupling_sandy.in Projects/Sandy/coupling_sandy.in.old
    #cp Projects/Sandy/ocean_sandy.in Projects/Sandy/ocean_sandy.in.old

    #cd "${COAWST_FOLDER}"/COAWST/

    #mpirun -np 3 ./coawstM Projects/Sandy/coupling_sandy.in 2>&1 | tee Sandy_Coupled_Test.out
jcwarner-usgs commented 3 weeks ago

i doubt it is the compiling method. When you run your WRF tests, are you using the same version of WRF that is in COAWST? It could be the WPS, not sure. Do you want to put your case somewhere as a tar file and i try it? -j

antonioricchi commented 3 weeks ago

Dear John, Joe and others, My WRFinput files, when used in other WRFs, work (compiled using intel for example). I have done many experiments. For example if I use WPS from COAWST, same problem. If I use WPS etc from the same WRF version as COAWST (for example version 4.2.2), same problem. Even if I compile a new WRF (pure WRF installation, no COAWST) of the same version of the WRF as COAWST, obtain same problem. I have "virtually removed" any possibility that it is related to COAWST or grid/preprocessing. The problem lies in the installation technique and/or the unix environment I generated after installing COAWST. I used a attached script to install libraries and dependencies, then extracted the environment so I could recompile and run. This is the environment I load every time I launch a run (I am attaching both the install script and the env source file). I hope you find some suggestions, it is extremely technical for me as a scope. Thanks in advance for the help.

regards

saved_env.txt install_test.txt