search

NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
75 stars 171 forks source link

FIXfv3 not being set up correctly for atmos-only experiment in the latest workflow #863

Closed yangfanglin closed 1 year ago

yangfanglin commented 2 years ago

Issue:
An attempt to run atmos-only experiment using the latest workflow (develop branch) and the model tag p8c revealed that FIXfv3 is not set up correctly for atmos-only runs if FRAC_GRID=.true.. The new VIIRS veg type datasets and new mask/oro datasets are only made available in the fix_fv3_fracoro directory.

In the original exglobal_forecast.sh I have the following settings.

reso=025
[[ ${CASE} = "C192" ]] && reso=050
[[ ${CASE} = "C96" ]]  && reso=100
frac_grid=${frac_grid:-".true."}
if [ $frac_grid = ".false." ]; then
 FIXfv3=$FIX_DIR/fix_fv3_gmted2010/${CASE}
else
 FIXfv3=$FIX_DIR/fix_fv3_fracoro/${CASE}.mx${reso}_frac
fi

I think the latest fix, mask and oro datasets created by Shan/Mike/Helin should work for both fractional and non-fractional grid.  So the default FIXfv3 should be pointing to fix_fv3_fracoro instead of  fix_fv3_gmted2010 for all new UFS applications.   We need to use the new VIIRS veg type,  VIIRS based land-sea-lake mask and the updated oro dataset that was made to be consistent with MOM6, no matter if we are running coupled or atmos-only experiments. 

 fix_fv3_gmted2010 should only be used for running current operational model (GFS.v16)

Helin checked the model RT settings and found out the RTs are using the correct (new) datasets.

yangfanglin commented 2 years ago

From Shan Sun:

​​/scratch1/NCEPDEV/global/glopara/fix_NEW/fix_fv3_fracoro are the latest oro_data for p8 based on the latest 'MODIS+' dataset.

This oro_data, made for fractional grid, should be able to run in the case of non-fractional grid by setting frac_grid=F. Also the fractional version of the sfc_data should work for the nonfrac case as well. I tested it in p6 (and assume the same applies for p8 as well)

When running with frac_grid=F, make sure to set cplmode = nems_orig, since cplmode = nems_frac for frac_grid=T.

yangfanglin commented 2 years ago

Assuming the "develop" branch will never be used to run current ops GFS.v16, I made the following changes

In config/config.fcst export FRAC_GRID=".true." export cplmode="nems_frac"

+if [ $FRAC_GRID = ".false." ]; then
+  export cplmode="nems_orig"
+fi

In /ush/forecast_predet.sh


 -  FIXfv3=${FIXfv3:-$FIX_DIR/fix_fv3_gmted2010}
+
+  #FIXfv3=${FIXfv3:-$FIX_DIR/fix_fv3_gmted2010}
+  reso=025
+  [[ ${CASE} = "C192" ]] && reso=050
+  [[ ${CASE} = "C96" ]]  && reso=100
+  subdir=fix_fv3_fracoro/${CASE}.mx${reso}_frac
+  FIXfv3=${FIXfv3:-$FIX_DIR/$subdir}
+

@ChunxiZhang-NOAA Chunxi, when you are done with the GOCART aerosol PR, could you please check, test and get this update into the workflow before p8 starts to run ? Even for coupled forecast with fractional grid, I cannot find anywhere in the current workflow fix_fv3_fracoro/${CASE}.mx${reso}_frac is defined and used. This is definitely not correct.

@aerorahul Rahul, It would be fine with me if you'd like to have someone else from EIB to test and make the update.

HelinWei-NOAA commented 2 years ago

@yangfanglin The setting is fine for for coupled forecasts in global workflow. @KateFriedman-NOAA pointed this out to me before. In ush/forecast_postdet.sh the following is used to determine which set of fixed fields will be used.

    if [ $FRAC_GRID = ".true." ]; then
            OROFIX=${OROFIX:-"${FIX_DIR}/fix_fv3_fracoro/${CASE}.mx${OCNRES}_frac"}
            FIX_SFC=${FIX_SFC:-"${OROFIX}/fix_sfc"}
            for n in $(seq 1 $ntiles); do
                    $NLN ${OROFIX}/oro_${CASE}.mx${OCNRES}.tile${n}.nc $DATA/INPUT/oro_data.tile${n}.nc
            done
    else
            OROFIX=${OROFIX:-"${FIXfv3}/${CASE}"}
            FIX_SFC=${FIX_SFC:-"${FIXgrd}/fix_sfc"}
            for n in $(seq 1 $ntiles); do
                    $NLN ${OROFIX}/${CASE}_oro_data.tile${n}.nc $DATA/INPUT/oro_data.tile${n}.nc
            done
    fi
yangfanglin commented 2 years ago

@HelinWei-NOAA Thanks. I missed the postdet script.

However, changes are still needed if we agree to use the same oro and fix datasets for both FRAC_GRD=T and FRAC_GRID=F. This is necessary if we want to compare coupled vs atmos-only runs. The new "MODIS+" based oro and masks etc should be used for all UFS global applications from this point forward.

SMoorthi-emc commented 2 years ago

Can someone point where all of these fixed fields are located on WCOSS2 for all resolutions (including C3072)? Until now things have been very fluid and hard to keep up. Thanks Moorthi

On Thu, Jun 16, 2022 at 8:35 PM Fanglin Yang @.***> wrote:

@HelinWei-NOAA https://github.com/HelinWei-NOAA Thanks. I missed the postdet script.

However, changes are still needed if we agree to use the same oro and fix datasets for both FRAC_GRD=T and FRAC_GRID=F. This is necessary if we want to compare coupled vs atmos-only runs. The new "MODIS+" based oro and masks etc should be used for all UFS global applications from this point forward.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/863#issuecomment-1158341144, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLVRYTRRAA7V3JINHVO6C3VPPB6PANCNFSM5Y63DLTA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718

e-mail: @.*** Phone: (301) 683-3718 Fax: (301) 683-3718

ChunxiZhang-NOAA commented 2 years ago

@yangfanglin Sure, I will work on this case after GOCART aerosol PR.

yangfanglin commented 2 years ago

Can someone point where all of these fixed fields are located on WCOSS2 for all resolutions (including C3072)? Until now things have been very fluid and hard to keep up. Thanks Moorthi

I found it on Dogwood at /lfs/h2/emc/global/save/emc.global/FIX/fix_NEW. New datasets for C1152 and C3072 are yet to be created and included in fix_fv3_fracoro

yangfanglin commented 2 years ago

@GeorgeGayno-NOAA George, for consistency UFS_UTIL should also be using the "new" fix_fv3_fracoro datasets instead of fix_fv3_gmted2010 for future UFS-based applications.

SMoorthi-emc commented 2 years ago

Well, I don't see any C1152 or C3072 files there! Moorthi

On Fri, Jun 17, 2022 at 11:03 AM Fanglin Yang @.***> wrote:

Can someone point where all of these fixed fields are located on WCOSS2 for all resolutions (including C3072)? Until now things have been very fluid and hard to keep up. Thanks Moorthi … <#m7432428067391458560> On Thu, Jun 16, 2022 at 8:35 PM Fanglin Yang @.> wrote: @HelinWei-NOAA https://github.com/HelinWei-NOAA https://github.com/HelinWei-NOAA https://github.com/HelinWei-NOAA Thanks. I missed the postdet script. However, changes are still needed if we agree to use the same oro and fix datasets for both FRAC_GRD=T and FRAC_GRID=F. This is necessary if we want to compare coupled vs atmos-only runs. The new "MODIS+" based oro and masks etc should be used for all UFS global applications from this point forward. — Reply to this email directly, view it on GitHub <#863 (comment) https://github.com/NOAA-EMC/global-workflow/issues/863#issuecomment-1158341144>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLVRYTRRAA7V3JINHVO6C3VPPB6PANCNFSM5Y63DLTA https://github.com/notifications/unsubscribe-auth/ALLVRYTRRAA7V3JINHVO6C3VPPB6PANCNFSM5Y63DLTA . You are receiving this because you are subscribed to this thread.Message ID: @.> -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: @.*** Phone: (301) 683-3718 Fax: (301) 683-3718

I found it on Dogwood at /lfs/h2/emc/global/save/emc.global/FIX/fix_NEW

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/global-workflow/issues/863#issuecomment-1158960697, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLVRYVMDAVM2FRCLDR6INDVPSHTRANCNFSM5Y63DLTA . You are receiving this because you commented.Message ID: @.***>

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718

e-mail: @.*** Phone: (301) 683-3718 Fax: (301) 683-3718

GeorgeGayno-NOAA commented 2 years ago

@GeorgeGayno-NOAA George, for consistency UFS_UTIL should also be using the "new" fix_fv3_fracoro datasets instead of fix_fv3_gmted2010 for future UFS-based applications.

I will work this under - https://github.com/ufs-community/UFS_UTILS/issues/659

yangfanglin commented 2 years ago

@KateFriedman-NOAA Kate, could you please copy the C1152 dataset from /scratch1/NCEPDEV/global/Helin.Wei/sfc_fixed_mar2022 to the standard ./fix_fv3_fracoro directory and make it available on all supported platforms ? @HelinWei-NOAA where are the corresponding oro/mask dataset you used?

HelinWei-NOAA commented 2 years ago

I used those data under /scratch1/NCEPDEV/global/Helin.Wei/lake_p8_viirs2. Bing got them from Shan.

yangfanglin commented 2 years ago

@HelinWei-NOAA I do not see C1152 oro/mask data under /scratch1/NCEPDEV/global/Helin.Wei/lake_p8_viirs2

HelinWei-NOAA commented 2 years ago

@yangfanglin Actually I used the data under /scratch1/NCEPDEV/global/Helin.Wei/lake_p8_water_fraction2020 ( a copy from /scratch2/NCEPDEV/ensemble/Bing.Fu/lake_p8_water_fraction2020).

KateFriedman-NOAA commented 2 years ago

@yangfanglin Just catching up from being on leave and taking care of WCOSS2 go-live tasks...you want the files from here:

/scratch1/NCEPDEV/global/Helin.Wei/sfc_fixed_mar2022/sfc.C1152/

...to where? fix_NEW/fix_fv3_fracoro/??? Right under the fix_fv3_fracoro folder or within a C1152 subfolder? Here is a sample view of that folder right now:

.../emc.glopara/git/fv3gfs/fix_NEW/fix_fv3_fracoro> pwd
/gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/fv3gfs/fix_NEW/fix_fv3_fracoro
[ =0 20:17:11 1058 emc.glopara@v71a1 ]
.../emc.glopara/git/fv3gfs/fix_NEW/fix_fv3_fracoro> ll
total 20
drwxr-sr-x 3 emc.glopara emcmodel 2048 May  4 01:56 C192.mx050_frac
drwxr-sr-x 3 emc.glopara emcmodel 2048 May  4 01:56 C384.mx025_frac
drwxr-sr-x 3 emc.glopara emcmodel 2048 May  4 01:55 C48.mx400_frac
drwxr-sr-x 3 emc.glopara emcmodel 2048 May  4 01:57 C768.mx025_frac
drwxr-sr-x 3 emc.glopara emcmodel 2048 May  4 01:55 C96.mx100_frac
[ =0 20:17:12 1059 emc.glopara@v71a1 ]
.../emc.glopara/git/fv3gfs/fix_NEW/fix_fv3_fracoro> ll C768.mx025_frac/
total 3623432
-rwxr-xr-x 1 emc.glopara emcmodel 132258685 May  4 01:38 C768_grid.tile1.nc
-rwxr-xr-x 1 emc.glopara emcmodel 132258685 May  4 01:38 C768_grid.tile2.nc
-rwxr-xr-x 1 emc.glopara emcmodel 132258685 May  4 01:38 C768_grid.tile3.nc
-rwxr-xr-x 1 emc.glopara emcmodel 132258685 May  4 01:38 C768_grid.tile4.nc
-rwxr-xr-x 1 emc.glopara emcmodel 132258685 May  4 01:38 C768_grid.tile5.nc
-rwxr-xr-x 1 emc.glopara emcmodel 132258685 May  4 01:38 C768_grid.tile6.nc
-rwxr-xr-x 1 emc.glopara emcmodel     22018 May  4 01:38 C768_mosaic.nc
drwxr-sr-x 2 emc.glopara emcmodel      4096 May  4 01:57 fix_sfc
-rwxr-xr-x 1 emc.glopara emcmodel 901252888 May  4 01:38 fv3_SCRIP_C768_GRIDSPEC_lon3072_lat1536.gaussian.bilinear.nc
-rwxr-xr-x 1 emc.glopara emcmodel 674760496 May  4 01:38 fv3_SCRIP_C768_GRIDSPEC_lon3072_lat1536.gaussian.neareststod.nc
-rw-r--r-- 1 emc.glopara emcmodel  51928390 May  4 01:52 oro_C768.mx025.tile1.nc
-rw-r--r-- 1 emc.glopara emcmodel  51928390 May  4 01:52 oro_C768.mx025.tile2.nc
-rw-r--r-- 1 emc.glopara emcmodel  51928390 May  4 01:52 oro_C768.mx025.tile3.nc
-rw-r--r-- 1 emc.glopara emcmodel  51928390 May  4 01:52 oro_C768.mx025.tile4.nc
-rw-r--r-- 1 emc.glopara emcmodel  51928390 May  4 01:52 oro_C768.mx025.tile5.nc
-rw-r--r-- 1 emc.glopara emcmodel  51928390 May  4 01:52 oro_C768.mx025.tile6.nc
-rwxr-xr-x 1 emc.glopara emcmodel 466875993 May  4 01:38 remap_weights_C768_0p125deg.nc
-rwxr-xr-x 1 emc.glopara emcmodel 253747609 May  4 01:38 remap_weights_C768_0p25deg.nc
-rwxr-xr-x 1 emc.glopara emcmodel 168966297 May  4 01:38 remap_weights_C768_0p5deg.nc
-rwxr-xr-x 1 emc.glopara emcmodel 132001817 May  4 01:38 remap_weights_C768_1deg.nc

Any other files? @HelinWei-NOAA ?

yangfanglin commented 2 years ago

@KateFriedman-NOAA Please add to /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/fv3gfs/fix_NEW/fix_fv3_fracoro/C1152.mx025_frac

KateFriedman-NOAA commented 2 years ago

@yangfanglin I've pulled the new files into the requested subfolder in the fix_NEW set on WCOSS2 (new primary set). See here:

/lfs/h2/emc/global/save/emc.global/FIX/fix_NEW/fix_fv3_fracoro/C1152.mx025_frac

Before I rsync it to the other platforms...do you want it to match the other folders that have those files in a fix_sfc folder...like this one:

/lfs/h2/emc/global/save/emc.global/FIX/fix_NEW/fix_fv3_fracoro/C192.mx050_frac/fix_sfc

yangfanglin commented 2 years ago

Yes

KateFriedman-NOAA commented 2 years ago

@yangfanglin I have pulled the new C1152.mx025_frac fix files into fix_NEW and rsync'd them to Venus, Mars, Dogwood, Cactus, Hera, and Jet. Will rsync to Orion after today's maintenance concludes. I'm also making a new tarball on HPSS of fix_NEW:

/5year/NCEPDEV/emc-global/emc.glopara/fv3fix/fix_NEW_20220629.tar

.../modeling/noscrub/emc.glopara/git/fv3gfs> ll fix_NEW/fix_fv3_fracoro/C1152.mx025_frac/fix_sfc/
total 2565120
-rw-r--r-- 1 emc.glopara global  15934201 Mar 15 20:31 C1152.facsf.tile1.nc
-rw-r--r-- 1 emc.glopara global  15934201 Mar 15 20:31 C1152.facsf.tile2.nc
-rw-r--r-- 1 emc.glopara global  15934201 Mar 15 20:31 C1152.facsf.tile3.nc
-rw-r--r-- 1 emc.glopara global  15934201 Mar 15 20:31 C1152.facsf.tile4.nc
-rw-r--r-- 1 emc.glopara global  15934201 Mar 15 20:31 C1152.facsf.tile5.nc
-rw-r--r-- 1 emc.glopara global  15934201 Mar 15 20:31 C1152.facsf.tile6.nc
-rw-r--r-- 1 emc.glopara global  15934146 Mar 15 20:31 C1152.maximum_snow_albedo.tile1.nc
-rw-r--r-- 1 emc.glopara global  15934146 Mar 15 20:31 C1152.maximum_snow_albedo.tile2.nc
-rw-r--r-- 1 emc.glopara global  15934146 Mar 15 20:31 C1152.maximum_snow_albedo.tile3.nc
-rw-r--r-- 1 emc.glopara global  15934146 Mar 15 20:31 C1152.maximum_snow_albedo.tile4.nc
-rw-r--r-- 1 emc.glopara global  15934146 Mar 15 20:31 C1152.maximum_snow_albedo.tile5.nc
-rw-r--r-- 1 emc.glopara global  15934146 Mar 15 20:31 C1152.maximum_snow_albedo.tile6.nc
-rw-r--r-- 1 emc.glopara global  15934206 Mar 15 20:31 C1152.slope_type.tile1.nc
-rw-r--r-- 1 emc.glopara global  15934206 Mar 15 20:31 C1152.slope_type.tile2.nc
-rw-r--r-- 1 emc.glopara global  15934206 Mar 15 20:31 C1152.slope_type.tile3.nc
-rw-r--r-- 1 emc.glopara global  15934206 Mar 15 20:31 C1152.slope_type.tile4.nc
-rw-r--r-- 1 emc.glopara global  15934206 Mar 15 20:31 C1152.slope_type.tile5.nc
-rw-r--r-- 1 emc.glopara global  15934206 Mar 15 20:31 C1152.slope_type.tile6.nc
-rw-r--r-- 1 emc.glopara global 265435004 Mar 15 20:31 C1152.snowfree_albedo.tile1.nc
-rw-r--r-- 1 emc.glopara global 265435004 Mar 15 20:31 C1152.snowfree_albedo.tile2.nc
-rw-r--r-- 1 emc.glopara global 265435004 Mar 15 20:31 C1152.snowfree_albedo.tile3.nc
-rw-r--r-- 1 emc.glopara global 265435004 Mar 15 20:31 C1152.snowfree_albedo.tile4.nc
-rw-r--r-- 1 emc.glopara global 265435004 Mar 15 20:31 C1152.snowfree_albedo.tile5.nc
-rw-r--r-- 1 emc.glopara global 265435004 Mar 15 20:31 C1152.snowfree_albedo.tile6.nc
-rw-r--r-- 1 emc.glopara global  15934136 Mar 15 20:31 C1152.soil_type.tile1.nc
-rw-r--r-- 1 emc.glopara global  15934136 Mar 15 20:31 C1152.soil_type.tile2.nc
-rw-r--r-- 1 emc.glopara global  15934136 Mar 15 20:31 C1152.soil_type.tile3.nc
-rw-r--r-- 1 emc.glopara global  15934136 Mar 15 20:31 C1152.soil_type.tile4.nc
-rw-r--r-- 1 emc.glopara global  15934136 Mar 15 20:31 C1152.soil_type.tile5.nc
-rw-r--r-- 1 emc.glopara global  15934136 Mar 15 20:31 C1152.soil_type.tile6.nc
-rw-r--r-- 1 emc.glopara global  15934251 Mar 15 20:31 C1152.substrate_temperature.tile1.nc
-rw-r--r-- 1 emc.glopara global  15934251 Mar 15 20:31 C1152.substrate_temperature.tile2.nc
-rw-r--r-- 1 emc.glopara global  15934251 Mar 15 20:31 C1152.substrate_temperature.tile3.nc
-rw-r--r-- 1 emc.glopara global  15934251 Mar 15 20:31 C1152.substrate_temperature.tile4.nc
-rw-r--r-- 1 emc.glopara global  15934251 Mar 15 20:31 C1152.substrate_temperature.tile5.nc
-rw-r--r-- 1 emc.glopara global  15934251 Mar 15 20:31 C1152.substrate_temperature.tile6.nc
-rw-r--r-- 1 emc.glopara global  74326767 Mar 15 20:31 C1152.vegetation_greenness.tile1.nc
-rw-r--r-- 1 emc.glopara global  74326767 Mar 15 20:31 C1152.vegetation_greenness.tile2.nc
-rw-r--r-- 1 emc.glopara global  74326767 Mar 15 20:31 C1152.vegetation_greenness.tile3.nc
-rw-r--r-- 1 emc.glopara global  74326767 Mar 15 20:31 C1152.vegetation_greenness.tile4.nc
-rw-r--r-- 1 emc.glopara global  74326767 Mar 15 20:31 C1152.vegetation_greenness.tile5.nc
-rw-r--r-- 1 emc.glopara global  74326767 Mar 15 20:31 C1152.vegetation_greenness.tile6.nc
-rw-r--r-- 1 emc.glopara global  15934245 Mar 15 20:31 C1152.vegetation_type.tile1.nc
-rw-r--r-- 1 emc.glopara global  15934245 Mar 15 20:31 C1152.vegetation_type.tile2.nc
-rw-r--r-- 1 emc.glopara global  15934245 Mar 15 20:31 C1152.vegetation_type.tile3.nc
-rw-r--r-- 1 emc.glopara global  15934245 Mar 15 20:31 C1152.vegetation_type.tile4.nc
-rw-r--r-- 1 emc.glopara global  15934245 Mar 15 20:31 C1152.vegetation_type.tile5.nc
-rw-r--r-- 1 emc.glopara global  15934245 Mar 15 20:31 C1152.vegetation_type.tile6.nc
yangfanglin commented 2 years ago

Thanks, Kate

KateFriedman-NOAA commented 2 years ago

The Orion fix_NEW set are also now updated.

WalterKolczynski-NOAA commented 2 years ago

@yangfanglin Can you confirm fix is correct now so we can close this issue?

ChunxiZhang-NOAA commented 2 years ago

@WalterKolczynski-NOAA @yangfanglin I believe this fix has not been done yet. Fanglin sent me the modified code a while ago and asked me to create a pull request, but I didn't do it. I will create a pull request soon.

KateFriedman-NOAA commented 1 year ago

Is there any update on this issue? Thanks!

ChunxiZhang-NOAA commented 1 year ago

@KateFriedman-NOAA I believe that no updates for this issue.

KateFriedman-NOAA commented 1 year ago

@ChunxiZhang-NOAA Does the issue still exist? Is a fix in the works still? The last update from you in August stated a PR would be submitted soon. Is that still the plan? Looking to get this either resolved or closed. Thanks!

ChunxiZhang-NOAA commented 1 year ago

@KateFriedman-NOAA One issue still remains. I created draft PR#1182 and UFS_UTILS PR#741 to make all applications use the latest fix, mask and oro datasets for both fractional and non-fractional grid.