Grid imprinting when running C192 atmos with 1/4 deg ocean and seaice

guillaumevernieres commented 1 month ago

Description

We can see the imprinting of a grid on the sea ice and ocean. The figure attached depicts the seaice concentration standard deviation of a 30 member ensemble after a 6 hour forecast.

The initial conditions were created by: 1 - interpolating the gfs ensemble members from operations from C384 to C192 2 - adding the same ocean and seaice initial conditions for the 30 members 3 - omitting the mediator restarts oops-seice-stddev

To Reproduce:

I don't think this is machine dependent, but my test was done on Hercules.

Use a configuration of the coupled UFS that has FV3, MOM6 and CICE6
use C192 for FV3 and 1/4 deg for MOM6 and CICE
bootstrap/omit the mediator restart

guillaumevernieres commented 1 month ago

pinging @DeniseWorthen and @junwang-noaa

junwang-noaa commented 1 month ago

@DeniseWorthen Is this issue related to how ocean/ice grids/meshes are created or the interpolation issue between the two components?

DeniseWorthen commented 1 month ago

I believe the issue is related to the conservative mapping we do for ATM states->OCN and ICE. And at least one reason we don't do that is the very old issue of "stripes" that Ufuk mentioned recently. That issue is https://github.com/NOAA-EMC/CMEPS/issues/38#issuecomment-1059764509

benjamin-cash commented 1 month ago

@guillaumevernieres - could you point me to the ICs on Hercules and hash of the model/workflow you are using? I have the model set up to run C192mx025 on Frontera and can see if I get the same result. I'm also having an issue with the model crashing on start up that I sort of suspect is related to the ICs I'm using, so this would also let me test that.

guillaumevernieres commented 1 month ago

@guillaumevernieres - could you point me to the ICs on Hercules and hash of the model/workflow you are using? I have the model set up to run C192mx025 on Frontera and can see if I get the same result. I'm also having an issue with the model crashing on start up that I sort of suspect is related to the ICs I'm using, so this would also let me test that.

Warm restarts for m001:

/work/noaa/da/gvernier/sandboxes/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/GFSv17-3DVAR-C384mx025/COMROOT/GFSv17-3DVAR-C384mx025/enkfgdas.20210630/12/mem001/model

Commit of the UFS (ufs_model.fd): commit 6a4e09e94773ffa39ce7ab6a54a885efada91f21

benjamin-cash commented 1 month ago

@guillaumevernieres - It looks like I will need permissions added to see those files, starting with the sandboxes directory:

[cash@hercules-login-3 ~]$ ls -l /work/noaa/da/gvernier
total 679420
-rwxr-x---  1 gvernier da 690850711 Jun 13  2022 Anaconda3-2022.05-Linux-x86_64.sh
drwxr-sr-x 11 gvernier da      4096 Apr 17 15:10 ai
drwxr-s--- 28 gvernier da      4096 Jun 13  2022 anaconda3
drwxr-sr-x  8 gvernier da      4096 Sep 30 17:07 ensda
-rw-r-----  1 gvernier da     11021 Jun 13  2022 oceanview.yaml
drwxr-s---  6 gvernier da      4096 Oct 10 18:23 sandboxes

guillaumevernieres commented 1 month ago

@guillaumevernieres - It looks like I will need permissions added to see those files, starting with the sandboxes directory:

[cash@hercules-login-3 ~]$ ls -l /work/noaa/da/gvernier
total 679420
-rwxr-x---  1 gvernier da 690850711 Jun 13  2022 Anaconda3-2022.05-Linux-x86_64.sh
drwxr-sr-x 11 gvernier da      4096 Apr 17 15:10 ai
drwxr-s--- 28 gvernier da      4096 Jun 13  2022 anaconda3
drwxr-sr-x  8 gvernier da      4096 Sep 30 17:07 ensda
-rw-r-----  1 gvernier da     11021 Jun 13  2022 oceanview.yaml
drwxr-s---  6 gvernier da      4096 Oct 10 18:23 sandboxes

Sure, let's take this offline @benjamin-cash . My email is guillaume.vernieres@noaa.gov

guillaumevernieres commented 1 month ago

Let's try this: /work/noaa/da/gvernier/mem001 not sure this will work since da and gvernier are definitely da

Guillaume Vernieres

On Fri, Oct 11, 2024 at 3:01 PM benjamin-cash @.***> wrote:

@guillaumevernieres https://github.com/guillaumevernieres - It looks like I will need permissions added to see those files, starting with the sandboxes directory:

@.*** ~]$ ls -l /work/noaa/da/gvernier total 679420 -rwxr-x--- 1 gvernier da 690850711 Jun 13 2022 Anaconda3-2022.05-Linux-x86_64.sh drwxr-sr-x 11 gvernier da 4096 Apr 17 15:10 ai drwxr-s--- 28 gvernier da 4096 Jun 13 2022 anaconda3 drwxr-sr-x 8 gvernier da 4096 Sep 30 17:07 ensda -rw-r----- 1 gvernier da 11021 Jun 13 2022 oceanview.yaml drwxr-s--- 6 gvernier da 4096 Oct 10 18:23 sandboxes

— Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-weather-model/issues/2466#issuecomment-2407973259, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLBX4GYIM4SIRIOCCLJ4FDZ3AN75AVCNFSM6AAAAABPY624H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBXHE3TGMRVHE . You are receiving this because you were mentioned.Message ID: @.***>

sanAkel commented 1 month ago

Let's try this: /work/noaa/da/gvernier/mem001 not sure this will work since da and gvernier are definitely da --- Guillaume Vernieres …

It works for me!

CatherineThomas-NOAA commented 1 week ago

@junwang-noaa @LarissaReames-NOAA: There was some discussion in our GFS workflow meeting last week about a beta snapshot of ESMF that may potentially address this issue. Is there a test version installed somewhere that the marine DA could use to test?

DeniseWorthen commented 1 week ago

You may be referring to a snapshot that Ufuk has tested which resolves his "stripes" issue. That issue is tangentially related to the grid imprint issue. That is, we have not been using bilinear mapping of states from ATM->ICE (which ICE then uses to calculate it's own fluxes) because of the underlying stripes issue. But Guillaume reported that a test branch using bilinear mapping did not resolve the issue.

I've been using Ufuk's build on hercules with a modification (below) to the modulefiles, but I still see an "imprint" using a C48-1deg OCN/ICE.

But, heads up, I'm going to need to put further investigation and debugging on hold while I tackle issue #2486.

diff --git a/modulefiles/ufs_hercules.intel.lua b/modulefiles/ufs_hercules.intel.lua
index 455ea4d0..05823161 100644
--- a/modulefiles/ufs_hercules.intel.lua
+++ b/modulefiles/ufs_hercules.intel.lua
@@ -18,6 +18,10 @@ load("ufs_common")
 nccmp_ver=os.getenv("nccmp_ver") or "1.9.0.1"
 load(pathJoin("nccmp", nccmp_ver))

+unload("esmf/8.6.0")
+prepend_path("MODULEPATH", "/work2/noaa/nems/tufuk/LAND/esmf/modulefiles")
+load(pathJoin("esmf", "8.6.0"))
+
 setenv("CC", "mpiicc")
 setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")diff --git a/modulefiles/ufs_hercules.intel.lua b/modulefiles/ufs_hercules.intel.lua

ufs-community / ufs-weather-model