ufs-community / ufs-weather-model

UFS Weather Model
Other
136 stars 244 forks source link

Error running rrfs_v1nssl_intel with -DDEBUG=ON using ufs-community:develop #1944

Open zach1221 opened 11 months ago

zach1221 commented 11 months ago

Description

Experience failure when attempting to run rrfs_v1nssl_intel on Hera, with -DDEBUG=ON, against ufs-community:develop branch.

To Reproduce:

  1. log into Hera.
  2. Clone ufs-community:dev branch
  3. edit rt.conf to add -DDEBUG=ON to compile line of case rrfs_v1nssl_intel
  4. Run test

Additional context

Example from Denise's attempt /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/rt_301756/rrfs_v1nssl_intel image

That line is for bdsnow below

image

Output

DeniseWorthen commented 11 months ago

Clarification...I think your output above showing the failure in module_mp_nssl_2mom.F90 is the failure that @BrianCurtis-NOAA got w/ the PR #1915 branch on Acorn in non-debug mode. The failure I posted was for debug mode at develop 68050e5

zach1221 commented 11 months ago

Running debug with gnu on Hercules. Some failures, but mostly due to timeouts. rrfs_v1beta_gnu seemed to be the only one to fail from another cause. I'll re-test the others that failed with an extended wall clock. working dir = /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_2728437/
Logs: /work/noaa/nems/zshrader/hercules/rt-1915/tests/logs/RegressionTests_hercules.log

zach1221 commented 11 months ago

Clarification...I think your output above showing the failure in module_mp_nssl_2mom.F90 is the failure that @BrianCurtis-NOAA got w/ the PR #1915 branch on Acorn in non-debug mode. The failure I posted was for debug mode at develop 68050e5

Yes, that's true. I'll remove it.

DeniseWorthen commented 11 months ago

@zach1221 I don't see the rrfs_v1nssl test in your run directory.

zach1221 commented 11 months ago

@zach1221 I don't see the rrfs_v1nssl test in your run directory.

@DeniseWorthen it doesn't look like rrfs_v1nssl is part of the gnu tests in rt.conf actually. I can try to compile it with gnu.

zach1221 commented 11 months ago

It fails with gnu. Screenshot of the err log but I'm not sure it's helpful. image /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_2824213/rrfs_v1nssl_gnu

BrianCurtis-NOAA commented 11 months ago

It fails with gnu. Screenshot of the err log but I'm not sure it's helpful. image /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_2824213/rrfs_v1nssl_gnu

Can't see it well in the SS, but it looks like the glacier file where we see the issue with the intel compiler?

zach1221 commented 11 months ago

It fails with gnu. Screenshot of the err log but I'm not sure it's helpful. image /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_2824213/rrfs_v1nssl_gnu

Can't see it well in the SS, but it looks like the glacier file where we see the issue with the intel compiler?

Yes, sorry I cut it off just above the glacier file portion I think. image

zach1221 commented 11 months ago

pointing to the same bdsnow line in module_sf_noahmp_glacier.F90 , that Denise posted previously.