Failed to verify UFS-AQM simulations for the large domain

[JianpingHuang-NOAA]

Hi ,

I failed to run the melodies-monet for verifying the UFS-AQM simulations for the large domain (i.e., AQM_NA_13km) on Hera due to a memory error.

1) The run log file on Hera /scratch2/NCEPDEV/naqfc/Jianping.Huang/MONET/main/ufsaqm_analysis/v7.0.c3/slurm-34661362.out

2) data files /scratch2/NCEPDEV/naqfc/Jianping.Huang/MONET/regional/processed/v7.0.c3/data

Please let me know if I need to provide more information.

Thanks,

Jianping

[bbakernoaa]

@JianpingHuang-NOAA can you also put the Melodies-Monet configuration file here?

[JianpingHuang-NOAA]

You mean control file?

/scratch2/NCEPDEV/naqfc/Jianping.Huang/MONET/main/ufsaqm_analysis/v7.0.c3/control_ufsaqm_example_submit_job.yaml

Jianping

On Fri, Aug 12, 2022 at 3:01 PM Barry Baker @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA can you also put the Melodies-Monet configuration file here?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-CSL/MELODIES-MONET/issues/94#issuecomment-1213427211, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI54LRTJW55HRXC2HIDVY2NPZANCNFSM56MSL6PA . You are receiving this because you were mentioned.Message ID: @.***>

[zmoon]

Thanks @JianpingHuang-NOAA . I see that it failed in the sorting by vertical level in RRFS reader. I will update the reader to do that in a less memory intensive way (only need to reverse).

[rschwant]

@zmoon if your new method of sorting the vertical does not lower the memory enough. You could also try moving this if statement on line 208 up. And only do the sorting if you need all the vertical levels like for satellite and aircraft and else select the top value. Those data sets where we need the vertical we are going to process a day at a time instead of a whole month to avoid memory issues. Then you'd have to skip the altitude calculation when surf_only = true, but we don't need that for any unit conversions, so that's okay. We just need pressure and temperature for the unit conversions.

if surf_only:
    dset = dset.isel(z=0).expand_dims("z", axis=1)

[zmoon]

@JianpingHuang-NOAA your job should work now if you use my rrfs branch of monetio (https://github.com/zmoon/monetio/tree/rrfs) in your conda environment. Or I can let you know when we have a new monetio conda-forge release with this fix.

[bbakernoaa]

@zmoon @JianpingHuang-NOAA Lets make sure this works then figure out the next steps for MELODIES-MONET

Thanks @zmoon for looking into this!

[zmoon]

@JianpingHuang-NOAA merging the PM2.5 files with the dyn files is not working correctly in this job currently because the pm files have lat/lon in single precision while the dyn files have lat/lon in double.

[bbakernoaa]

@JianpingHuang-NOAA Is this resolved?

[JianpingHuang-NOAA]

Yes, it is done. Thanks !

On Mon, Sep 19, 2022 at 3:09 PM Barry Baker @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA Is this resolved?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-CSL/MELODIES-MONET/issues/94#issuecomment-1251432445, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIZJY7ICINKKEPPHHALV7C27VANCNFSM56MSL6PA . You are receiving this because you were mentioned.Message ID: @.***>