search

NOAA-EMC / CMEPS

NUOPC Community Mediator for Earth Prediction Systems
https://escomp.github.io/CMEPS/
0 stars 20 forks source link

Updated land-sea mask used for CFSR or GEFS data. #35

Closed binli2337 closed 3 years ago

binli2337 commented 3 years ago

Description of changes

The land-sea masks used in CDEPS for CFSR/GEFS data sources are different from those used in NEMS_datm.

Specific notes

The med_map_mod.F90 file needs to be updated.

CMEPS Issues Fixed: #34

Are changes expected to change answers? No

DeniseWorthen commented 3 years ago

Thanks Bin. I tested Ufuk's PIO changes when he had the PR to ESCOMP and I know they're fine for us. Do you need those changes brought in right away?

binli2337 commented 3 years ago

@DeniseWorthen Yes, please bring those changes to NOAA-EMC/CMEPS. Thanks!

DeniseWorthen commented 3 years ago

I'm confused about the process here. Are we merging this to develop independent of a ufs-weather pr?

I see this is a minor change; are we planning on merging it to emc/develop and then after that bring in the latest CMEPS master?

binli2337 commented 3 years ago

@DeniseWorthen Please submit a new CMEPS PR that will include changes from ESCOMP/CMEPS and my PR. My ufs-weather-model PR will be submitted by this Friday. The CMEPS PR will be merged to the develop branch after all ufs-weather-model regression tests are successful.

DeniseWorthen commented 3 years ago

@binli2337 Great, thanks. I understand. I'll create a PR soon for updating to the latest ESCOMP/master.

@junwang-noaa I did want to you about pnetcdf. Ufuk did some basic testing when he was merging the PIO changes to ESCOMP. He tested pio using netcdf vs pio using pnetcdf in the coupled model (c96) and he found dramatically different run times---126s w/ pnetcdf vs 700s for netcdf. We should of course verify this in our own testing, but what is preventing us from utilizing pnetcdf ?

junwang-noaa commented 3 years ago

Sorry for the confusion. It is good If the components such as CMEPS or CICE6 get better IO performance when using the updated PIO with pnetcdf, nothing prevents us from using it. I believe the timing you show (700s for netcdf) is sequential netcdf. We do have parallel netcdf (without using PIO and pnetcdf) in fv3 which gives us 10 times faster performance compared to sequential netcdf. Also We have our own customized lossy compression algorithm in FV3 which can reduce the data size to 16% original size. I am not sure if pnetcdf is going to implement those features in the future. What I am concerned about is putting PIO to the top level driver in NEMS that adds additional dependency for standalone FV3 which has advanced features and does not use PIO and pnetcdf at all, I am totally OK to use PIO in component level.

On Tue, Mar 9, 2021 at 3:23 PM Denise Worthen notifications@github.com wrote:

@binli2337 https://github.com/binli2337 Great, thanks. I understand. I'll create a PR soon for updating to the latest ESCOMP/master.

@junwang-noaa https://github.com/junwang-noaa I did want to you about pnetcdf. Ufuk did some basic testing when he was merging the PIO changes to ESCOMP. He tested pio using netcdf vs pio using pnetcdf in the coupled model (c96) and he found dramatically different run times---126s w/ pnetcdf vs 700s for netcdf. We should of course verify this in our own testing, but what is preventing us from utilizing pnetcdf ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/CMEPS/pull/35#issuecomment-794406889, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TOWBRVSDGP3BGGMAY3TCZYTVANCNFSM4Y4CQAUQ .

DeniseWorthen commented 3 years ago

@junwang-noaa I don't understand all the flavors of netcdf. Isn't pnetcdf another name for parallel-netcdf?

junwang-noaa commented 3 years ago

Here is some description from pnetcdf https://parallel-netcdf.github.io/: "The PnetCDF project started in 2001, independently from the Unidata's NetCDF project. Applications can use PnetCDF library completely without NetCDF library". While netcdf parallel io https://www.unidata.ucar.edu/software/netcdf/docs/parallel_io.html " provides parallel file access to both classic and netCDF-4/HDF5 files. The parallel I/O to netCDF-4 files is achieved through the HDF5 library while the parallel I/O to classic files is through PnetCDF. A few functions have been added to the netCDF C API to handle parallel I/O."

We are using netcdf parallel I/O though HDF5 library in FV3.

On Wed, Mar 10, 2021 at 7:56 AM Denise Worthen notifications@github.com wrote:

@junwang-noaa https://github.com/junwang-noaa I don't understand all the flavors of netcdf. Isn't pnetcdf another name for parallel-netcdf?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/CMEPS/pull/35#issuecomment-795375029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TN5XITVLAGU6TMDLKLTC5M7HANCNFSM4Y4CQAUQ .