search

NOAA-EMC / NCEPLIBS

Top level repo containing submodules for NCEPLIBS and associated dependencies for superproject builds
Other
42 stars 18 forks source link

Add new NCEPLIBS library: ncio #184

Closed aerorahul closed 3 years ago

aerorahul commented 3 years ago

In GFS v16 the model (UFS) will start producing netCDF output (as opposed to nemsio). The analysis system (GSI) and utilities such as recentering the ensemble in the workflow, GEFS and others use an netCDF user-interface library here to read and write netCDF files produced by the model. It is built as a self-contained library in the GSI. However, other components have plucked it out and there are multiple copies of this library in several places.

This issue is to start a discussion on:

  1. Can this library be turned into a NCEPLIBS-fv3gfsncio and made available via hpc-stack?
  2. If not, we accept that this code base will be duplicated in applications scattered wherever UFS netCDF output is used.

Describe the solution you'd like It deserves its own repository. It would be useful if this library is included in the hpc-stack for other UFS applications that use this library and its functionality.

@arunchawla-NOAA @jswhit2 @edwardhartnett @CoryMartin-NOAA @kgerheiser

jswhit commented 3 years ago

I did this a while ago (NCEPLIBS-ncio) but it is now out of sync with the version in GSI

aerorahul commented 3 years ago

@jswhit Thank you for pointing us to the repository. In your opinion, is it faster/worthwhile to merge the GSI version of fv3gfs_ncio into your repository or start fresh?
@arunchawla-NOAA @edwardhartnett This library is required for global-workflow utility of enkf_chgres_recenter_nc.fd. This utility is used in v16 as well as in GEFS workflows. This is a blocker on the issue https://github.com/NOAA-EMC/global-workflow/issues/164 in global-workflow

arunchawla-NOAA commented 3 years ago

@edwardhartnett

Can we make this part of NCEPLIBS? I would prefer if it did not have the name fv3gfs in it because that would imply that it works only for one application. UFS_ncio or plain NCEPLIBS-ncio is fine with me. We should be focussing on netcdf io anyway

edwardhartnett commented 3 years ago

I agree it would be great to isolate the I/O code. THat would really help my I/O work as well! For example we can including benchmarking tests, which can be used to help us know that we are getting the I/O we need out of all our systems.

@jswhit please make be an admin of the NCEPLIBS-ncio repo and I will integrate it into the rest of NCEPLIBS.

I will volunteer to write the tests, since this is netCDF code I am very interested in. Next time you guys present code as a new library, I trust you will have written your own test code.

The next step will be to include this in hpc_stack.

jswhit commented 3 years ago

@aerorahul I think it's probably better to start fresh, there have been a lot of changes (including addition of MPI by @CoryMartin-NOAA), plus a name change is warranted as pointed out by @arunchawla-NOAA

@edwardhartnett I'm not the admin of NCEPLIBS-ncio (I think @Hang-Lei-NOAA is)

aerorahul commented 3 years ago

@jswhit2 I will move the NCEPLIBS-ncio repo to NCEPLIBS-ncio-old and start with a new fresh code dump from the GSI. Then add CMake build and make @edwardhartnett the admin so he can do his magic. Is that agreeable to all?

jswhit commented 3 years ago

sounds great, thanks @aerorahul. Once that is done can we remove it from GSI?

arunchawla-NOAA commented 3 years ago

will this library work only for the atmospheric fields? Can it handle the capabilities that are right now in write grid component, i.e. parallel compressed netcdf I/O. I ask if this can be extended to be used by other component models in the UFS?

CoryMartin-NOAA commented 3 years ago

@arunchawla-NOAA this is what we use to read the model forecast atmf006 and sfcf006 files in GSI for GFSv16 in parallel compressed netCDF.

aerorahul commented 3 years ago

I think it is a general friendly API to the netCDF Fortran library and provides interfaces similar to netCDF-python for accessing data in the netCDF file. The only downside is for writing a file, it depends on a template.

Who produces the file, atm or ocean or anything else is probably irrelevant.

Hang-Lei-NOAA commented 3 years ago

This code is for GSI and originally created by Jeff. The NCEPLIBS team just managed it. Whatever the change is needed by GSI, it should be made. Once the new repo is used. We will clean the old repo, which is also a task under our refactoring plan.

On Mon, Feb 8, 2021 at 11:09 AM Rahul Mahajan notifications@github.com wrote:

I think it is a general friendly API to the netCDF Fortran library and provides interfaces similar to netCDF-python for accessing data in the netCDF file. The only downside is for writing a file, it depends on a template.

Who produces the file, atm or ocean or anything else is probably irrelevant.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS/issues/184#issuecomment-775256209, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFC2MIR3YYZI56YN7MDS6AEFDANCNFSM4XFOTGHQ .

aerorahul commented 3 years ago

2 PR's have been created in the NCEPLIBS-ncio repository https://github.com/NOAA-EMC/NCEPLIBS-ncio/pull/3 - clean up of cmake build (no source code changes) https://github.com/NOAA-EMC/NCEPLIBS-ncio/pull/4 - replace source code with GSI tag gfsda.v16.0.0 @edwardhartnett is an admin of NCEPLIBS-ncio

aerorahul commented 3 years ago

closing this issue.