search

NCAR / DART

Data Assimilation Research Testbed
https://dart.ucar.edu/
Apache License 2.0
197 stars 145 forks source link

Pangu-DART interface to support DA with ensemble ML models #737

Open Nuo-Chen opened 2 months ago

Nuo-Chen commented 2 months ago

Description:

Adding Pangu-DART interface to support DA with ensemble ML models.

Fixes issue

Types of changes

Documentation changes needed?

Tests

Please describe any tests you ran to verify your changes.

Checklist for merging

Checklist for release

Testing Datasets

Testing reanalysis dataset at /glade/work/chennuo/code/pangu/dart_pangu/era5_data/ Testing observation data at /glade/work/chennuo/code/cice_old/DART/observations/obs_converters/NCEP/prep_bufr/data/20240120/prepout

hkershaw-brown commented 2 months ago

Thanks for this Nuo!

hkershaw-brown commented 2 weeks ago

note on AI models, no terrain surface - problems for altimeter obs.

criedel40 commented 2 weeks ago

Yeah I had to make the code read in terrain surface data for this reason. -Chris

On Fri, Nov 8, 2024 at 7:58 AM Helen Kershaw @.***> wrote:

note on AI models, no terrain surface - problems for altimeter obs.

— Reply to this email directly, view it on GitHub https://github.com/NCAR/DART/pull/737#issuecomment-2464834338, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIVLWAOUXE66YBMJD77J53Z7S7SDAVCNFSM6AAAAABOZG3FVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRUHAZTIMZTHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

hkershaw-brown commented 2 weeks ago

Chris come back to dares! We were talking about this at the standup, reading in terrain from another source. We've got two pangu projects on the go, and I think we're going to hit a few things (e.g. terrain) that apply to AI atmosphere models in general.

criedel40 commented 2 weeks ago

So I think it will be dependent on the grid you are using right? Our training dataset is ERA5 so it makes perfect sense that I use the terrain data from ERA5. I guess it depends on the data source used for Pangu really. However I imagine it depends more on the grid structure and resolution. I have to write data from binary numpy files (yuck I know) to netcdf to make it clean for DART so I just add terrain data to the first member's netcdf file. The AI group at Windborne loves numpy binary files. I actually have some real reservations on how DA will actually work with AI models due to their makeup. The other thing I absolutely love (really hate) is our AI model is defined on pressure levels, which goes down to 1000hPa. In some areas, heights at 1000hPa are below the surface level hahaha. Just nuts that it actually works and produces forecasts that are that much different, however, we are just at cold-start DA at the moment.

Tell Jeff to get some money! -Chris

On Fri, Nov 8, 2024 at 8:25 AM Helen Kershaw @.***> wrote:

Chris come back to dares! We were talking about this at the standup, reading in terrain from another source. We've got two pangu projects on the go, and I think we're going to hit a few things (e.g. terrain) that apply to AI atmosphere models in general.

— Reply to this email directly, view it on GitHub https://github.com/NCAR/DART/pull/737#issuecomment-2464895261, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIVLWGN5MEK3UZ3NQJV7ATZ7TCWBAVCNFSM6AAAAABOZG3FVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRUHA4TKMRWGE . You are receiving this because you commented.Message ID: @.***>

Nuo-Chen commented 2 weeks ago

I think convert_pgout_to_nc.py reads in terrain.npy, though the path was linked to my work directory. terrain.npy and landmask.npy are located in model/pangu, both are created them with WRF-WPS.

hkershaw-brown commented 2 weeks ago

awesome thanks Nuo!