search

NOAA-CEFI-Portal / regional_mom6

CEFI regional MOM6 preprocessing package for CEFI data portal group
0 stars 0 forks source link

Forecast data restructure #31

Closed chiaweh2 closed 4 months ago

chiaweh2 commented 6 months ago

The original forecast file from GFDL is in single file with all initialization on March lump together in one file. This include all years between 1993-2022, all member, all lead time. This create a large file ~18 GB of even only single variable.

For better data organization and extendability in the future, we decided to restructure the forecast data file to one single initialization one variable one file. In the single file, it should have one single init: yyyy-mm, one variable, (one anomaly of the variable), all members, all lead time. This create a single file of ~600MB size which is more manageable and better to download if user are only interested in a single forecast result.

A code modification on the io part might be needed. Currently possible candidate are the mom6_io.MOM6forecast and other script that might use this to get the forecast data.

The data restructure will also need the renaming of the directory name store on-prem and opendap server from hindcast => forecast to avoid confusion on the meaning of "hindcast". We will avoid using the phrase in the comment and data directory since it means differently between oceanography world and atmospheric science world.

chiaweh2 commented 6 months ago
chiaweh2 commented 6 months ago

@KevinCounts the forecast data has been restructured as discussed during last week's meeting. The new code base in the develop branch is also accounting this data update. Meaning using the mom6_io method in the develop branch should be able to read in the new forecast data. Please try it out. The new forecast data is now in the forecast/ folder. The old hindcast/ is kept unchanged to let the data portal backend works during the CEFI summit this week (I froze the backend code changes to make sure data portal works during the summit).

The backend code will be update once the summit is over. Please make sure to no change the hindcast/ folder at the moment. I have keep the todo list there to remind me of the backend changes is unfinished. P.S. the main branch has not include this change yet. I intend to open a PR once the backend code is updated.

The new forecast data should be easier to work with for your bottom temperature area averaged code. Please feel free let me know if you have any difficulties on using the new restructured data.