search

ESCOMP / CDEPS

Community Data Models for Earth Prediction Systems
https://escomp.github.io/CDEPS/versions/master/html/index.html
18 stars 39 forks source link

CDEPS does not support data files on unstructured grid #249

Closed uturuncoglu closed 3 weeks ago

uturuncoglu commented 9 months ago

I think that current implementation of CDEPS does not allow to handle data files on unstructured mesh. It seems that shr_strdata_set_stream_iodesc() has some assumptions (like 2d fields are assuming both dimension is spatial dimension) about the time dimension. If I try to use those files with CDEPS, the case runs without any issue but the output fields are all zeros.

Following is an example data file:

netcdf Florence_HWRF_HRRR_HSOFS {
dimensions:
    level = 1 ;
    node = 1813443 ;
    element = 3564104 ;
    time = 522 ;
    noel = 3 ;
variables:
    double longitude(node) ;
        longitude:long_name = "longitude" ;
        longitude:units = "degree_east" ;
        longitude:field = "lon, scalar, series" ;
    double latitude(node) ;
        latitude:long_name = "latitude" ;
        latitude:units = "degree_north" ;
        latitude:field = "lat, scalar, series" ;
    double time(time) ;
        time:long_name = "julian day (UT)" ;
        time:units = "days since 1990-01-01 00:00:00" ;
        time:field = "time, scalar, series" ;
        time:conventions = "relative julian days with decimal part (as parts of the day )" ;
        time:axis = "T" ;
        time:calendar = "gregorian" ;
    int tri(element, noel) ;
    float uwnd(time, node) ;
        uwnd:units = "m s-1" ;
        uwnd:field = "U, scalar, series" ;
        uwnd:long_name = "eastward_wind" ;
        uwnd:_FillValue = 9.96920996838687e+36 ;
    float vwnd(time, node) ;
        vwnd:units = "m s-1" ;
        vwnd:field = "V, scalar, series" ;
        vwnd:long_name = "northward_wind" ;
        vwnd:_FillValue = 9.96920996838687e+36 ;
    float P(time, node) ;
        P:units = "Pa" ;
        P:field = "V, scalar, series" ;
        P:long_name = "Pressure Reduced to MSL" ;
        P:_FillValue = 9.96920996838687e+36 ;

We are handling these files by mapping them to regular lat-lon grid and use under CDEPS like that but it would be also nice to support unstructured data files. Anyway, @jedwards4b @mvertens let me know what do you think. Do you think bringing this support to CDEPS can be done quickly or requires more extensive work. We have a task under UFS Coastal project for it but it would be hard to estimate the time without knowing the limitations. Anyway, let me know if you have any idea or suggestions.

jedwards4b commented 9 months ago

I don't think that it would be too hard, I made a start in this branch. remote: Create a pull request for 'unstructured_data_support' on GitHub by visiting: remote: https://github.com/jedwards4b/CDEPS/pull/new/unstructured_data_support remote: To github.com:jedwards4b/CDEPS.git

Can you give it a try with your data?

klindsay28 commented 9 months ago

CASEROOT=/glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD has datm reading fields with dimensions (time, atmImp_ny, atmImp_nx) atmImp_nx = 48600 atmImp_ny = 1

Perhaps it will work for you if you add a singleton dimension to your data.

On Thu, Oct 19, 2023 at 1:06 PM Jim Edwards @.***> wrote:

I don't think that it would be too hard, I made a start in this branch. remote: Create a pull request for 'unstructured_data_support' on GitHub by visiting: remote: https://github.com/jedwards4b/CDEPS/pull/new/unstructured_data_support remote: To github.com:jedwards4b/CDEPS.git

  • [new branch] unstructured_data_support -> unstructured_data_support

Can you give it a try with your data?

— Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CDEPS/issues/249#issuecomment-1771556832, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADWZPO3Z3GY5GBXJLNVK4PLYAF24HAVCNFSM6AAAAAA6HUSG3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGU2TMOBTGI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

uturuncoglu commented 9 months ago

@jedwards4b Okay. Thanks. Let me try. @klindsay28 Adding singleton dimension also a good idea. Do you need to do anything special in the ESMF mesh file?

klindsay28 commented 9 months ago

I'm not very knowledgeable about ESMF mesh files, so I don't know what counts as special. That said, the mesh file that datm is being pointed to in the case above is at /glade/p/cesmdata/cseg/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc and its netCDF header is

netcdf ne30pg3_ESMFmesh_cdf5_c20211018 {
dimensions:
    nodeCount = 48602 ;
    elementCount = 48600 ;
    maxNodePElement = 4 ;
    coordDim = 2 ;
variables:
    double nodeCoords(nodeCount, coordDim) ;
        nodeCoords:units = "degrees" ;
    int elementConn(elementCount, maxNodePElement) ;
        elementConn:long_name = "Node indices that define the element connectivity" ;
        elementConn:_FillValue = -1 ;
    int numElementConn(elementCount) ;
        numElementConn:long_name = "Number of nodes per element" ;
    double centerCoords(elementCount, coordDim) ;
        centerCoords:units = "degrees" ;
    double elementArea(elementCount) ;
        elementArea:units = "radians^2" ;
        elementArea:long_name = "area weights" ;
    int elementMask(elementCount) ;

// global attributes:
        :gridType = "unstructured mesh" ;
        :version = "0.9" ;
        :inputFile = "/glade/work/aherring/grids/SCRIP_files/ne30pg3_scrip_170611.nc" ;
        :timeGenerated = "Mon Jan 18 17:52:10 2021" ;
}
uturuncoglu commented 9 months ago

I am getting following error after pull your changes,

/work/noaa/nems/tufuk/COASTAL/ufs-coastal/CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90(1985): error #6311: A block was not terminated before an ELSEIF statement was reached.
    else if (ndims == 3) then
----^
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90(2005): error #6321: An unterminated block exists.
    else if (ndims == 4) then

I think there is a missing end if in the previous if statement. The if (sdat%mainproc) then needs to be closed before pio_initdecomp. It is also same for ndims == 4 case but I am not sure about the logic in there. When I run the case, I got following error,

112: forrtl: error (76): Abort trap signal
112: Image              PC                Routine            Line        Source             
112: fv3_coastal.exe    0000000001EEAF5B  Unknown               Unknown  Unknown
112: libpthread-2.17.s  00002B1D309D95D0  Unknown               Unknown  Unknown
112: libc-2.17.so       00002B1D30C1C2C7  gsignal               Unknown  Unknown
112: libc-2.17.so       00002B1D30C1D9B8  abort                 Unknown  Unknown
112: libc-2.17.so       00002B1D30C5EE17  Unknown               Unknown  Unknown
112: libc-2.17.so       00002B1D30C68E16  Unknown               Unknown  Unknown
112: libc-2.17.so       00002B1D30C6B32C  __libc_malloc         Unknown  Unknown
112: libc-2.17.so       00002B1D30C6D89C  posix_memalign        Unknown  Unknown
112: libucs.so.0.0.0    00002B1E728E7098  ucs_posix_memalig     Unknown  Unknown
112: libucs.so.0.0.0    00002B1E728E93AD  ucs_rcache_get        Unknown  Unknown
112: libuct_ib.so.0.0.  00002B1E72E655C4  Unknown               Unknown  Unknown
112: libucp.so.0.0.0    00002B1E723DA490  ucp_mem_rereg_mds     Unknown  Unknown
112: libucp.so.0.0.0    00002B1E723DDC84  ucp_request_memor     Unknown  Unknown
112: libucp.so.0.0.0    00002B1E72404677  Unknown               Unknown  Unknown
112: libucp.so.0.0.0    00002B1E7240490B  ucp_rndv_progress     Unknown  Unknown
112: libucp.so.0.0.0    00002B1E72409029  ucp_rndv_receive      Unknown  Unknown
112: libucp.so.0.0.0    00002B1E7241B518  ucp_tag_recv_nbx      Unknown  Unknown
112: libucp.so.0.0.0    00002B1E7241B7A3  ucp_tag_recv_nb       Unknown  Unknown
112: libmlx-fi.so       00002B1E7216ADBB  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F599320  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F2D830D  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F2D91D6  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F3272C5  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F3067EC  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F4495B0  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F2D9A05  PMPI_Alltoallw        Unknown  Unknown
112: libpioc.so         00002B1D2D19AD5B  pio_swapm             Unknown  Unknown
112: libpioc.so         00002B1D2D1A0E04  compute_counts        Unknown  Unknown
112: libpioc.so         00002B1D2D19EAE4  box_rearrange_cre     Unknown  Unknown
112: libpioc.so         00002B1D2D198C69  PIOc_InitDecomp       Unknown  Unknown
112: libpiof.so         00002B1D2CF28016  piolib_mod_mp_pio     Unknown  Unknown
112: libpiof.so         00002B1D2CF25F3E  piolib_mod_mp_pio     Unknown  Unknown
112: libpiof.so         00002B1D2CF282E7  piolib_mod_mp_pio     Unknown  Unknown
112: fv3_coastal.exe    0000000001A1EDDC  dshr_strdata_mod_        1975  dshr_strdata_mod.F90
112: fv3_coastal.exe    0000000001A291B6  dshr_strdata_mod_        1509  dshr_strdata_mod.F90
112: fv3_coastal.exe    0000000001A25823  dshr_strdata_mod_        1368  dshr_strdata_mod.F90
112: fv3_coastal.exe    0000000001A202B6  dshr_strdata_mod_         934  dshr_strdata_mod.F90
112: fv3_coastal.exe    00000000019CEC2A  cdeps_datm_comp_m         720  atm_comp_nuopc.F90
112: fv3_coastal.exe    00000000019D1D59  cdeps_datm_comp_m         506  atm_comp_nuopc.F90

Let me also try to add extra dimension to the file and see what happens.

jedwards4b commented 9 months ago

@uturuncoglu I fixed that branch for you to try again.

uturuncoglu commented 9 months ago

@jedwards4b still getting same error. Let me try with fresh copy since the last one had conflicts and I fixed them but maybe after getting last updates, it did not work as expected.

uturuncoglu commented 9 months ago

@jedwards4b Okay. Tried again with fresh copy and getting same error.

uturuncoglu commented 9 months ago

@klindsay28 Okay. I restructure the netcdf file like following. In this case, I also added node_y a dimension with size of 1.

netcdf Florence_HWRF_HRRR_HSOFS_shifted_v3 {
dimensions:
    time = UNLIMITED ; // (522 currently)
    node_x = 1813443 ;
    node_y = 1 ;
    element = 3564104 ;
    noel = 3 ;
variables:
    float P(time, node_x, node_y) ;
        P:field = "V, scalar, series" ;
        P:long_name = "Pressure Reduced to MSL" ;
        P:missing_value = 9.96921e+36f ;
        P:units = "Pa" ;
    double latitude(node_x, node_y) ;
        latitude:field = "lat, scalar, series" ;
        latitude:long_name = "latitude" ;
        latitude:units = "degree_north" ;
    double longitude(node_x, node_y) ;
        longitude:field = "lon, scalar, series" ;
        longitude:long_name = "longitude" ;
        longitude:units = "degree_east" ;
    double time(time) ;
        time:standard_name = "time" ;
        time:long_name = "julian day (UT)" ;
        time:units = "days since 1990-01-01 00:00:00" ;
        time:calendar = "gregorian" ;
        time:axis = "T" ;
    int tri(element, noel) ;
    float uwnd(time, node_x, node_y) ;
        uwnd:field = "U, scalar, series" ;
        uwnd:long_name = "eastward_wind" ;
        uwnd:missing_value = 9.96921e+36f ;
        uwnd:units = "m s-1" ;
    float vwnd(time, node_x, node_y) ;
        vwnd:field = "V, scalar, series" ;
        vwnd:long_name = "northward_wind" ;
        vwnd:missing_value = 9.96921e+36f ;
        vwnd:units = "m s-1" ;

but I am getting following error trace (this is the original CDEPS without @jedwards4b mods),

 40: fv3_coastal.exe    0000000001A2C9C2  dshr_strdata_mod_        1993  dshr_strdata_mod.F90
 40: fv3_coastal.exe    0000000001A26F96  dshr_strdata_mod_        1509  dshr_strdata_mod.F90
 40: fv3_coastal.exe    0000000001A23603  dshr_strdata_mod_        1368  dshr_strdata_mod.F90
 40: fv3_coastal.exe    0000000001A1E096  dshr_strdata_mod_         934  dshr_strdata_mod.F90
 40: fv3_coastal.exe    00000000019CEBD5  cdeps_datm_comp_m         709  atm_comp_nuopc.F90
 40: fv3_coastal.exe    00000000019D1C79  cdeps_datm_comp_m         497  atm_comp_nuopc.F90

and still getting log like,

(shr_strdata_set_stream_iodesc) setting iodesc for : uwnd with dimlens(1), dimlens(2) =        1   1813443   variable as time dimension time

So, maybe this modification is not enough. @klindsay28 I could not find the run directory in /glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD. If you could, can you point me the data file. Then, I could try to mimic the same in my file and test again.

klindsay28 commented 9 months ago

The run directory is on cheyenne/casper at /glade/scratch/oleson/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD/run. There are multiple data files in the directory /glade/scratch/hannay/archive/f.cam6_3_112.FLT1850.ne30.landspinup.001/cpl/hist

uturuncoglu commented 9 months ago

@klindsay28 I could not access those directories (probably permission issue). Since, I know the case, maybe I could try to find the original data under CESMDATA.

uturuncoglu commented 9 months ago

@klindsay28 Okay. I could aces to the data: /glade/scratch/hannay/archive/f.cam6_3_112.FLT1850.ne30.landspinup.001/cpl/hist/f.cam6_3_112.FLT1850.ne30.landspinup.001.cpl.hx.atm.1h.inst.0027-05-28-03600.nc In this case, lat, lon variables have also time dimension. Other than that it seems very similar to my file. Maybe that is the trick.

klindsay28 commented 9 months ago

@uturuncoglu , I made a world-readable copy of CASEROOT at /glade/scratch/klindsay/for_turuncu/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD. As is usual, the CaseDocs subdirectory has component namelists and CDEPS stream xml files. There is a single source file in SourceMods/src.datm, but the diff (see below) seems to be related to the shortwave computation time variable, not how data is processed.

diff /glade/work/oleson/cesm2_3_alpha14a/components/cdeps/datm/atm_comp_nuopc.F90 SourceMods/src.datm/
875,876c875,880
<        if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
<           nextsw_cday = julday + 2*dtime/shr_const_cday
---
> !KO       if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
> !KO          nextsw_cday = julday + 2*dtime/shr_const_cday
> !KO
>        if (mod(tod,delta_radsw) == 0 .and. stepno > 0) then
>           nextsw_cday = julday + 1*dtime/shr_const_cday
> !KO
uturuncoglu commented 9 months ago

@klindsay28 Thanks for all your help. I am trying to restructure my file at this point and then I'll try again. Keep you updated.

klindsay28 commented 9 months ago

@uturuncoglu, FYI, the experiment that I'm pointing you to is using cdeps1.0.14. I don't follow CDEPS development closely, so I don't know if updates to CDEPS between that tag and the most recent would interact with CDEPS' ability to process 1D-like data.

jedwards4b commented 9 months ago

@uturuncoglu I haven't been able to reproduce your error in my branch - can you supply a reproducer please?

uturuncoglu commented 9 months ago

@jedwards4b I am using CDEPS under UFS with additional data mode and Orion. Let me try to create reproducer for you, it could take little bit time.

jedwards4b commented 8 months ago

@uturuncoglu has this issue been resolved?

uturuncoglu commented 8 months ago

@jedwards4b I could not continue to that work but I'll do. Let's keep this open if you don't mind. I have very similar mesh file to the case that work on CESM side but I my case it is not working. I have some doubt that our data file has some issue since it has more elements than the nodes which seems weird to me. I am discussing with UFS coastal developer to fix the data file then I'll create mesh again and test it. It will probably take time.

jedwards4b commented 6 months ago

@uturuncoglu Were you ever able to get this working? I've had another request for this feature and thought I would ask before starting over.

uturuncoglu commented 6 months ago

@jedwards4b first off all sorry for late response. last couple of weeks were crazy but I hope I'll be fine after AMS. Anyway, I could cot continue to testing this. It was failing in my case. If you have time and want to implement something that is fine. I could test it with one of the case from UFS Coastal to see what happens.