search

schism-dev / schism

Semi-implicit Cross-scale Hydroscience Integrated System Model (SCHISM)
http://ccrm.vims.edu/schismweb/
Apache License 2.0
87 stars 86 forks source link

Update ugrid conventions #114

Open veenstrajelmer opened 10 months ago

veenstrajelmer commented 10 months ago

Recently I recieved SCHISM model output, converted to ugrid format. We could not directly read this data with our generic ugrid reader, since there were some attributes missing or incorrect. I have written a preprocess function with which most things seem to work. Could you consider adding the adjustments to your ugrid writer/converter? I am not sure if this issue is the most convenient way to communicate this, but it might be a start.

import xugrid as xu

file_nc = "conv_schism_raw_2.nc"

def preprocess_schism(ds):
    # convert topology_dimension attribute: str to int
    topodim = int(ds["schism_mesh"].attrs["topology_dimension"])
    mesh_attrs = {"topology_dimension":topodim,
                  "name":"SCHISM_hgrid"}
    ds["schism_mesh"] = ds["schism_mesh"].assign_attrs(mesh_attrs)
    # add necessary attributes for face_node_connectivity
    fnc_attrs = {"_FillValue":-1, "start_index":1}
    ds["SCHISM_hgrid_face_nodes"] = ds["SCHISM_hgrid_face_nodes"].assign_attrs(fnc_attrs)
    # set x/y coords, seems not necessary
    # ds = ds.set_coords(["SCHISM_hgrid_node_x","SCHISM_hgrid_node_y"])
    datavar_attrs = {"mesh":"schism_mesh"}
    for varn in ds.data_vars:
        ds[varn] = ds[varn].assign_attrs(datavar_attrs)
    return ds

uds = xu.open_mfdataset(file_nc, preprocess=preprocess_schism)
uds.salinity.isel(time=-1, nSCHISM_vgrid_layers=0).ugrid.plot()
josephzhang8 commented 10 months ago

Thx @veenstrajelmer. Did you do this for OLDIO or new scribed IO? For the latter we have UGRID part added, so we can easily add those changes. @platipodium @water-e : can u please chime in? Thx,

veenstrajelmer commented 10 months ago

Thanks for your reply. I realized after the creation of this issue that it does not have enough context. At the moment i cannot answer your questions, but I will get back to you after a meeting tomorrow.

veenstrajelmer commented 10 months ago

Hi @josephzhang8. After meeting with @BenjaminJacob86 I realized I used scribedio schism output that was already partly processed. In the below function I have defined the preprocessing that is needed in order for our ugrid reader to read and visualize the raw schism scribedio output. This is a bit more than I communicated before, but I hope this provides useful information to enrich the schism output dataset.

import xarray as xr
import numpy as np
from netCDF4 import default_fillvals
import xugrid as xu # pip install xugrid, adds ugrid and grid accessors to xarray datasets

def preprocess_schism_scribedio(ds):
    """
    This preprocessing function describes the minimal changes that would have to be made 
    in the SCHISM output in order for it to be read via ugrid conventions with xugrid.
    It is probably not a complete overview.
    """
    # set varnames
    gridname = "SCHISM_hgrid"
    fnc_varn = f"{gridname}_face_nodes"
    enc_varn = f"{gridname}_edge_nodes"
    node_x = f"{gridname}_node_x"
    node_y = f"{gridname}_node_y"

    # set topology attributes to empty variable
    topo_attrs = {"cf_role": "mesh_topology",
                 "topology_dimension": 2, # has to be int, not str
                 "node_coordinates": f"{node_x} {node_y}",
                 "face_node_connectivity": fnc_varn,
                 "edge_node_connectivity": enc_varn,
                 }
    ds[gridname] = xr.DataArray(np.array(default_fillvals["i4"], dtype=np.int32), attrs=topo_attrs)

    # assign necessary attributes to connectivity variables
    fnc_attrs = {"_FillValue":-1, "start_index":1}
    ds[fnc_varn] = ds[fnc_varn].assign_attrs(fnc_attrs)
    ds[enc_varn] = ds[enc_varn].assign_attrs(fnc_attrs)

    # set node_x/node_y as coordinate variables instead of data_vars
    ds = ds.set_coords([node_x,node_y])

    # to prevent xugrid UserWarning, but this is hardcoded and it should be different for lat/lon models.
    # "UserWarning: No standard_name of ('projection_x_coordinate', 'longitude', 'projection_y_coordinate', 'latitude') in
    # ['SCHISM_hgrid_node_x', 'SCHISM_hgrid_node_y']. Using SCHISM_hgrid_node_x and SCHISM_hgrid_node_y as projected x and y coordinates."
    projected = True
    if projected:
        ds[node_x] = ds[node_x].assign_attrs({"standard_name":"projection_x_coordinate"})
        ds[node_y] = ds[node_y].assign_attrs({"standard_name":"projection_y_coordinate"})
    else:
        ds[node_x] = ds[node_x].assign_attrs({"standard_name":"longitude"})
        ds[node_y] = ds[node_y].assign_attrs({"standard_name":"latitude"})

    # add variable with coordinate system, optional but convenient for loading into QGIS and other tools
    # not yet properly read/updated by xugrid: https://github.com/Deltares/xugrid/issues/42
    if projected:
        grid_mapping_name = 'Unknown projected'
        crs_varn = 'projected_coordinate_system'
        crs_num = 25832 #UTM Zone 32N from communication with BJ
    else:
        grid_mapping_name = 'latitude_longitude'
        crs_varn = 'wgs84'
        crs_num = 4326
    crs_str = f'EPSG:{crs_num}'
    crs_attrs = {'epsg': crs_num, # epsg or EPSG_code are required for correct interpretation by QGIS
                  'EPSG_code': crs_str, # epsg or EPSG_code are required for correct interpretation by QGIS
                  'grid_mapping_name': grid_mapping_name,
                  }
    ds[crs_varn] = xr.DataArray(np.array(default_fillvals['i4'],dtype=int),dims=(),attrs=crs_attrs)

    # mesh attribute is required for d-ecoimpact
    # valueable other attrs are "location" (node/face/edge), 
    # "standard_name", "long_name", "units", "grid_mapping"
    for varn in ds.data_vars:
        ds[varn] = ds[varn].assign_attrs({'mesh': gridname})

    # time requires "units" attribute to be converted by xarray and other tools
    # refdate taken from params.nml
    ds['time'] = ds['time'].assign_attrs({"units":"seconds since 2017-01-02 00:00:00"})
    ds = xr.decode_cf(ds)

    #TODO: set _FillValue attribute for data_vars, test dataset did not seem to have nans

    return ds

file_nc_pat = r"c:\Users\veenstra\Downloads\example\1_raw_outputs\*_1.nc"

# open the file with xarray, add ugrid conventions with preprocessing function and convert to UgridDataset
# in case of a pre-merged file this oneliner also works: `uds = xu.open_dataset(ds, preprocess=preprocess_schism_scribedio)`
# that would also work if the topology variables would be present in all datasets
ds = xr.open_mfdataset(file_nc_pat)
ds = preprocess_schism_scribedio(ds)
uds = xu.UgridDataset(ds)

# test with a ugrid plot to show the salinity variable was properly connected to the ugrid topology
uds.salinity.isel(time=-1, nSCHISM_vgrid_layers=0).ugrid.plot()

It mostly comes down to creating an empty variable containing the mesh topology and adding some attributes. In the comments some other suggestions or the reasoning behind it is given.

Other suggestions would be:

platipodium commented 10 months ago

Thanks @veenstrajelmer for following up on this. We @josephzhang8 should in any case add the relevant (empty variable) metadata to any output.

@veenstrajelmer Could you attach a netcdf -h header dump of both the original and your desired version of the test file you were using?

We had a discussion previously on your suggestion

Currently the salinity file can only be processed properly by using information from the 2d file. Instead add topology data to all separate datasets, so these files are all self-contained.

Our opinions differ on this issue, as this would require more storage space. @josephzhang8 would you reconsider?

josephzhang8 commented 10 months ago

Regarding the comments:

  1. Currently the salinity file can only be processed properly by using information from the 2d file. Instead add topology data to all separate datasets, so these files are all self-contained.

  2. Adding this info in all 3D outputs would increase the size by a non-trivial amount for large meshes (e.g. STOFS3D), so it was decided to leave it out. Is there a way to add a note in 3D outputs to point to 2D?

  3. add attributes to all variables so it is clear what it contains

    In principle this is doable, but since we support A LOT OF outputs, we decided instead to make the output file names as explicit as possible More importantly, this save some communication cost.

veenstrajelmer commented 10 months ago

Thanks for the responses.

I recommend to not add the topology variable if the actual topology is not added, it will result in a corrupt file according to the ugrid conventions since the variables where the topology attributes point to are not present. The current setup is in that case better, so I would recommend to leave that as is.

However, enriching the dataset with the topology variables inside is still beneficial. I do not have netcdf/ncdump installed outside of python, but used this python workaround. I have pasted the entire output of both files below.

Original schism scribedio output (out2d_1.nc):

NetCDF Global Attributes:
NetCDF dimension information:
    Name: nSCHISM_hgrid_node
        size: 515582
        WARNING: nSCHISM_hgrid_node does not contain variable attributes
    Name: nSCHISM_hgrid_face
        size: 1019883
        WARNING: nSCHISM_hgrid_face does not contain variable attributes
    Name: nSCHISM_hgrid_edge
        size: 1535511
        WARNING: nSCHISM_hgrid_edge does not contain variable attributes
    Name: nMaxSCHISM_hgrid_face_nodes
        size: 4
        WARNING: nMaxSCHISM_hgrid_face_nodes does not contain variable attributes
    Name: nSCHISM_vgrid_layers
        size: 21
        WARNING: nSCHISM_vgrid_layers does not contain variable attributes
    Name: one
        size: 1
        WARNING: one does not contain variable attributes
    Name: two
        size: 2
        WARNING: two does not contain variable attributes
    Name: time
        size: 24
        type: dtype('float64')
        i23d: 0
NetCDF variable information:
    Name: minimum_depth
        dimensions: ('one',)
        size: 1
        type: dtype('float64')
    Name: SCHISM_hgrid_node_x
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('float64')
    Name: SCHISM_hgrid_node_y
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('float64')
    Name: depth
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('float32')
    Name: bottom_index_node
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('int32')
    Name: SCHISM_hgrid_face_nodes
        dimensions: ('nSCHISM_hgrid_face', 'nMaxSCHISM_hgrid_face_nodes')
        size: 4079532
        type: dtype('int32')
    Name: SCHISM_hgrid_edge_nodes
        dimensions: ('nSCHISM_hgrid_edge', 'two')
        size: 3071022
        type: dtype('int32')
    Name: dryFlagNode
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: elevation
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: bottomStressX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: bottomStressY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: windSpeedX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: windSpeedY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: depthAverageVelX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: depthAverageVelY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sigWaveHeight
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: meanWavePeriod
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: zeroDowncrossPeriod
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: meanDirSpreading
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: peakPeriod
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: dominantDirection
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: peakSpreading
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: orbitalVelocity
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: rmsOrbitalVelocity
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: bottomExcursionPerio
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: frictionalVelocity
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: rougnessLength
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: waveEnergyDirX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: waveEnergyDirY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_1
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_1
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_2
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_2
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_3
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_3
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_4
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_4
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_5
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_5
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_6
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_6
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_7
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_7
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadX_8
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedloadY_8
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_1
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_2
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_3
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_4
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_5
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_6
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_7
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: sedBedFraction_8
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        i23d: 1
    Name: dryFlagElement
        dimensions: ('time', 'nSCHISM_hgrid_face')
        size: 24477192
        type: dtype('float32')
        i23d: 4
    Name: dryFlagSide
        dimensions: ('time', 'nSCHISM_hgrid_edge')
        size: 36852264
        type: dtype('float32')
        i23d: 7

Enriched file:

NetCDF Global Attributes:
    Conventions: 'CF-1.9 UGRID-1.0'
NetCDF dimension information:
    Name: nSCHISM_hgrid_face
        size: 1019883
        type: dtype('int64')
    Name: nMaxSCHISM_hgrid_face_nodes
        size: 4
        WARNING: nMaxSCHISM_hgrid_face_nodes does not contain variable attributes
    Name: nSCHISM_hgrid_edge
        size: 1535511
        type: dtype('int64')
    Name: two
        size: 2
        WARNING: two does not contain variable attributes
    Name: nSCHISM_hgrid_node
        size: 515582
        type: dtype('int64')
    Name: time
        size: 24
        type: dtype('float64')
        _FillValue: nan
        i23d: 0
        units: 'seconds since 2017-01-02'
        calendar: 'proleptic_gregorian'
    Name: one
        size: 1
        WARNING: one does not contain variable attributes
NetCDF variable information:
    Name: SCHISM_hgrid
        dimensions: ()
        size: 1
        type: dtype('int32')
        cf_role: 'mesh_topology'
        long_name: 'Topology data of 2D mesh'
        topology_dimension: 2
        node_dimension: 'nSCHISM_hgrid_node'
        edge_dimension: 'nSCHISM_hgrid_edge'
        face_dimension: 'nSCHISM_hgrid_face'
        max_face_nodes_dimension: 'SCHISM_hgrid_nMax_face_nodes'
        edge_node_connectivity: 'SCHISM_hgrid_edge_nodes'
        face_node_connectivity: 'SCHISM_hgrid_face_nodes'
        node_coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
        name: 'SCHISM_hgrid'
    Name: SCHISM_hgrid_face_nodes
        dimensions: ('nSCHISM_hgrid_face', 'nMaxSCHISM_hgrid_face_nodes')
        size: 4079532
        type: dtype('int32')
        _FillValue: -1
        start_index: 1
        mesh: 'SCHISM_hgrid'
    Name: SCHISM_hgrid_edge_nodes
        dimensions: ('nSCHISM_hgrid_edge', 'two')
        size: 3071022
        type: dtype('int32')
        _FillValue: -1
        start_index: 1
        mesh: 'SCHISM_hgrid'
    Name: SCHISM_hgrid_node_x
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('float64')
        _FillValue: nan
        standard_name: 'projection_x_coordinate'
    Name: SCHISM_hgrid_node_y
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('float64')
        _FillValue: nan
        standard_name: 'projection_y_coordinate'
    Name: minimum_depth
        dimensions: ('one',)
        size: 1
        type: dtype('float64')
        _FillValue: nan
        mesh: 'SCHISM_hgrid'
    Name: depth
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('float32')
        _FillValue: nan
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: bottom_index_node
        dimensions: ('nSCHISM_hgrid_node',)
        size: 515582
        type: dtype('int32')
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: dryFlagNode
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: elevation
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: bottomStressX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: bottomStressY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: windSpeedX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: windSpeedY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: depthAverageVelX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: depthAverageVelY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sigWaveHeight
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: meanWavePeriod
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: zeroDowncrossPeriod
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: meanDirSpreading
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: peakPeriod
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: dominantDirection
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: peakSpreading
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: orbitalVelocity
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: rmsOrbitalVelocity
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: bottomExcursionPerio
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: frictionalVelocity
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: rougnessLength
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: waveEnergyDirX
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: waveEnergyDirY
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_1
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_1
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_2
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_2
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_3
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_3
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_4
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_4
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_5
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_5
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_6
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_6
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_7
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_7
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadX_8
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedloadY_8
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_1
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_2
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_3
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_4
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_5
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_6
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_7
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: sedBedFraction_8
        dimensions: ('time', 'nSCHISM_hgrid_node')
        size: 12373968
        type: dtype('float32')
        _FillValue: nan
        i23d: 1
        mesh: 'SCHISM_hgrid'
        coordinates: 'SCHISM_hgrid_node_x SCHISM_hgrid_node_y'
    Name: dryFlagElement
        dimensions: ('time', 'nSCHISM_hgrid_face')
        size: 24477192
        type: dtype('float32')
        _FillValue: nan
        i23d: 4
        mesh: 'SCHISM_hgrid'
    Name: dryFlagSide
        dimensions: ('time', 'nSCHISM_hgrid_edge')
        size: 36852264
        type: dtype('float32')
        _FillValue: nan
        i23d: 7
        mesh: 'SCHISM_hgrid'
    Name: crs
        dimensions: ()
        size: 1
        type: dtype('int32')
        epsg: 25832
        EPSG_code: 'EPSG:25832'
        mesh: 'SCHISM_hgrid'
josephzhang8 commented 10 months ago

Yes I believe there are very simple tools to add this info, e.g. NCL/NCO etc.

veenstrajelmer commented 10 months ago

Sure, but I guess the output is comparable to what I send. Is there any additional information you need from my side? Do you think it would be a valuable update to add the ugrid metadata to schism output?

josephzhang8 commented 10 months ago

I'll let @platipodium add the missing meta in SCHISM code. As I explained, we need to be frugal about adding mesh info in all outputs.

veenstrajelmer commented 10 months ago

Yes, understandable, I am not sure if my response was clear. But I think in that case the current setup is good. Plain 3D variables in separate files, that are dependent on the out2d file in the same folder that contains all the connectivity/coordinates (and in the future also the topology metadata). Anyway, happy to be able to contribute something.

josephzhang8 commented 10 months ago

Great; thx @veenstrajelmer

@platipodium : can u plz add the missing metadata. Thx

veenstrajelmer commented 3 months ago

@platipodium Was this enhancement by any chance already implemented? I am asking this since we are in the continuing process of simplifying our processing tools and specifically making https://github.com/Deltares/D-EcoImpact support different model outputs. This SCHISM enhancement would be very helpful to make this process more transparent and simplify workflows for all D-EcoImpact users that want to work with SCHISM output.

platipodium commented 3 months ago

@platipodium Was this enhancement by any chance already implemented? I am asking this since we are in

Sorry @veenstrajelmer, this has not yet been added. Feel free to do so :=)