Data access criteria for different types of datasets

[havardf]

• Increasing append style datasets.
• Latest reference_time, e.g a dataset where content is completely replaced with each new version.
• Static, separate datasets. Each new production run results in a separate dataset, semantically connected through search metadata.

[havardf]

@mortenwh I need to add something to the criterias, see description in this issue. They are however related to the use-cases so I was not sure how this should be done. Suggestions?

[havardf]

Also, what do you think about the dataset types? Any comments, or perhaps there are more categories of datasets?

[mortenwh]

I'm not sure what you mean... If it is criteria related to the data access, I guess they could be added to the existing list? Then we can reorganize later..

Regarding the netcdf files - the choice of static vs dynamic/appendable depends on the data, doesn't it? Both are quite straightforward. Is this what you mean by dataset types?

[havardf]

As for the first, yup, I agree.

Its not really netcdf files anymore, its datasets beeing access through a url, either over DAP or Zarr. And for both protocols, you first access metadata and then stream various data variables. If you e.g. change that dataset, then the clients who have just accessed the metadata could potentially get access to erroneous data.

[havardf]

As for "latest" case, object stores has no symlink functionality, so its not straight forward how this should be solved. So there are issues with the various types of datasets.

[havardf]

Dont need a separate issues for this, so will just close this one.