search

rom-py / rompy

Relocatable Ocean Modelling in PYthon (rompy) combines templated cookie-cutter model configuration with various xarray extensions to assist in the setup and evaluation of coastal ocean model
https://rom-py.github.io/rompy/
BSD 3-Clause "New" or "Revised" License
2 stars 9 forks source link

Plugin approaches for extending DataGrid source DATA_SOURCE_TYPES #33

Open benjaminleighton opened 10 months ago

benjaminleighton commented 10 months ago

I've been implementing a prospective new SourceBase called SourceFiles that wraps xarray open_mfdataset. To do this I need to modify the static list of allowed types in DATA_SOURCE_TYPES. Here are some ideas https://chat.openai.com/share/1f48f3ae-e2e9-41c7-b9ca-494a5d469f26 on a plugin based approach that sacrifices some of the Pydantic static type checking capabilities for plugin types and adds a little bit of complexity. What do you think?

rafa-guedes commented 10 months ago

@benjaminleighton this idea is definitely worth exploring

pbranson commented 10 months ago

I would like to +1 this, the current main includes a dependency for from oceanum.datamesh import Connector We need to rationalise the type checking to the extent that the data/init.py doesnt import non-generic sources.

@rafa-guedes , @tomdurrant Could perhaps the SourceDatamesh be abstracted via intake so that this additional class is not necessary?

I think the general design principle should be that all of rompy's data interfaces expect an xr.Dataset or xr.DataArray object to which an Xarray accessor exposes the munging code to go from Xarray to model specific file formats.

rafa-guedes commented 10 months ago

@rafa-guedes , @tomdurrant Could perhaps the SourceDatamesh be abstracted via intake so that this additional class is not necessary?

@pbranson yes this is a good idea - will put it as a new issue