search

rom-py / rompy

Relocatable Ocean Modelling in PYthon (rompy) combines templated cookie-cutter model configuration with various xarray extensions to assist in the setup and evaluation of coastal ocean model
https://rom-py.github.io/rompy/
BSD 3-Clause "New" or "Revised" License
3 stars 9 forks source link

Symlinks instead of copying files #82

Closed benjaminleighton closed 3 months ago

benjaminleighton commented 6 months ago

In some use cases it makes sense not to copy files into a simulation workspace directory. It maybe that forcing files are shared across many simulation variants, e.g differing parameters or where another forcing files is changed. In these cases it is inefficient to refilter and rewrite files many times. Rather we should have a mechanism that can point to existing files using symlinks.

This is challenging in our current implementation because https://github.com/rom-py/rompy/blob/45fb70a713bf69f0c2e55862c6c3af4ead64fbf5/rompy/core/data.py#L484 writes a netcdf file.

I might be misunderstanding how we use pydantic here and whether we can generate a partial configuration but I have a feeling that some of the forcing fields across swan and schism are required and can't be omitted. Even if this was the case we'd want to be able to symlink back to a canonical set of forcing.

For the timebeing I'm going to work around this by generating some, empty, placeholder datasets and implementing a "symlink" pass after workspace creation.

benjaminleighton commented 5 months ago

working on this in symlinks_and_datablob_options_schism