It would be good to have some machine-readable log of collate operations so there is a track of how files that were output from the model have been transformed.
For example restart files that are collated break the link between the restart.yaml manifest and the restarts that were used in the model run. If that link could be re-established then collated restarts could be reliably used in place of the original tiled restarts.
For a collate log to be used in this manner it would have to include md5 hashes for the collated and uncollated files and a relationship between them.
It would be good to have some machine-readable log of collate operations so there is a track of how files that were output from the model have been transformed.
For example restart files that are collated break the link between the
restart.yamlmanifest and the restarts that were used in the model run. If that link could be re-established then collated restarts could be reliably used in place of the original tiled restarts.For a collate log to be used in this manner it would have to include
md5hashes for the collated and uncollated files and a relationship between them.How to store this information in the commit history of the experiment repo is a problem that is shared with https://github.com/payu-org/payu/issues/437