search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
18 stars 25 forks source link

Log sync operations #437

Open aidanheerdegen opened 2 months ago

aidanheerdegen commented 2 months ago

With the new sync features it would be good to log what files were sync'ed and where. This is a good idea so users can come back to an experiment and discover what was done, but also from the point of view of keeping track of output and restart files.

The location of restarts is important as this makes it possible to add this information to a database to make it easier for users to create experiments that are based on previous experiments.

aidanheerdegen commented 2 months ago

One difficulty is how we save this information.

It would be good to be git commit this log to the control experiment git repo. That is tricky: do we put up with random commits? The path to the committed file might reference an archive path that doesn't exist in the future when files in scratch are purged.