payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
21 stars 27 forks source link

Changed input paths are not properly checked and added/removed from input manifest #357

Open aidanheerdegen opened 1 year ago

aidanheerdegen commented 1 year ago

When an input is a full path to a file the logic for updating and/or removing file paths that have changed from the path in the input manifest is not correct.

For the specific case of this MOM6 config

https://github.com/COSIMA/MOM6-CICE6/blob/main/config.yaml

When the all the local filepaths are unchanged but fullpaths are updated, but the original files still exist in the original location, then a change is not detected.

By the time check_manifests is called all the existing files have been happily linked in the model setup and removed from existing_filepaths when add_filepath is called here

https://github.com/payu-org/payu/blob/105fdc0930a43ff0802c9091ebbc6d8c85d6fa40/payu/models/model.py#L231

Need to add some logic to check if the fullpath has changed in the manifest.