search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
18 stars 25 forks source link

Add check for UUIDs when checking for existence of archives #425

Closed jo-basevi closed 2 months ago

jo-basevi commented 3 months ago

In #420, there’s a bug with using branching logic, where an existing experiment is not detected if its archive does not exist. So if an experiment was created then deleted on a branch in a control directory that has a corresponding legacy archive, subsequent payu setup/run calls, it’ll use the legacy archive.

This PR so far adds an additional check when checking for the existence of archives, also check for metadata with a matching UUID.

This isn’t a perfect check as it requires payu setup/payu run/ payu checkout to be run on the legacy experiment which will then subsequently create a UUID and metadata in the legacy archive if it doesn’t already exist. Otherwise, if there is no metadata associated with the legacy archive and a new UUID has not been generated, it’s difficult to know whether to use a legacy archive or not.

Closes #420

TODO:

jo-basevi commented 3 months ago

I've simplified the logic in the check to only fail when there exists a UUID in the archive metadata but it is not equal to the UUID in the control directory metadata.

So it'll still work for legacy archives with no UUIDs or metadata. So the recommendation when running a legacy experiment but want to run a related experiment on a different branch in the same control directory is to firstly run: payu setup on the legacy experiment branch to ensure a UUID is generated, and metadata is copied across to the legacy archive.

aidanheerdegen commented 2 months ago

Doesn't this interact rather heavily with https://github.com/payu-org/payu/pull/427? I'm seeing code I think was modified in the other PR.

jo-basevi commented 2 months ago

Yes, I'll need rebase the changes onto main and then run a couple tests