search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
21 stars 27 forks source link

Usecase when parent directory is also a git repository #478

Closed ashjbarnes closed 3 months ago

ashjbarnes commented 3 months ago

I've got all of my experiments in sub folders of my main github repository here. However, when I try to run payu run -f over lots of experiments at once, they sporadically crash because

fatal: Unable to create '/home/149/ab8992/topographic-NIWs/.git/index.lock': File exists.

@aekiss suggested that with newer iterations of Payu there's a better way to do these perturbation experiments with payu branch. I just wanted to highlight this use case though in case a nested git repo structure would cause any problems here too

aidanheerdegen commented 3 months ago

Your link 404s, there is trailing character that is incorrect.

FYI for others the link is https://github.com/ashjbarnes/topographic-NIWs

Wow! Generating and running payu jobs from within a jupyter notebook is a heck of scene, and not a use case I'd ever considered.

Anytime automated git commands throw an error like that you can add

runlog: false

to your config.yaml and payu won't try and do anything git-related, but you also won't get nice automated git logging of configs and manifests.

As for the new methods, yeah payu now supports branching as a first-class operation, and auto-generates unique experiment IDs to enable working with multiple branches from the same git repo/directory/config directory.

If you want to find out more you, and anyone else, can sign up for the training day at our upcoming workshop

https://www.access-nri.org.au/event/access-training-day-2-september-2024/

It's on campus at ANU, so should be easy enough to come along. I'm running a session about using payu, particularly highlighting the new branching features.

ashjbarnes commented 3 months ago

Awesome thanks for a very comprehensive reply Aidan!! I won't be around in September unfortunately but I'd love to learn about this for my next project. These runs are re-hashed for a paper revision - I didn't have the time or energy to re-write the code with more up to date Payu methods but would like to next time for future projects.

I'd done a similar thing for my tasman-tides project, but I think not turning off the runlog didn't cause issues because I was only running one of 12 experiments at at time, not 100 perturbation experiments at once that would cause Payu to all try and write to the same .index file at once

aidanheerdegen commented 3 months ago

I won't be around in September unfortunately but I'd love to learn about this for my next project.

That's a shame. We'll have some training material associated with it that should be available after the workshop.

Has your question been answered? If so are you happy for me to close this issue?

ashjbarnes commented 3 months ago

Yes, thanks again!