search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
18 stars 25 forks source link

Support symbolic links in inputs #385

Open aidanheerdegen opened 7 months ago

aidanheerdegen commented 7 months ago

Currently using symbolic links in input in config.yaml does not fully resolve the target of the link when making a link in the work directory during setup.

For examples, making a symbolic link like this in the control dir:

inputs -> /g/data/vk83/experiments/inputs/access-om2

and then referencing inputs in the input path:

          - ./inputs/ocean/chlorophyll/global.1deg/2020.05.30/chl.nc

leads to this symbolic link in work

lrwxrwxrwx 1 aph502 tm70 56 Nov 22 16:43 chl.nc -> ./inputs/ocean/chlorophyll/global.1deg/2020.05.30/chl.nc

The manifest also has the incorrect fullpath

work/ocean/INPUT/chl.nc:
  fullpath: ./inputs/ocean/chlorophyll/global.1deg/2020.05.30/chl.nc

Why is this important?

Being able to utilise symlinks in the control experiment directory can remove a lot of repeated text in config.yaml, which makes the configuration more legible and understandable. Also putting a symlink in the control directory makes the inputs discoverable easily from within the control directory, just as archive and work are.

Note I am not suggesting payu makes an input directory symlink, simply that it support such usage.

aidanheerdegen commented 7 months ago

I think the solution is to do something like this

               f_orig = os.path.realpath(os.path.join(path, f_name))

here

https://github.com/payu-org/payu/blob/9523ae3e73ab65b681e27a38c2a77e95d4f8e843/payu/models/model.py#L273