search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
18 stars 25 forks source link

postscript only runs if collating #289

Closed aidanheerdegen closed 3 years ago

aidanheerdegen commented 3 years ago

The postscript hook currently only runs if the model does a collate step.

https://github.com/payu-org/payu/blob/4dc4a2b2d431b337a55743ee1c870edced3c0e2a/payu/subcommands/collate_cmd.py#L112-L113

There are models that do not collate, and situations were models that can collate don't need to.

Currently postscript is the only hook that is available that is guaranteed to run after all possible collation, but it should also be available where collation is not required, for consistency.

aidanheerdegen commented 3 years ago

Current plan is to add some logic after here

https://github.com/payu-org/payu/blob/master/payu/experiment.py#L804

to call postscript if collate is False.

marshallward commented 3 years ago

Postscript is a legacy hook that was specifically designed to be used with collation.

Before making this change, look into the userscript feature, which allows insertion of scripts and commands at various stages of a run.

marshallward commented 3 years ago

Looks like it was documented here:

https://payu.readthedocs.io/en/latest/config.html#postprocessing

aidanheerdegen commented 3 years ago

True dat (documentation)

The nice thing about postscript is that it is called after everything else has completed. I agree that the archive hook is very similar for an experiment which does not call collate, but the user needs to know that this is the case. I think it adds to the usability to make it work consistently regardless of collation.

marshallward commented 3 years ago

Yeah, that is a good point. There is no way to signal a "finalize" task at the moment.

Postscript was never a well thought out feature, and I guess that I had hoped to integrate postscript into the userscripts system (which is more powerful), but its not necessary, and I suppose there's no harm in bumping it up to the run itself.