search

dessn / Pippin

Pipeline for photometric SN analysis
MIT License
9 stars 10 forks source link

Allow tasks to see forward #89

Closed OmegaLambda1998 closed 1 year ago

OmegaLambda1998 commented 2 years ago

It is becoming increasingly obvious that having the ability to know what tasks will run in the future will be useful.

OmegaLambda1998 commented 2 years ago

As an example, create_cov (which was initially built for cosmomc) now no longer always needs to produce the (large) cosmomc input. At the moment you can specify to not produce this input via the COSMOMC flag, but instead create_cov could just check whether a cosmomc job exists.

OmegaLambda1998 commented 2 years ago

Another way this could be really useful is with optimised compressing and uncompressing. At the moment tasks will compress once they finish, then uncompress if they are a dependency of another task, then compress again once that other task succeeds or fails. Instead we can say "if this task is a dependency, then don't bother compressing it yet". This should save us a TON of compressing and uncompressing