BD2KGenomics / toil-scripts

Toil workflows for common genomic pipelines
Apache License 2.0
32 stars 18 forks source link

Create job factories that automatically compute requirements #445

Closed hannes-ucsc closed 7 years ago

hannes-ucsc commented 8 years ago

Instead of

from toil_lib.tools import foo_job
job.addChild( foo_job, disk=PromisedRequirement( lambda x: x.size * 3.17, other_job.rv() ) )

We should be able to

from toil_lib.tools import foo_job_factory
job.addChild( **foo_job_factory( other_job.rv() ) )

This way the knowledge of how foo_job's disk requirement relates to the size of its input file is encapsulated in a library function that is colocated with the job function.

Instead of creating a factory, we could also convert foo_job from a job function to a class-based job, i.e. a subclass of Job. Those can define their own requirements in the constructor, IIRC.

hannes-ucsc commented 8 years ago

Cat hit send. Fixing.

hannes-ucsc commented 8 years ago

There, I fixed it. Pinging @jvivian, @jpfeil @fnothaft for input.

hannes-ucsc commented 7 years ago

Too complicated.