pepkit / looper

A job submitter for Portable Encapsulated Projects
http://looper.databio.org
BSD 2-Clause "Simplified" License
20 stars 7 forks source link

It's unclear why the pipeline type is specified both in the pipeline interface and the looper config #465

Closed nsheff closed 1 week ago

nsheff commented 4 months ago

Right now, a pipeline interface can say pipeline_type: sample or pipeline_type: project.

Then, in the .looper.yaml, you say something like:

pipeline_interfaces:
  sample: ../pipeline/pipeline_interface.yaml

So, you're specifying the type of the pipeline there, too. What if these don't match? Like, you say sample: piface.yaml but that actually says pipeline_type: project inside?

Why can't looper just read the pipeline_type from the piface in the first place? We could simplify the looper config to:

pipeline interfaces:
  - ../pipeline/piface.yaml
donaldcampbelljr commented 1 week ago

This work is now complete with the above PR. It also maintains backwards compatibility.

Now, the user can give a list of pifaces in the looper config and Looper will assign them to sample or project pifaces given the pipeline_type within the piface:

pep_config: project/project_config.yaml # local path to pep config
output_dir: "results"
pipeline_interfaces:
  - pipeline/pipeline_interface.yaml
  - pipeline/pipeline_interface_project.yaml
donaldcampbelljr commented 1 week ago

When working on the above changes, I had forgotten that, for Looper 2.0, we are considering placing both sample and project-level interfaces into one pipeline interface: https://github.com/pepkit/looper/issues/493