pepkit / peppy

Project metadata manager for PEPs in Python
https://pep.databio.org/peppy
BSD 2-Clause "Simplified" License
37 stars 12 forks source link

separation of looper-specific config attributes #200

Open nsheff opened 6 years ago

nsheff commented 6 years ago

In discussion with @johanneskoester it came up that the metadata.pipeline_interfaces section could be considered looper-specific. There are several other looper-specific sections (which I've already divided out in the docs), but this one is unique because it's part of the metadata section.

What about collecting all of these things into a new looper section of the project config?

The rationale is, each tool that reads a PEP could specify an optional section and do what it wants there. Then you just need to define that section in your PEP if you're using that tool. If it made sense, there could equivalently be introduced a snakemake section that could be encode app-specific project configuration (if any exists).

So I guess what I'm proposing is to group the sections for pipeline_args, pipeline_config, and compute(maybe?), plus the attribute metadata.pipeline_interfaces, into a new section called looper. I guess the point of this is to have a home for metadata.pipeline_interfaces. I guess metadata.results_subdir and metadata.submission_subdir also belong in this class of attributes, so they would follow pipeline_interfaces.

Just a thought. We can also just leave it how it is, which is optional and under metadata.

What do you think, @afrendeiro, @vreuter ?

vreuter commented 6 years ago

I really like that grouping and separation. I'm fine with it being called looper and think that makes the most sense for now since that's the primary consumer of a pep, but I think that we could even call it something like engine, consumer, or executor with the idea that a pep could also be used by a looper analogue that handles submission of samples to pipelines as defined by a pep.

afrendeiro commented 6 years ago

Yep, makes sense those would be under looper. I think that as you point out, a pep should not necessarily be restricted in what sections it have and could allow for any tool that uses a pep to extend it as long as the minimum is there.

nsheff commented 6 years ago

ok thanks, I will change the docs to reflect this.