Is the new "paths" section a misnomer?

[nsheff]

Should we rename the paths section?

we recently implemented a new section called paths that provides templates, which we are using to refer to either:

• paths to the main script
• paths to pre-submission hook scripts, which provides generic functionalty that supersedes the dynamic variables command templates.

In the process, we made it so these paths are no longer automatically interpreted as relative, so you have to say {looper.output_dir}/blah or something if you want to make the path relative to the piface or output dir.

But now that they aren't automatically relatives, is paths the right name? Could you use this section for some other variable? Sure...it is actually now essentially mimicking the function of derived columns in peppy -- except more powerful and broad because it can draw not only from sample attributes, but from the other namespaces as well.

So you could say,

paths:
  pipeline_mode: {project.attr}-{sample.attr}

That would work -- but it's not a path.

Or is it needed at all?

In thinking about this, I realized that these are only used in the templates anyway; so maybe this section isn't even needed. for example, we now have this:

paths:
  refgenie: "refgenie"
  compute_vars: "{looper.piface_dir}/build-compute-params.py"
pre_submit:
  command_template: >
    {pipeline.paths.compute_vars} -a {sample.asset} -g {sample.genome}
command_template: >
  {pipeline.paths.refgenie} build { sample.genome }/{ sample.asset }

but since we eliminated the relativity, you can also do:

pre_submit:
  command_template: >
    {looper.piface_dir}/build-compute-params.py -a {sample.asset} -g {sample.genome}
command_template: >
  refgenie build { sample.genome }/{ sample.asset }

you're just building the commands right into the command templates, which also derive from namespaces.

[nsheff]

The value of the paths section is that it's the only way to pass a derived parameter to a python presubmission function.

Need to think of a less exclusive name for paths that's still short since we'll refer to them as pipeline.paths.key, which is long.

[nsheff]

• attribute_templates
• var_templates
• vars
• attr_templates

[nsheff]

var_templates

[nsheff]

I couldn't find any documentation of this var_templates behavior, did we document this?

[stolarczyk]

http://looper.databio.org/en/latest/pipeline-interface-specification/#var_templates