results subdir

[nsheff]

Something strange is going on.

All my recent jobs are no longer putting results in a 'results_pipeline' subdir. As best I can tell, right now if you don't put a results subdir, it's no longer using the default and is just putting all results into the parent subdirectory. I haven't experimented with other scenarios.

On the other hand, check appears to still be checking for the default results subdir in results_pipeline if nothing is specified.

So, there's clearly a disconnect...

@stolarczyk did you intentionally remove the results_subdir default?

[stolarczyk]

@stolarczyk did you intentionally remove the results_subdir default?

No

Could you name the pipeline interface you used for the jobs? and the PEP?

I guess it's related to the change in looper namespace, so the pipeline interfaces need to be updated (did it for PEPATAC@cfg2 today) to point to an appropriate output directory.

[nsheff]

I was using pepatac and paqc... but what do you mean, what has to be changed in the piface? what does that have to do with the results subdir? I'm missing something...

[nsheff]

Ok now I see....

https://github.com/databio/pepatac/commit/8cb83af9bbf238c0a6be67268439f4d96af44f55

That's interesting...

[nsheff]

but that's a relative path, not an absolute path, right? doesnt it need to be

-O {looper.output_dir}/{looper.results_subdir}

[stolarczyk]

no, looper.results_subdir is an absolute path to the results directory for the project. So it is: Project.looper.output_dir/Project.looper.results_subdirand Project.looper.results_subdir is "results_pipeline" by default

The docs describe that this way as well

[nsheff]

ah. I thought we had decided to do it the other way.

[nsheff]

Ok, I think this is making sense to me now... not a looper issue, but all command templates need to refer to the subdir directly now.

[nsheff]

actually, now when I set a custom results subdir, it does not appear to be honored...

[nsheff]

https://github.com/pepkit/geofetch/blob/dev/docs_jupyter/tutorial.ipynb

In this example, I would expect the sra_convert pipeline to put its results into red_algae/sra_convert_results, because of this:

project_modifiers:
  amend:
    sra_convert:
      looper:
        results_subdir: sra_convert_results      

but instead, those results are put into red_algae/results_pipeline... am I doing something wrong?

[stolarczyk]

I think I know what the issue is. The choice of name is respected but not when you set it as an amendment.

This is caused by the fact that I utilized the cli options setting route in looper section to bring back the top level looper settings functionality in order not to have two ways of treating things.

[nsheff]

I see. that seems reasonable -- but I guess you can only do that after processing amendments, right?

[stolarczyk]

it should be possible now

[nsheff]

Yes, that fixed it for that example. thanks.