Report number or fraction of reads that pass each step

[benjjneb]

I'm thinking just add printed reports to the standard output that goes into the log file. The read tracking information is very useful for diagnosing problems.

[jairideout]

:+1:, this sounds really useful!

Originally came up on the forum here.

[gregcaporaso]

:+1: a user that I'm working with now has requested this information. We should now be able to do better than dumping it to standard out - we should be able to put the data into a qzv if/when we turn these methods into Pipelines.

[benjjneb]

A visualization of something along these lines was requested in the Q2 forum as well: https://forum.qiime2.org/t/summary-statistics-after-dada2/1860/7

I am over-busy until my class ends, but adding this is high on my to-do list for December.

[gregcaporaso]

I realized another way to do this would be to output stats artifacts which could then be viewable as metadata with qiime metadata tabulate... Anyway, @benjjneb, there are a few ways we could proceed with this, so just let us know when you're ready to work on it.

[ebolyen]

Came up on the forum: x-ref

[jairideout]

another forum xref

[benjjneb]

The updated R scripts in the 1.6 branch now perform this tracking. However, for now they simply report the results for the top few samples to stdout.

Often that will be sufficient, but it would be easy to write this out in tabular format as well along the lines of @gregcaporaso comment. Suggestions? What format would be appropriate for subsequent Q2 viewing?

[gregcaporaso]

... it would be easy to write this out in tabular format as well along the lines of @gregcaporaso comment. Suggestions? What format would be appropriate for subsequent Q2 viewing?

I think this would be ideal.

@benjjneb, could you paste a few lines of the report that goes to stdout here (or point us at an example)? That'll help us to advise on the most appropriate format.

[jairideout]

Discussed this on Slack with @gregcaporaso. For the 2017.12 release next week, let's just go with the new stdout that @benjjneb described, that way users will have access to it in the debug/verbose logs and it won't require any more work on our end once https://github.com/qiime2/q2-dada2/pull/78 is merged.

In the 2018.1 release, we can implement the better solution of outputting DADA2 stats that are viewable as Metadata (i.e. what @gregcaporaso suggested above). That solution should be fairly easy to implement, but it'll require a few pieces: a new semantic type, file/dir format, and a transformer to view as Metadata, and the DADA2 methods denoise-single and denoise-paired will need to become Pipelines. @benjjneb, we are happy to help with this for 2018.1 -- having an example of the tabular output from DADA2 would be helpful when you have the chance.

[benjjneb]

The output right now looks like this:

                                 input filtered denoised merged non-chimeric
F3D0_S188_L001_R1_001.fastq.gz    7793     7113     7113   6600         6572
F3D1_S189_L001_R1_001.fastq.gz    5869     5299     5299   5078         5067
F3D141_S207_L001_R1_001.fastq.gz  5958     5463     5463   5047         4928
F3D142_S208_L001_R1_001.fastq.gz  3183     2914     2914   2663         2600
F3D143_S209_L001_R1_001.fastq.gz  3178     2941     2941   2575         2550
F3D144_S210_L001_R1_001.fastq.gz  4827     4312     4312   3668         3517

Entries will all be integers (numbers of reads).

[jairideout]

Thanks for the example @benjjneb! Having that printed to stdout for now will be a nice incremental improvement for users debugging their data, and that output should be easy to turn into Metadata for the 2018.1 release :shipit:

[thermokarst]

Fixed in #78