iljungr
commented
9 years ago Here's what I've done and what remains to complete the read depletion pipeline. You had suggested in your earlier email that the desired steps are something like: 1) bmtagger to pull aside microbial reads for later use. 2) bmtagger to deplete human reads. 3) m-vicuna for duplicate removal 4) blastn for more thorough human depletion. 5) Add back microbial reads from step 1, fix anything that needs fixing, convert to bam.
What is done so far is the partition_bmtagger utility to be used for steps 1 and 2, and the installation of blastn for step 4. I think I can do the rest in a few days except for the following issues:
a) I don't know what to do about installation of m-vicuna. All I have is a path on broad -- I can't find anything public. It is compiled C++, not just some wrapper script around vicuna. For our internal testing I can use the broad path, but I don't know what to do for Travis or the released product. I think we should contact Xiao and ask about it. Will you do that or should I?
b) For blastn there is the issue of managing the use of multiple nodes. It should be easy enough to start by implementing the 3 pieces: split, blastn one piece, merge. I was also thinking of putting in 3 higher level utilities to bsub the jobs, check the progress of those jobs, and merge the results. We would use that internally and offer it to anyone who has bsub available, but it couldn't be tested on Travis. For an initial test run that could be done manually.
c) Of course I have no idea what will come up in the "fix anything that needs fixing" part of step 5.
As to matching the old intermediate steps, we would have to use the old sequence of steps, with novoalign. I don't see any particular problem with this for in-house use; it would test our low-level utilities but not the pipeline. For now, it would test the bmtagger tool and perhaps filter_lastal and trimmomatic if they are part of the same workflow.
I'm fine with working on that if that's the next thing to do.
- Irwin
On Nov 3, 2014, at 9:54 AM, Daniel Park notifications@github.com wrote:
How close are you to completing most of the necessary steps in the human & contaminant read depletion pipeline? (skipping the novoalign steps that we won't use).
I ask because I just a set of data files (input, intermediates, and final output) from Kristian for two samples run through his depletion steps, and we can try to compare how close our outputs are to his. In theory, assuming all steps are deterministic, some of these files might look exactly the same run through his steps and our steps, but I think we'd have to be prepared for the possibility that the files may be mostly the same and we'd need to do an analysis of whether it's "close enough". In particular, maybe just read counts at each stage (beginning, post-bmtagger, post-blast-and-concatenation, etc) and see how close they are his way and with our code.
@iljungr, would you be up for this kind of an analysis project or would you prefer to keep coding new tools and I can find someone else to validate this?
For now, it would involve running each command manually. If the regression test looks good, then I can introduce a pipelining framework that's our currently leading option for stringing this all together in an automated way. This depletion pipeline is probably the best place to start trying that framework, since it's a small and defined unit that can be thought of separately from the rest of the analyses.
— Reply to this email directly or view it on GitHub.
dpark01
How close are you to completing most of the necessary steps in the human & contaminant read depletion pipeline? (skipping the novoalign steps that we won't use).
I ask because I just a set of data files (input, intermediates, and final output) from Kristian for two samples run through his depletion steps, and we can try to compare how close our outputs are to his. In theory, assuming all steps are deterministic, some of these files might look exactly the same run through his steps and our steps, but I think we'd have to be prepared for the possibility that the files may be mostly the same and we'd need to do an analysis of whether it's "close enough". In particular, maybe just read counts at each stage (beginning, post-bmtagger, post-blast-and-concatenation, etc) and see how close they are his way and with our code.
@iljungr, would you be up for this kind of an analysis project or would you prefer to keep coding new tools and I can find someone else to validate this?
For now, it would involve running each command manually. If the regression test looks good, then I can introduce a pipelining framework that's our currently leading option for stringing this all together in an automated way. This depletion pipeline is probably the best place to start trying that framework, since it's a small and defined unit that can be thought of separately from the rest of the analyses.