Closed taltman closed 4 years ago
/cc @JustinChu
Here are the tools I'm looking into: khmer (https://github.com/dib-lab/khmer) kmernorm (http://sourceforge.net/projects/kmernorm) ORNA (https://github.com/SchulzLab/ORNA)
Are there specific transcriptomes I should focus on for this? Otherwise I have some of my own that I can use a tester.
My loose plan is:
Thanks for jumping in @McGlock !
We are trying to develop some validation datasets, but they are not ready yet. I'd say that @ababaian can suggest some datasets that you can use for now just for the sake of building something. Ideally you can make a test example using Singularity or Docker, if not, just a reproducible script checked in to a dev branch in the 'notebook' directory would be good. You can use a dev branch like 'mcglock-dev'.
BTW, don't assemblers like SPAdes and MegaHIT do this kind of kmer normalization?
The PEDV high coverage are probably a good toy dataset. They're pretty robust: https://github.com/ababaian/serratus/issues/96#issuecomment-629440696
I know certainly that some datasets suffer from much higher 3' bias than others, I can't recall where I saw it really bad though
Libraries with clear CoV bias are: SRR1194065, SRR1194066, SRR1194067, SRR1194068.
You can measure runtime using the shell builtin 'time', or /usr/bin/time. For memory usage profiling, you can use valgrind. Informally, you can just monitor the progress using 'top' and noting how much memory & cores it seems to utilize.
BTW, please read over the CONTRIBUTING.md file in this repo to understand where experiments like this go, in terms of code & output data.
BTW, don't assemblers like SPAdes and MegaHIT do this kind of kmer normalization?
I looked around a little but couldn't verify if either of them use digital/kmer normalization. I did find some reference to it being tried as a perprocessing step before assembly. Let me know if you have any more infor on this though.
You can measure runtime using the shell builtin 'time', or /usr/bin/time. For memory usage profiling, you can use valgrind. Informally, you can just monitor the progress using 'top' and noting how much memory & cores it seems to utilize.
Cool thanks for this, I know time
but I'll look into valgrind
. Otherwise, yeah I was just going to lurk on top
.
No lurking required -- you can use the -b option to run in "batch" mode and capture the output for later analysis, something like
while true top -n1 -b | grep ProgramName >> top.log; sleep 10; done
Kill the loop when the process completes. Finding maximum usage for memory or other resource can be done my massaging the file with sed or whatever followed by sort | head -n1.
Check out the SPAdes manual, sections 3 & 4, regarding read correction by kmer counting:
Check out the SPAdes manual, sections 3 & 4, regarding read correction by kmer counting:
I did see those standalone scripts, I just didn't know if those were baked into the pipelines or just convenience scripts to aid in preprocessing. But yes indeed, those tools should on on the list of things to try for sure!
Tools presented in section 4 of the manual are essentially standalone binaries that use some internal SPAdes algorithms. We decided that it would be interesting to provide them standalone.
In addition to this, I would not suggest doing any "digital normalization". Often these approaches create coverage gaps. This is even more likely for the data in question as the coverage across the genome is very non-uniform.
Here is example of coverage from PEDV dataset:
Good to close for now?
Yes I believe so, after input from Tomer and the SPAdes crew, I don't think this is needed.
On Tue, Jun 16, 2020 at 4:15 PM Artem Babaian notifications@github.com wrote:
Good to close for now?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ababaian/serratus/issues/97#issuecomment-645058199, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUJ3JR57J5ZSM5H5YNSFWDRW74KLANCNFSM4NCLU3QA .
-- Ryan J. McLaughlin, MSc
e1: mclaughlinr2@gmail.com c: 814.758.4570 linkedin https://www.linkedin.com/in/ryan-mclaughlin-725b09117/
This is an approach to identify k-mers that are disproportionately represented in the reads, which can cause problems for the assembler. Once identified, they can be thinned out.
Once Ryan is on-board, I will assign this to him.