search

biocore / mg-scripts

Knight Lab internal Metagenomic processing scripts for demultiplexing, QC and host removal
BSD 3-Clause "New" or "Revised" License
1 stars 5 forks source link

shm and se operation changes #114

Open wasade opened 9 months ago

wasade commented 9 months ago

...I think this covers the differences.

The changed demux method is here (/home/mcdonadt/2023.08.09-pangenome-human-depletion/human-depletion-se-pe/demux-inline-all.py). The splitter is (/home/mcdonadt/2023.08.09-pangenome-human-depletion/human-depletion-se-pe/splitter/target/release/splitter) with the source in the corresponding parent directory. This code depends as well on fastq_pair which is installable from bioconda

wasade commented 9 months ago

Was running out the door before. We've been doing processing at 5gb chunks, and requesting 120GB to ensure we have sufficient shared memory, until barnacle supports shared memory as a resource. We have been using 16 cores but our efficiency great, so recommend reducing to 10 or 12