Closed splaisan closed 4 years ago
Hi Stephane,
Thank you for pointing it out. A bug caused GNU parallel to spawn thousands of jobs exceeding the job limit on most systems. On some systems this crashes the pipeline. The bug has been fixed in 09077c6.
Would you mind trying to install the most recent version and running the test again?
regards Søren
please ignore I put main in BRANCH instead of master getting late ;-)
not sure if it is me but a fresh full-install stalls at git cloning
git clone \
> --branch "$BRANCH" \
> https://github.com/SorenKarst/longread-UMI-pipeline.git \
> $CONDA_PREFIX/longread_umi
Cloning into '/opt/biotools/miniconda3/envs/longread_umi/longread_umi'...
fatal: Remote branch main not found in upstream origin
finished OK after clean install also performed the next test and also finished OK looks like we are back in business :-) best S
longread_umi qc_pipeline \
-d test_reads.fq \ -c test/consensus_raconx3_medakax1.fa \ -r zymo_curated \ -t 1 \ -u test \ -o test/qc [M::mm_idx_gen::0.0140.86] collected minimizers [M::mm_idx_gen::0.0170.93] sorted minimizers [M::main::0.0170.93] loaded/built the index for 43 target sequence(s) [M::mm_mapopt_update::0.0180.91] mid_occ = 45 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 43 [M::mm_idx_stat::0.0180.89] distinct minimizers: 4812 (17.60% are singletons); average occurrences: 7.226; average spacing: 5.454 [M::worker_pipeline::0.8970.99] mapped 581 sequences [M::main] Version: 2.17-r941 [M::main] CMD: minimap2 -x map-ont -t 1 /opt/biotools/miniconda3/envs/longread_umi/longread_umi/scripts/zymo-ref-uniq_2019-10-28.fa test_reads.fq [M::main] Real time: 0.899 sec; CPU: 0.888 sec; Peak RSS: 0.008 GB [M::mm_idx_gen::0.0130.93] collected minimizers [M::mm_idx_gen::0.0170.97] sorted minimizers [M::main::0.0170.97] loaded/built the index for 43 target sequence(s) [M::mm_mapopt_update::0.0170.94] mid_occ = 45 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 43 [M::mm_idx_stat::0.0170.92] distinct minimizers: 4812 (17.60% are singletons); average occurrences: 7.226; average spacing: 5.454 [M::worker_pipeline::0.1590.98] mapped 9 sequences [M::main] Version: 2.17-r941 [M::main] CMD: minimap2 -ax map-ont -t 1 --cs /opt/biotools/miniconda3/envs/longread_umi/longread_umi/scripts/zymo-ref-uniq_2019-10-28.fa test/qc/consensus_raconx3_medakax1.fa [M::main] Real time: 0.160 sec; CPU: 0.160 sec; Peak RSS: 0.006 GB [M::mm_idx_gen::0.0160.74] collected minimizers [M::mm_idx_gen::0.0200.81] sorted minimizers [M::main::0.0200.81] loaded/built the index for 43 target sequence(s) [M::mm_mapopt_update::0.0200.79] mid_occ = 45 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 43 [M::mm_idx_stat::0.0200.78] distinct minimizers: 4812 (17.60% are singletons); average occurrences: 7.226; average spacing: 5.454 [M::worker_pipeline::4.4250.99] mapped 581 sequences [M::main] Version: 2.17-r941 [M::main] CMD: minimap2 -ax map-ont -t 1 --cs /opt/biotools/miniconda3/envs/longread_umi/longread_umi/scripts/zymo-ref-uniq_2019-10-28.fa test/qc/test_reads.fa [M::main] Real time: 4.427 sec; CPU: 4.364 sec; Peak RSS: 0.030 GB usearch v11.0.667_i86linux32, 4.0Gb RAM (528Gb total), 88 cores (C) Copyright 2013-18 Robert C. Edgar, all rights reserved. https://drive5.com/usearch
00:00 41Mb 100.0% Reading test/consensus_raconx3_medakax1.fa 00:00 44Mb 100.0% Reading /opt/biotools/miniconda3/envs/longread_umi/longread_umi/scripts/zymo-ref-uniq_2019-10-28.fa 00:00 11Mb 100.0% Converting to upper case
00:00 12Mb 100.0% Word stats
00:00 12Mb 100.0% Alloc rows 00:00 12Mb 100.0% Build index 00:00 12Mb CPU has 88 cores, defaulting to 10 threads 00:00 326Mb 100.0% Chimeras 0/9 (0.0%), in db 1 (11.1%), not matched 8 (88.9%)
Awesome :)
Using a fresh conda install under ubuntu 16 with 88 cores and 512GB RAM FR: could you please add a '-v / --version' argument to your tools to get the pipeline version
I do get 9 consensus sequences but the last step (near variant analysis) reports errors and fails any idea?
Thanks Stephane
longread_umi_nanopore_pipeline_log_2020-02-12-10:51:57.txt