Closed pcodyhe closed 2 years ago
byee4
commented
3 years ago Perhaps it could be related to this issue? https://github.com/CGATOxford/UMI-tools/issues/409
There is a pipeline that might work, which basically removes the 'stats' output and should theoretically lower the memory requirement. Can you try with cwl/wf_get_peaks_nostats_se.cwl ?
pcodyhe
commented
3 years ago Hi Bryan,
Thanks, I tried with wf_get_peaks_nostats_se.cwl but I'm getting errors that may have to do with the slightly different yaml input file required for this workflow, see below.
INFO [step gzip_demux] start DEBUG [job gzip_demux] initializing from file:///home/pcodyhe/results/eclip.pipeline/eclip/cwl/gzip.cwl as part of step gzip_demux DEBUG [job gzip_demux] { "input": { "location": "file:///tmp/8qusbbkz/2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq", "basename": "2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq", "nameroot": "2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1", "nameext": ".fq", "class": "File", "checksum": "sha1$ef37a8c992dd30cbe714989adc1f18ca267549aa", "size": 16030782861, "http://commonwl.org/cwltool#generation": 0 }, "stdout": true } DEBUG [job gzip_demux] path mappings is { "file:///tmp/8qusbbkz/2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq": [ "/tmp/8qusbbkz/2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq", "/tmp/tmpjko9iekf/stg4758dd72-e593-4b08-9b28-bd1f23d72401/2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq", "File", true ] } ERROR Unexpected exception Traceback (most recent call last): File "/home/pcodyhe/.local/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 865, in job "stdout must be a relative path, got '%s'" % j.stdout schema_salad.exceptions.ValidationException: stdout must be a relative path, got '2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq.gz'
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/pcodyhe/.local/lib/python3.7/site-packages/cwltool/workflow.py", line 443, in job runtimeContext, File "/home/pcodyhe/.local/lib/python3.7/site-packages/cwltool/command_line_tool.py", line 865, in job "stdout must be a relative path, got '%s'" % j.stdout File "/home/pcodyhe/.local/lib/python3.7/site-packages/schema_salad/sourceline.py", line 251, in exit raise self.makeError(str(exc_value)) from exc_value schema_salad.exceptions.ValidationException: cwl/gzip.cwl:23:1: stdout must be a relative path, got '2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq.gz' ERROR [step gzip_demux] Cannot make job: cwl/gzip.cwl:23:1: stdout must be a relative path, got '2020.9.29.eclip.pilot.rbm15.test.nostats..RBM15.input.umi.r1.fq.gz'
byee4
commented
3 years ago Let me check that workflow out. Would you mind sending me a small example if your manifest YAML file?
Hi,
I am using the SE pipeline (cwltool --debug ./cwl/wf_get_peaks_scatter_se.cwl single_end_clip.yml) to processe SE eCLIP data and ran into a memory error on step barcode collapse 2. I have 60 million reads/library, and if I downsample the FASTQ files to 6 million reads, the workfow runs succesfully, so it seems like the number of reads is the problem? I have ~500 gb of memory available. Is there some work around for this? Thanks!
DEBUG [job X_barcodecollapsese_2] initial work dir {} WARNING [job X_barcodecollapsese_2] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. INFO [job X_barcodecollapsese_2] /tmp/_zimihx5$ docker \ run \ -i \ --mount=type=bind,source=/tmp/_zimihx5,target=/lRfJUo \ --mount=type=bind,source=/tmp/j813jzsx,target=/tmp \ --mount=type=bind,source=/tmp/e45nvhsy/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam,target=/var/lib/cwl/stgfdb50d41-64ed-4ded-8f14-d448aaaa7971/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam,readonly \ --mount=type=bind,source=/tmp/ds03kaue/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam.bai,target=/var/lib/cwl/stgfdb50d41-64ed-4ded-8f14-d448aaaa7971/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam.bai,readonly \ --workdir=/lRfJUo \ --read-only=true \ --user=1011:1011 \ --rm \ --env=TMPDIR=/tmp \ --env=HOME=/lRfJUo \ --cidfile=/tmp/0dvfj2rp/20210310134104-914376.cid \ brianyee/umi_tools:1.0.0 \ umi_tools \ dedup \ --random-seed \ 1 \ -I \ /var/lib/cwl/stgfdb50d41-64ed-4ded-8f14-d448aaaa7971/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam \ --method \ unique \ --output-stats \ 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo \ -S \ 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.rmDup.bam
UMI-tools version: 1.0.0
output generated by dedup --random-seed 1 -I /var/lib/cwl/stgfdb50d41-64ed-4ded-8f14-d448aaaa7971/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam --method unique --output-stats 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo -S 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.rmDup.bam
job started at Wed Mar 10 19:41:11 2021 on 7c957b085e91 -- e4477934-26db-426d-adb6-0bac1268f279
pid: 1, system: Linux 4.4.0-190-generic #220-Ubuntu SMP Fri Aug 28 23:02:15 UTC 2020 x86_64
assigned_tag : None
cell_tag : None
cell_tag_delim : None
cell_tag_split : -
chimeric_pairs : use
chrom : None
compresslevel : 6
detection_method : None
gene_tag : None
gene_transcript_map : None
get_umi_method : read_id
ignore_umi : False
in_sam : False
log2stderr : False
loglevel : 1
mapping_quality : 0
method : unique
no_sort_output : False
out_sam : False
output_unmapped : False
paired : False
per_cell : False
per_contig : False
per_gene : False
random_seed : 1
read_length : False
short_help : None
skip_regex : ^(__|Unassigned)
soft_clip_threshold : 4
spliced : False
stats : 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo
stderr : <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>
stdin : <_io.TextIOWrapper name='/var/lib/cwl/stgfdb50d41-64ed-4ded-8f14-d448aaaa7971/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam' mode='r' encoding='UTF-8'>
stdlog : <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>
stdout : <_io.TextIOWrapper name='2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.rmDup.bam' mode='w' encoding='UTF-8'>
subset : None
threshold : 1
timeit_file : None
timeit_header : None
timeit_name : all
tmpdir : None
umisep :
umi_tag : RX
umi_tag_delim : None
umi_tag_split : None
unmapped_reads : discard
unpaired_reads : use
whole_contig : False
2021-03-10 19:41:11,247 INFO command: dedup --random-seed 1 -I /var/lib/cwl/stgfdb50d41-64ed-4ded-8f14-d448aaaa7971/2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.bam --method unique --output-stats 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo -S 2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo.rmDup.bam 2021-03-10 19:45:20,177 INFO total_umis 38365775 2021-03-10 19:45:20,178 INFO #umis 1028185 2021-03-10 19:46:58,880 INFO Written out 100000 reads 2021-03-10 19:47:03,865 INFO Written out 200000 reads 2021-03-10 19:47:13,656 INFO Written out 300000 reads 2021-03-10 19:47:23,359 INFO Written out 400000 reads 2021-03-10 19:47:28,370 INFO Written out 500000 reads 2021-03-10 19:47:33,373 INFO Written out 600000 reads 2021-03-10 19:47:38,375 INFO Written out 700000 reads 2021-03-10 19:47:43,451 INFO Written out 800000 reads 2021-03-10 19:47:46,894 INFO Parsed 1000000 input reads 2021-03-10 19:47:48,478 INFO Written out 900000 reads 2021-03-10 19:47:53,472 INFO Written out 1000000 reads 2021-03-10 19:47:58,461 INFO Written out 1100000 reads 2021-03-10 19:48:03,464 INFO Written out 1200000 reads 2021-03-10 19:48:08,479 INFO Written out 1300000 reads 2021-03-10 19:48:13,487 INFO Written out 1400000 reads 2021-03-10 19:48:18,515 INFO Written out 1500000 reads 2021-03-10 19:48:23,528 INFO Written out 1600000 reads 2021-03-10 19:48:28,513 INFO Written out 1700000 reads 2021-03-10 19:48:30,392 INFO Parsed 2000000 input reads 2021-03-10 19:48:33,507 INFO Written out 1800000 reads 2021-03-10 19:48:38,527 INFO Written out 1900000 reads 2021-03-10 19:48:44,404 INFO Written out 2000000 reads 2021-03-10 19:48:49,498 INFO Written out 2100000 reads 2021-03-10 19:48:54,845 INFO Written out 2200000 reads 2021-03-10 19:49:00,019 INFO Written out 2300000 reads 2021-03-10 19:49:05,211 INFO Written out 2400000 reads 2021-03-10 19:50:45,205 INFO Written out 2500000 reads 2021-03-10 19:50:50,243 INFO Written out 2600000 reads 2021-03-10 19:50:51,050 INFO Parsed 3000000 input reads 2021-03-10 19:50:55,393 INFO Written out 2700000 reads 2021-03-10 19:51:03,527 INFO Written out 2800000 reads 2021-03-10 19:51:08,613 INFO Written out 2900000 reads 2021-03-10 19:51:13,743 INFO Written out 3000000 reads 2021-03-10 19:51:19,050 INFO Written out 3100000 reads 2021-03-10 19:51:24,622 INFO Written out 3200000 reads 2021-03-10 19:51:29,848 INFO Written out 3300000 reads 2021-03-10 19:51:35,437 INFO Written out 3400000 reads 2021-03-10 19:51:39,891 INFO Parsed 4000000 input reads 2021-03-10 19:51:40,714 INFO Written out 3500000 reads 2021-03-10 19:51:45,949 INFO Written out 3600000 reads 2021-03-10 19:51:51,237 INFO Written out 3700000 reads 2021-03-10 19:51:56,498 INFO Written out 3800000 reads 2021-03-10 19:52:01,773 INFO Written out 3900000 reads 2021-03-10 19:52:07,214 INFO Written out 4000000 reads 2021-03-10 19:52:12,533 INFO Written out 4100000 reads 2021-03-10 19:52:17,992 INFO Written out 4200000 reads 2021-03-10 19:52:23,270 INFO Written out 4300000 reads 2021-03-10 19:52:26,529 INFO Parsed 5000000 input reads 2021-03-10 19:52:32,464 INFO Written out 4400000 reads 2021-03-10 19:52:37,882 INFO Written out 4500000 reads 2021-03-10 19:52:43,237 INFO Written out 4600000 reads 2021-03-10 19:54:32,883 INFO Written out 4700000 reads 2021-03-10 19:54:38,745 INFO Written out 4800000 reads 2021-03-10 19:54:43,925 INFO Written out 4900000 reads 2021-03-10 19:54:49,403 INFO Written out 5000000 reads 2021-03-10 19:54:54,809 INFO Written out 5100000 reads 2021-03-10 19:55:00,123 INFO Written out 5200000 reads 2021-03-10 19:55:01,808 INFO Parsed 6000000 input reads 2021-03-10 19:55:05,959 INFO Written out 5300000 reads 2021-03-10 19:55:11,307 INFO Written out 5400000 reads 2021-03-10 19:55:16,733 INFO Written out 5500000 reads 2021-03-10 19:56:45,597 INFO Written out 5600000 reads 2021-03-10 19:56:51,302 INFO Written out 5700000 reads 2021-03-10 19:56:56,696 INFO Written out 5800000 reads 2021-03-10 19:57:02,004 INFO Written out 5900000 reads 2021-03-10 19:57:07,366 INFO Written out 6000000 reads 2021-03-10 19:57:12,667 INFO Written out 6100000 reads 2021-03-10 19:57:13,041 INFO Parsed 7000000 input reads 2021-03-10 19:57:17,973 INFO Written out 6200000 reads 2021-03-10 19:57:23,266 INFO Written out 6300000 reads 2021-03-10 19:57:28,545 INFO Written out 6400000 reads 2021-03-10 19:57:33,880 INFO Written out 6500000 reads 2021-03-10 19:57:39,219 INFO Written out 6600000 reads 2021-03-10 19:57:44,959 INFO Written out 6700000 reads 2021-03-10 19:57:50,360 INFO Written out 6800000 reads 2021-03-10 19:57:55,805 INFO Written out 6900000 reads 2021-03-10 19:58:00,060 INFO Parsed 8000000 input reads 2021-03-10 19:58:01,370 INFO Written out 7000000 reads 2021-03-10 20:00:36,565 INFO Written out 7100000 reads 2021-03-10 20:00:41,930 INFO Written out 7200000 reads 2021-03-10 20:00:47,224 INFO Written out 7300000 reads 2021-03-10 20:00:52,516 INFO Written out 7400000 reads 2021-03-10 20:00:57,944 INFO Written out 7500000 reads 2021-03-10 20:01:03,351 INFO Written out 7600000 reads 2021-03-10 20:01:08,628 INFO Written out 7700000 reads 2021-03-10 20:01:14,061 INFO Written out 7800000 reads 2021-03-10 20:01:16,848 INFO Parsed 9000000 input reads 2021-03-10 20:01:19,388 INFO Written out 7900000 reads 2021-03-10 20:01:24,791 INFO Written out 8000000 reads 2021-03-10 20:01:30,203 INFO Written out 8100000 reads 2021-03-10 20:01:35,507 INFO Written out 8200000 reads 2021-03-10 20:01:40,763 INFO Written out 8300000 reads 2021-03-10 20:01:46,061 INFO Written out 8400000 reads 2021-03-10 20:01:51,407 INFO Written out 8500000 reads 2021-03-10 20:01:56,700 INFO Written out 8600000 reads 2021-03-10 20:02:01,942 INFO Written out 8700000 reads 2021-03-10 20:02:03,147 INFO Parsed 10000000 input reads 2021-03-10 20:02:07,246 INFO Written out 8800000 reads 2021-03-10 20:02:12,541 INFO Written out 8900000 reads 2021-03-10 20:02:28,950 INFO Written out 9000000 reads 2021-03-10 20:04:35,771 INFO Written out 9100000 reads 2021-03-10 20:04:40,998 INFO Written out 9200000 reads 2021-03-10 20:04:46,036 INFO Written out 9300000 reads 2021-03-10 20:04:51,090 INFO Written out 9400000 reads 2021-03-10 20:04:56,152 INFO Written out 9500000 reads 2021-03-10 20:05:01,327 INFO Written out 9600000 reads 2021-03-10 20:05:01,327 INFO Parsed 11000000 input reads 2021-03-10 20:05:06,504 INFO Written out 9700000 reads 2021-03-10 20:05:11,564 INFO Written out 9800000 reads 2021-03-10 20:05:16,616 INFO Written out 9900000 reads 2021-03-10 20:05:21,766 INFO Written out 10000000 reads 2021-03-10 20:05:27,010 INFO Written out 10100000 reads 2021-03-10 20:05:32,210 INFO Written out 10200000 reads 2021-03-10 20:05:38,144 INFO Written out 10300000 reads 2021-03-10 20:05:43,453 INFO Written out 10400000 reads 2021-03-10 20:05:51,834 INFO Parsed 12000000 input reads 2021-03-10 20:05:53,446 INFO Written out 10500000 reads 2021-03-10 20:05:58,701 INFO Written out 10600000 reads 2021-03-10 20:06:04,222 INFO Written out 10700000 reads 2021-03-10 20:06:09,472 INFO Written out 10800000 reads 2021-03-10 20:06:14,769 INFO Written out 10900000 reads 2021-03-10 20:06:20,086 INFO Written out 11000000 reads 2021-03-10 20:06:25,442 INFO Written out 11100000 reads 2021-03-10 20:06:30,805 INFO Written out 11200000 reads 2021-03-10 20:06:36,174 INFO Written out 11300000 reads 2021-03-10 20:06:38,302 INFO Parsed 13000000 input reads 2021-03-10 20:06:41,599 INFO Written out 11400000 reads 2021-03-10 20:06:47,136 INFO Written out 11500000 reads 2021-03-10 20:06:52,615 INFO Written out 11600000 reads 2021-03-10 20:06:58,030 INFO Written out 11700000 reads 2021-03-10 20:07:03,469 INFO Written out 11800000 reads 2021-03-10 20:07:08,883 INFO Written out 11900000 reads 2021-03-10 20:07:14,305 INFO Written out 12000000 reads 2021-03-10 20:08:44,537 INFO Written out 12100000 reads 2021-03-10 20:08:49,606 INFO Written out 12200000 reads 2021-03-10 20:08:50,427 INFO Parsed 14000000 input reads 2021-03-10 20:08:55,136 INFO Written out 12300000 reads 2021-03-10 20:09:00,206 INFO Written out 12400000 reads 2021-03-10 20:09:05,314 INFO Written out 12500000 reads 2021-03-10 20:09:10,545 INFO Written out 12600000 reads 2021-03-10 20:09:15,718 INFO Written out 12700000 reads 2021-03-10 20:09:21,014 INFO Written out 12800000 reads 2021-03-10 20:09:26,257 INFO Written out 12900000 reads 2021-03-10 20:09:31,749 INFO Written out 13000000 reads 2021-03-10 20:09:36,237 INFO Parsed 15000000 input reads 2021-03-10 20:09:37,033 INFO Written out 13100000 reads 2021-03-10 20:09:42,371 INFO Written out 13200000 reads 2021-03-10 20:25:14,603 INFO Written out 13300000 reads 2021-03-10 20:25:20,001 INFO Written out 13400000 reads 2021-03-10 20:25:25,302 INFO Written out 13500000 reads 2021-03-10 20:25:30,599 INFO Written out 13600000 reads 2021-03-10 20:25:36,133 INFO Written out 13700000 reads 2021-03-10 20:25:41,504 INFO Written out 13800000 reads 2021-03-10 20:25:46,889 INFO Written out 13900000 reads 2021-03-10 20:25:50,291 INFO Parsed 16000000 input reads 2021-03-10 20:27:16,138 INFO Written out 14000000 reads 2021-03-10 20:27:21,344 INFO Written out 14100000 reads 2021-03-10 20:27:26,533 INFO Written out 14200000 reads 2021-03-10 20:27:31,717 INFO Written out 14300000 reads 2021-03-10 20:27:36,897 INFO Written out 14400000 reads 2021-03-10 20:27:42,168 INFO Written out 14500000 reads 2021-03-10 20:27:47,342 INFO Written out 14600000 reads 2021-03-10 20:27:52,550 INFO Written out 14700000 reads 2021-03-10 20:27:57,864 INFO Written out 14800000 reads 2021-03-10 20:27:59,287 INFO Parsed 17000000 input reads 2021-03-10 20:28:03,157 INFO Written out 14900000 reads 2021-03-10 20:28:08,516 INFO Written out 15000000 reads 2021-03-10 20:28:13,766 INFO Written out 15100000 reads 2021-03-10 20:28:19,000 INFO Written out 15200000 reads 2021-03-10 20:28:24,278 INFO Written out 15300000 reads 2021-03-10 20:28:29,615 INFO Written out 15400000 reads 2021-03-10 20:28:34,915 INFO Written out 15500000 reads 2021-03-10 20:28:40,517 INFO Written out 15600000 reads 2021-03-10 20:28:45,838 INFO Written out 15700000 reads 2021-03-10 20:28:46,068 INFO Parsed 18000000 input reads 2021-03-10 20:28:51,240 INFO Written out 15800000 reads 2021-03-10 20:30:05,122 INFO Written out 15900000 reads 2021-03-10 20:30:10,243 INFO Written out 16000000 reads 2021-03-10 20:30:15,333 INFO Written out 16100000 reads 2021-03-10 20:30:20,694 INFO Written out 16200000 reads 2021-03-10 20:30:25,794 INFO Written out 16300000 reads 2021-03-10 20:30:30,929 INFO Written out 16400000 reads 2021-03-10 20:30:36,093 INFO Written out 16500000 reads 2021-03-10 20:30:39,840 INFO Parsed 19000000 input reads 2021-03-10 20:30:41,262 INFO Written out 16600000 reads 2021-03-10 20:30:46,455 INFO Written out 16700000 reads 2021-03-10 20:30:51,680 INFO Written out 16800000 reads 2021-03-10 20:30:56,917 INFO Written out 16900000 reads 2021-03-10 20:31:02,584 INFO Written out 17000000 reads 2021-03-10 20:31:07,925 INFO Written out 17100000 reads 2021-03-10 20:31:13,242 INFO Written out 17200000 reads 2021-03-10 20:31:18,552 INFO Written out 17300000 reads 2021-03-10 20:32:32,733 INFO Written out 17400000 reads 2021-03-10 20:32:35,291 INFO Parsed 20000000 input reads 2021-03-10 20:32:37,853 INFO Written out 17500000 reads 2021-03-10 20:32:46,095 INFO Written out 17600000 reads 2021-03-10 20:32:51,209 INFO Written out 17700000 reads 2021-03-10 20:32:56,414 INFO Written out 17800000 reads 2021-03-10 20:33:02,661 INFO Written out 17900000 reads 2021-03-10 20:33:08,106 INFO Written out 18000000 reads 2021-03-10 20:33:13,302 INFO Written out 18100000 reads 2021-03-10 20:34:06,261 INFO Written out 18200000 reads 2021-03-10 20:34:11,590 INFO Written out 18300000 reads 2021-03-10 20:34:12,558 INFO Parsed 21000000 input reads 2021-03-10 20:34:17,033 INFO Written out 18400000 reads 2021-03-10 20:34:22,327 INFO Written out 18500000 reads 2021-03-10 20:34:27,842 INFO Written out 18600000 reads 2021-03-10 20:35:37,354 INFO Written out 18700000 reads 2021-03-10 20:35:42,674 INFO Written out 18800000 reads 2021-03-10 20:35:47,967 INFO Written out 18900000 reads 2021-03-10 20:35:53,271 INFO Written out 19000000 reads 2021-03-10 20:35:58,674 INFO Written out 19100000 reads 2021-03-10 20:36:03,225 INFO Parsed 22000000 input reads 2021-03-10 20:36:03,921 INFO Written out 19200000 reads 2021-03-10 20:36:09,232 INFO Written out 19300000 reads 2021-03-10 20:36:14,508 INFO Written out 19400000 reads 2021-03-10 20:36:19,788 INFO Written out 19500000 reads 2021-03-10 20:36:25,137 INFO Written out 19600000 reads 2021-03-10 20:36:30,472 INFO Written out 19700000 reads 2021-03-10 20:36:35,858 INFO Written out 19800000 reads 2021-03-10 20:38:00,609 INFO Written out 19900000 reads 2021-03-10 20:38:07,171 INFO Written out 20000000 reads 2021-03-10 20:38:10,456 INFO Parsed 23000000 input reads 2021-03-10 20:38:12,307 INFO Written out 20100000 reads 2021-03-10 20:38:17,446 INFO Written out 20200000 reads 2021-03-10 20:38:22,635 INFO Written out 20300000 reads 2021-03-10 20:38:28,120 INFO Written out 20400000 reads 2021-03-10 20:38:33,472 INFO Written out 20500000 reads 2021-03-10 20:38:38,708 INFO Written out 20600000 reads 2021-03-10 20:38:44,095 INFO Written out 20700000 reads 2021-03-10 20:38:49,418 INFO Written out 20800000 reads 2021-03-10 20:38:55,732 INFO Written out 20900000 reads 2021-03-10 20:38:57,545 INFO Parsed 24000000 input reads 2021-03-10 20:39:01,134 INFO Written out 21000000 reads 2021-03-10 20:40:41,174 INFO Written out 21100000 reads 2021-03-10 20:40:46,609 INFO Written out 21200000 reads 2021-03-10 20:42:00,884 INFO Written out 21300000 reads 2021-03-10 20:42:05,997 INFO Written out 21400000 reads 2021-03-10 20:42:11,081 INFO Written out 21500000 reads 2021-03-10 20:42:16,285 INFO Written out 21600000 reads 2021-03-10 20:42:21,426 INFO Written out 21700000 reads 2021-03-10 20:42:26,567 INFO Written out 21800000 reads 2021-03-10 20:42:27,131 INFO Parsed 25000000 input reads 2021-03-10 20:42:31,922 INFO Written out 21900000 reads 2021-03-10 20:42:43,719 INFO Written out 22000000 reads 2021-03-10 20:42:48,986 INFO Written out 22100000 reads 2021-03-10 20:42:54,335 INFO Written out 22200000 reads 2021-03-10 20:42:59,677 INFO Written out 22300000 reads 2021-03-10 20:43:05,044 INFO Written out 22400000 reads 2021-03-10 20:43:10,501 INFO Written out 22500000 reads 2021-03-10 20:43:15,935 INFO Written out 22600000 reads 2021-03-10 20:43:20,273 INFO Parsed 26000000 input reads 2021-03-10 20:43:21,405 INFO Written out 22700000 reads 2021-03-10 20:43:26,869 INFO Written out 22800000 reads 2021-03-10 20:44:42,592 INFO Written out 22900000 reads 2021-03-10 20:44:47,774 INFO Written out 23000000 reads 2021-03-10 20:44:52,952 INFO Written out 23100000 reads 2021-03-10 20:44:58,134 INFO Written out 23200000 reads 2021-03-10 20:45:03,318 INFO Written out 23300000 reads 2021-03-10 20:45:08,525 INFO Written out 23400000 reads 2021-03-10 20:45:13,947 INFO Written out 23500000 reads 2021-03-10 20:45:16,904 INFO Parsed 27000000 input reads 2021-03-10 20:45:19,206 INFO Written out 23600000 reads 2021-03-10 20:45:24,487 INFO Written out 23700000 reads 2021-03-10 20:45:29,844 INFO Written out 23800000 reads 2021-03-10 20:45:35,491 INFO Written out 23900000 reads 2021-03-10 20:46:29,342 INFO Written out 24000000 reads 2021-03-10 20:46:34,525 INFO Written out 24100000 reads 2021-03-10 20:46:39,879 INFO Written out 24200000 reads 2021-03-10 20:46:45,080 INFO Written out 24300000 reads 2021-03-10 20:46:50,310 INFO Written out 24400000 reads 2021-03-10 20:46:51,685 INFO Parsed 28000000 input reads 2021-03-10 20:46:55,585 INFO Written out 24500000 reads 2021-03-10 20:47:00,930 INFO Written out 24600000 reads 2021-03-10 20:47:06,457 INFO Written out 24700000 reads 2021-03-10 20:47:56,250 INFO Written out 24800000 reads 2021-03-10 20:48:01,340 INFO Written out 24900000 reads 2021-03-10 20:48:06,511 INFO Written out 25000000 reads 2021-03-10 20:48:11,700 INFO Written out 25100000 reads 2021-03-10 20:53:25,256 INFO Written out 25200000 reads 2021-03-10 20:53:35,209 INFO Written out 25300000 reads 2021-03-10 20:53:35,215 INFO Parsed 29000000 input reads 2021-03-10 20:53:44,602 INFO Written out 25400000 reads 2021-03-10 20:55:00,534 INFO Written out 25500000 reads 2021-03-10 20:55:50,619 INFO Written out 25600000 reads 2021-03-10 20:55:56,205 INFO Written out 25700000 reads 2021-03-10 20:56:01,574 INFO Written out 25800000 reads 2021-03-10 20:56:09,328 INFO Written out 25900000 reads 2021-03-10 21:01:50,734 INFO Written out 26000000 reads 2021-03-10 21:01:56,039 INFO Written out 26100000 reads 2021-03-10 21:01:59,516 INFO Parsed 30000000 input reads 2021-03-10 21:02:01,652 INFO Written out 26200000 reads 2021-03-10 21:02:07,088 INFO Written out 26300000 reads 2021-03-10 21:03:04,032 INFO Written out 26400000 reads 2021-03-10 21:03:13,665 INFO Written out 26500000 reads 2021-03-10 21:03:18,896 INFO Written out 26600000 reads 2021-03-10 21:03:24,301 INFO Written out 26700000 reads 2021-03-10 21:03:29,542 INFO Written out 26800000 reads 2021-03-10 21:03:34,896 INFO Written out 26900000 reads 2021-03-10 21:04:26,780 INFO Written out 27000000 reads 2021-03-10 21:04:28,246 INFO Parsed 31000000 input reads 2021-03-10 21:04:31,997 INFO Written out 27100000 reads 2021-03-10 21:04:37,196 INFO Written out 27200000 reads 2021-03-10 21:04:42,963 INFO Written out 27300000 reads 2021-03-10 21:04:48,396 INFO Written out 27400000 reads 2021-03-10 21:04:53,731 INFO Written out 27500000 reads 2021-03-10 21:04:59,841 INFO Written out 27600000 reads 2021-03-10 21:05:20,022 INFO Written out 27700000 reads 2021-03-10 21:05:25,322 INFO Written out 27800000 reads 2021-03-10 21:05:30,580 INFO Written out 27900000 reads 2021-03-10 21:05:30,895 INFO Parsed 32000000 input reads 2021-03-10 21:06:36,976 INFO Written out 28000000 reads 2021-03-10 21:06:42,189 INFO Written out 28100000 reads 2021-03-10 21:06:47,425 INFO Written out 28200000 reads 2021-03-10 21:06:52,669 INFO Written out 28300000 reads 2021-03-10 21:06:58,012 INFO Written out 28400000 reads 2021-03-10 21:07:03,415 INFO Written out 28500000 reads 2021-03-10 21:07:31,927 INFO Written out 28600000 reads 2021-03-10 21:07:37,058 INFO Written out 28700000 reads 2021-03-10 21:07:40,903 INFO Parsed 33000000 input reads 2021-03-10 21:07:42,232 INFO Written out 28800000 reads 2021-03-10 21:07:47,835 INFO Written out 28900000 reads 2021-03-10 21:09:53,573 INFO Parsed 34000000 input reads 2021-03-10 21:09:53,573 INFO Parsed 35000000 input reads 2021-03-10 21:09:53,573 INFO Parsed 36000000 input reads Traceback (most recent call last): File "/opt/conda/bin/umi_tools", line 11, in
sys.exit(main())
File "/opt/conda/lib/python3.7/site-packages/umi_tools/umi_tools.py", line 61, in main
module.main(sys.argv)
File "/opt/conda/lib/python3.7/site-packages/umi_tools/dedup.py", line 273, in main
average_distance = umi_methods.get_average_umi_distance(bundle.keys())
File "/opt/conda/lib/python3.7/site-packages/umi_tools/umi_methods.py", line 133, in get_average_umi_distance
x, y in itertools.combinations(umis, 2)]
File "/opt/conda/lib/python3.7/site-packages/umi_tools/umi_methods.py", line 133, in
x, y in itertools.combinations(umis, 2)]
MemoryError
INFO [job X_barcodecollapsese_2] Max memory used: 113916MiB
ERROR [job X_barcodecollapsese_2] Job error:
("Error collecting output for parameter 'output_barcodecollapsese_metrics':\ncwl/barcodecollapse_se.cwl:97:7: Did not find output file with glob pattern: '['2020.9.29.eclip.pilot.rbm15.test.RBM15.input.umi.r1.fq.genome-mappedSoSo_per_umi.tsv']'", {})
WARNING [job X_barcodecollapsese_2] completed permanentFail