YaqiangCao
commented
4 years ago Dear User, May I ask how many memories do you have in the computer? One problem I notice in your parameters is -minPts 0, please assign a big value, for your data, maybe Hi-C, such as 30 or 50. In the log information, your total PETs is about 297 million. I guess your data is Hi-C. In my experience, 300 million PETs Hi-C, for chr1, it will cost about 30G memory. Here your -p is 16, maybe there is out-of-memory issue. I suggest run with -cut 10000 to remove close PETs for calling loops, and reduce -p, which can reduce the memory usage. Best, Yaqiang
On Wed, Jul 15, 2020 at 7:55 AM Roberto Vera Alvarez < notifications@github.com> wrote:
Hi, I'm running cLopps with 4 bedpe files and it hang for more than 24 hours without consuming any resource in the computer.
This is my output:
$ cLoops \ -f CHLA_1_S1_L001_R1_001.fastq_sorted_fixmate.bedpe.gz,CHLA_1_S1_L002_R1_001.fastq_sorted_fixmate.bedpe.gz,CHLA_2_S4_L001_R1_001.fastq_sorted_fixmate.bedpe.gz,CHLA_2_S4_L002_R1_001.fastq_sorted_fixmate.bedpe.gz \ -o \ CHLA \ -m \ 3 \ -eps \ 5000,7500,10000 \ -minPts \ 0 \ -p \ 16 \ -c \ chr1,chr2,chr3,chr4,chr5,chr6,chr7,chr8,chr9,chr10,chr11,chr12,chr13,chr14,chr15,chr16,chr17,chr18,chr19,chrX,chrY \ -w \ -j \ -s \ -hic \ -plot 2020-07-14 13:20:43,940 INFO Command line: cLoops -f /tmp/tmpzvcg8_fw/stgbab360e9-91ca-4353-bd46-e1c0a23caa3e/CHLA_1_S1_L001_R1_001.fastq_sorted_fixmate.bedpe.gz,/tmp/tmpzvcg8_fw/stg60cf6a5f-dfa9-4ec1-9dd8-254e57420dfa/CHLA_1_S1_L002_R1_001.fastq_sorted_fixmate.bedpe.gz,/tmp/tmpzvcg8_fw/stgbd15da67-02ed-4e88-99b6-1aab652edea7/CHLA_2_S4_L001_R1_001.fastq_sorted_fixmate.bedpe.gz,/tmp/tmpzvcg8_fw/stg04f31652-6ef1-4445-ac9b-da8de561bffe/CHLA_2_S4_L002_R1_001.fastq_sorted_fixmate.bedpe.gz -o CHLA -m 3 -eps 5000,7500,10000 -minPts 0 -p 16 -w True -j True -s True -c chr1,chr2,chr3,chr4,chr5,chr6,chr7,chr8,chr9,chr10,chr11,chr12,chr13,chr14,chr15,chr16,chr17,chr18,chr19,chrX,chrY -hic True -cut 0 -plot True -max_cut False 2020-07-14 13:20:43,940 INFO mode:3 eps:[5000, 7500, 10000] minPts:[50, 40, 30, 20] hic:1 2020-07-14 13:20:43,940 INFO Parsing PETs from /tmp/tmpzvcg8_fw/stgbab360e9-91ca-4353-bd46-e1c0a23caa3e/CHLA_1_S1_L001_R1_001.fastq_sorted_fixmate.bedpe.gz, requiring initial distance cutoff > 0 92700000 PETs processed from /tmp/tmpzvcg8_fw/stgbab360e9-91ca-4353-bd46-e1c0a23caa3e/CHLA_1_S1_L001_R1_001.fastq_sorted_fixmate.bedpe.gz2020-07-14 13:31:55,055 INFO Parsing PETs from /tmp/tmpzvcg8_fw/stg60cf6a5f-dfa9-4ec1-9dd8-254e57420dfa/CHLA_1_S1_L002_R1_001.fastq_sorted_fixmate.bedpe.gz, requiring initial distance cutoff > 0 184800000 PETs processed from /tmp/tmpzvcg8_fw/stg60cf6a5f-dfa9-4ec1-9dd8-254e57420dfa/CHLA_1_S1_L002_R1_001.fastq_sorted_fixmate.bedpe.gz2020-07-14 13:43:04,119 INFO Parsing PETs from /tmp/tmpzvcg8_fw/stgbd15da67-02ed-4e88-99b6-1aab652edea7/CHLA_2_S4_L001_R1_001.fastq_sorted_fixmate.bedpe.gz, requiring initial distance cutoff > 0 241200000 PETs processed from /tmp/tmpzvcg8_fw/stgbd15da67-02ed-4e88-99b6-1aab652edea7/CHLA_2_S4_L001_R1_001.fastq_sorted_fixmate.bedpe.gz2020-07-14 13:49:51,955 INFO Parsing PETs from /tmp/tmpzvcg8_fw/stg04f31652-6ef1-4445-ac9b-da8de561bffe/CHLA_2_S4_L002_R1_001.fastq_sorted_fixmate.bedpe.gz, requiring initial distance cutoff > 0 297100000 PETs processed from /tmp/tmpzvcg8_fw/stg04f31652-6ef1-4445-ac9b-da8de561bffe/CHLA_2_S4_L002_R1_001.fastq_sorted_fixmate.bedpe.gz() 2020-07-14 13:56:39,257 INFO Totaly 297190142 PETs from /tmp/tmpzvcg8_fw/stgbab360e9-91ca-4353-bd46-e1c0a23caa3e/CHLA_1_S1_L001_R1_001.fastq_sorted_fixmate.bedpe.gz,/tmp/tmpzvcg8_fw/stg60cf6a5f-dfa9-4ec1-9dd8-254e57420dfa/CHLA_1_S1_L002_R1_001.fastq_sorted_fixmate.bedpe.gz,/tmp/tmpzvcg8_fw/stgbd15da67-02ed-4e88-99b6-1aab652edea7/CHLA_2_S4_L001_R1_001.fastq_sorted_fixmate.bedpe.gz,/tmp/tmpzvcg8_fw/stg04f31652-6ef1-4445-ac9b-da8de561bffe/CHLA_2_S4_L002_R1_001.fastq_sorted_fixmate.bedpe.gz, in which 231457356 cis PETs Clustering chrX and chrX using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr13 and chr13 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr18 and chr18 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr19 and chr19 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr15 and chr15 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr9 and chr9 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr10 and chr10 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr11 and chr11 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr8 and chr8 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr4 and chr4 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr17 and chr17 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr7 and chr7 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr6 and chr6 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr5 and chr5 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr2 and chr2 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr1 and chr1 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr18 and chr18 finished. Estimated 4159373 self-ligation reads and 4787 inter-ligation reads Clustering chrX and chrX finished. Estimated 3892349 self-ligation reads and 30128 inter-ligation reads Clustering chr12 and chr12 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr13 and chr13 finished. Estimated 4807700 self-ligation reads and 23345 inter-ligation reads Clustering chr14 and chr14 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr16 and chr16 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr15 and chr15 finished. Estimated 6001390 self-ligation reads and 148954 inter-ligation reads Clustering chr19 and chr19 finished. Estimated 4936452 self-ligation reads and 31579 inter-ligation reads Clustering chr3 and chr3 using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr4 and chr4 finished. Estimated 8868202 self-ligation reads and 23993 inter-ligation reads Clustering chrY and chrY using eps as 5000, minPts as 50,pre-set distance cutoff as > 0 Clustering chr9 and chr9 finished. Estimated 7327541 self-ligation reads and 231139 inter-ligation reads Clustering chr11 and chr11 finished. Estimated 8249398 self-ligation reads and 8235 inter-ligation reads Clustering chrY and chrY finished. Estimated 460686 self-ligation reads and 60557 inter-ligation reads Clustering chr5 and chr5 finished. Estimated 9604772 self-ligation reads and 100824 inter-ligation reads Clustering chr6 and chr6 finished. Estimated 9308583 self-ligation reads and 15008 inter-ligation reads Clustering chr7 and chr7 finished. Estimated 9012988 self-ligation reads and 138554 inter-ligation reads Clustering chr10 and chr10 finished. Estimated 8305044 self-ligation reads and 71578 inter-ligation reads Clustering chr14 and chr14 finished. Estimated 5752252 self-ligation reads and 22969 inter-ligation reads Clustering chr8 and chr8 finished. Estimated 8440548 self-ligation reads and 125033 inter-ligation reads Clustering chr17 and chr17 finished. Estimated 9138294 self-ligation reads and 171760 inter-ligation reads Clustering chr2 and chr2 finished. Estimated 14067151 self-ligation reads and 108679 inter-ligation reads Clustering chr16 and chr16 finished. Estimated 5992621 self-ligation reads and 232902 inter-ligation reads Clustering chr3 and chr3 finished. Estimated 11099243 self-ligation reads and 7972 inter-ligation reads Clustering chr12 and chr12 finished. Estimated 12273217 self-ligation reads and 20282 inter-ligation reads ^CProcess PoolWorker-21: Process PoolWorker-25: Process PoolWorker-28: Process PoolWorker-23: Process PoolWorker-19: Process PoolWorker-22: Process PoolWorker-27: Process PoolWorker-32: Process PoolWorker-31: Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Process PoolWorker-20: File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Traceback (most recent call last): Traceback (most recent call last): File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Process PoolWorker-24: File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Traceback (most recent call last): Process PoolWorker-17: File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Traceback (most recent call last): File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Process PoolWorker-33: Process PoolWorker-26: Traceback (most recent call last): File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Process PoolWorker-30: Traceback (most recent call last): Traceback (most recent call last): File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap Process PoolWorker-29: Traceback (most recent call last): File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() self.run() self.run() self.run() self.run() self.run() File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, *self._kwargs) self.run() self._target(self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker self._target(*self._args, self._kwargs) self._target(*self._args, *self._kwargs) self._target(self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker self._target(*self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker self.run() self.run() self.run() self.run() File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, *self._kwargs) self._target(self._args, self._kwargs) self._target(*self._args, self._kwargs) self.run() self._target(*self._args, *self._kwargs) self._target(self._args, self._kwargs) self._target(*self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker self._target(*self._args, *self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker self.run() File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker self.run() File "/bin/cloops/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, self._kwargs) File "/bin/cloops/lib/python2.7/multiprocessing/pool.py", line 102, in worker task = get() task = get() task = get() task = get() File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get task = get() task = get() task = get() task = get() task = get() task = get() task = get() task = get() task = get() File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get task = get() File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get task = get() File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 360, in get task = get() File "/bin/cloops/lib/python2.7/site-packages/joblib/pool.py", line 362, in get racquire() racquire() racquire() racquire() KeyboardInterrupt racquire() racquire() racquire() racquire() racquire() racquire() racquire() racquire() racquire() KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt racquire() KeyboardInterrupt racquire() KeyboardInterrupt return recv() KeyboardInterrupt Traceback (most recent call last): File "/bin/jupyter/bin/cwltool", line 10, in
sys.exit(run()) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/main.py", line 1240, in run sys.exit(main(*args, *kwargs)) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/main.py", line 1105, in main tool, initialized_job_order_object, runtimeContext, logger=_logger File "/bin/jupyter/lib/python3.7/site-packages/cwltool/executors.py", line 54, in call return self.execute(args, kwargs) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/executors.py", line 137, in execute self.run_jobs(process, job_order_object, logger, runtime_context) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/executors.py", line 244, in run_jobs job.run(runtime_context) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/job.py", line 570, in run self._execute([], env, runtimeContext, monitor_function) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/job.py", line 373, in _execute monitor_function=monitor_function, File "/bin/jupyter/lib/python3.7/site-packages/cwltool/job.py", line 957, in _job_popen monitor_function(sproc) File "/bin/jupyter/lib/python3.7/site-packages/cwltool/job.py", line 497, in process_monitor sproc.wait() File "/bin/jupyter/lib/python3.7/subprocess.py", line 1019, in wait return self._wait(timeout=timeout) File "/bin/jupyter/lib/python3.7/subprocess.py", line 1653, in _wait (pid, sts) = self._try_wait(0) File "/bin/jupyter/lib/python3.7/subprocess.py", line 1611, in _try_wait (pid, sts) = os.waitpid(self.pid, wait_flags) KeyboardInterrupt— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/21, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKMQOYPPUOEKNYTOPODR3WKLTANCNFSM4O2OVHAA .
Hi, I'm running cLopps with 4 bedpe files and it hang for more than 24 hours without consuming any resource in the computer.
This is my output: