Closed hodgett closed 6 years ago
I should be able to reproduce the error if I have .fasta files for the two read blocks. You can produce these as follows:
DBshow raw_reads.56 >block.56.fasta DBshow raw_reads.213 >block.213.fasta
The files are likely to be 300-400Mbp each depending on how you split the DB, so you'll need to make them available with something like dropbox or a public ftp download.
Best, Gene
On 1/25/17, 1:33 AM, hodgett wrote:
[ from https://github.com/PacificBiosciences/FALCON/issues/515] We are using Falcon-integrate 1.8.6 and for our large sequences we are still getting the occasional Duplicate Overlap that is killing the whole process.
raw_reads.55.raw_reads.213.N3: 324,318 all OK raw_reads.56.raw_reads.213.C0: 332,440 all OK raw_reads.56.raw_reads.213.C1: Duplicate overlap (1573849 vs 6403244) raw_reads.56.raw_reads.213.C2: 325,068 all OK raw_reads.56.raw_reads.213.C3: 336,228 all OK Based on our fc_run.cfg config are there any suggestions? We have tried reducing -s 100 and increasing the cutoff but still get the problem. We actually need to reduce the cutoff.
|[General]
config file for FALCON v1.8.6
job_type = local
job_type = pbs input_fofn = input.fofn input_type = raw length_cutoff = 5000 length_cutoff_pr = 5000 job_queue = lyra
sge_option = -l nodes=1:ppn=2,walltime=128:00:00,mem=20gb -W umask=0007 sge_option_da = -l nodes=1:ppn=4,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_la = -l nodes=1:ppn=2,walltime=96:00:00,mem=24gb -W umask=0007 sge_option_pda = -l nodes=1:ppn=8,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_pla = -l nodes=1:ppn=2,walltime=96:00:00,mem=26gb -W umask=0007 sge_option_fc = -l nodes=1:ppn=1,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_cns = -l nodes=1:ppn=8,walltime=96:00:00,mem=24gb -W umask=0007
pa_concurrent_jobs = 8 cns_concurrent_jobs = 8 ovlp_concurrent_jobs = 8
pa_HPCdaligner_option = -v -B128 -e.70 -l1000 -s1000 -M24 ovlp_HPCdaligner_option = -v -B128 -h60 -e.96 -l500 -s1000 -M24
pa_DBsplit_option = -x500 -s300 ovlp_DBsplit_option = -x500 -s300
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 0 overlap_filtering_setting = --max_diff 80 --max_cov 80 --min_cov 2 --bestn 10 --n_core 8 |
pc-dunn says "If this is on the latest DALIGNER, it is a serious problem. I suggest providing a test-case to thegenemyers, the owner of DALIGNER." What information can I provide to assist resolve this, keeping in mind that our dataset is over 100Gb.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55, or mute the thread https://github.com/notifications/unsubscribe-auth/AGkkNn8ZSs-B6Qp7BHtMDYtrhQpPfPr7ks5rVphjgaJpZM4Ls9rS.
Gene,
I am uploading these blocks as they are from a more current attempt; raw_reads.122.raw_reads.134.C2: Duplicate overlap (3591039 vs 3951857) raw_reads.134.raw_reads.122.C2: Duplicate overlap (3951857 vs 3591039)
https://cloudstor.aarnet.edu.au/plus/index.php/s/vPncp5WgHqbwQuC https://cloudstor.aarnet.edu.au/plus/index.php/s/MclohoOn7WRdjvB
Let me know if you have any trouble getting the files and I hope this helps.
Regards,
Matt
On 25/01/17 16:40, Eugene W Myers Jr wrote:
I should be able to reproduce the error if I have .fasta files for the two read blocks. You can produce these as follows:
DBshow raw_reads.56 >block.56.fasta DBshow raw_reads.213 >block.213.fasta
The files are likely to be 300-400Mbp each depending on how you split the DB, so you'll need to make them available with something like dropbox or a public ftp download.
Best, Gene
On 1/25/17, 1:33 AM, hodgett wrote:
[ from https://github.com/PacificBiosciences/FALCON/issues/515] We are using Falcon-integrate 1.8.6 and for our large sequences we are still getting the occasional Duplicate Overlap that is killing the whole process.
raw_reads.55.raw_reads.213.N3: 324,318 all OK raw_reads.56.raw_reads.213.C0: 332,440 all OK raw_reads.56.raw_reads.213.C1: Duplicate overlap (1573849 vs 6403244) raw_reads.56.raw_reads.213.C2: 325,068 all OK raw_reads.56.raw_reads.213.C3: 336,228 all OK Based on our fc_run.cfg config are there any suggestions? We have tried reducing -s 100 and increasing the cutoff but still get the problem. We actually need to reduce the cutoff.
|[General]
config file for FALCON v1.8.6
job_type = local
job_type = pbs input_fofn = input.fofn input_type = raw length_cutoff = 5000 length_cutoff_pr = 5000 job_queue = lyra
sge_option = -l nodes=1:ppn=2,walltime=128:00:00,mem=20gb -W umask=0007 sge_option_da = -l nodes=1:ppn=4,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_la = -l nodes=1:ppn=2,walltime=96:00:00,mem=24gb -W umask=0007 sge_option_pda = -l nodes=1:ppn=8,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_pla = -l nodes=1:ppn=2,walltime=96:00:00,mem=26gb -W umask=0007 sge_option_fc = -l nodes=1:ppn=1,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_cns = -l nodes=1:ppn=8,walltime=96:00:00,mem=24gb -W umask=0007
pa_concurrent_jobs = 8 cns_concurrent_jobs = 8 ovlp_concurrent_jobs = 8
pa_HPCdaligner_option = -v -B128 -e.70 -l1000 -s1000 -M24 ovlp_HPCdaligner_option = -v -B128 -h60 -e.96 -l500 -s1000 -M24
pa_DBsplit_option = -x500 -s300 ovlp_DBsplit_option = -x500 -s300
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 0 overlap_filtering_setting = --max_diff 80 --max_cov 80 --min_cov 2 --bestn 10 --n_core 8 |
pc-dunn says "If this is on the latest DALIGNER, it is a serious problem. I suggest providing a test-case to thegenemyers, the owner of DALIGNER." What information can I provide to assist resolve this, keeping in mind that our dataset is over 100Gb.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55, or mute the thread
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-275031342, or mute the thread https://github.com/notifications/unsubscribe-auth/AXi0iRNEM-mPMDk_hlB6rne_GZAH4BiDks5rVu5hgaJpZM4Ls9rS.
Great, got both blocks. I'll let you know if I can reproduce the problem at my end soon. Cheers, G
On 1/27/17, 4:33 AM, hodgett wrote:
Gene,
I am uploading these blocks as they are from a more current attempt; raw_reads.122.raw_reads.134.C2: Duplicate overlap (3591039 vs 3951857) raw_reads.134.raw_reads.122.C2: Duplicate overlap (3951857 vs 3591039)
https://cloudstor.aarnet.edu.au/plus/index.php/s/vPncp5WgHqbwQuC https://cloudstor.aarnet.edu.au/plus/index.php/s/MclohoOn7WRdjvB
Let me know if you have any trouble getting the files and I hope this helps.
Regards,
Matt
On 25/01/17 16:40, Eugene W Myers Jr wrote:
I should be able to reproduce the error if I have .fasta files for the two read blocks. You can produce these as follows:
DBshow raw_reads.56 >block.56.fasta DBshow raw_reads.213 >block.213.fasta
The files are likely to be 300-400Mbp each depending on how you split the DB, so you'll need to make them available with something like dropbox or a public ftp download.
Best, Gene
On 1/25/17, 1:33 AM, hodgett wrote:
[ from https://github.com/PacificBiosciences/FALCON/issues/515] We are using Falcon-integrate 1.8.6 and for our large sequences we are still getting the occasional Duplicate Overlap that is killing the whole process.
raw_reads.55.raw_reads.213.N3: 324,318 all OK raw_reads.56.raw_reads.213.C0: 332,440 all OK raw_reads.56.raw_reads.213.C1: Duplicate overlap (1573849 vs 6403244) raw_reads.56.raw_reads.213.C2: 325,068 all OK raw_reads.56.raw_reads.213.C3: 336,228 all OK Based on our fc_run.cfg config are there any suggestions? We have tried reducing -s 100 and increasing the cutoff but still get the problem. We actually need to reduce the cutoff.
|[General]
config file for FALCON v1.8.6
job_type = local
job_type = pbs input_fofn = input.fofn input_type = raw length_cutoff = 5000 length_cutoff_pr = 5000 job_queue = lyra
sge_option = -l nodes=1:ppn=2,walltime=128:00:00,mem=20gb -W umask=0007 sge_option_da = -l nodes=1:ppn=4,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_la = -l nodes=1:ppn=2,walltime=96:00:00,mem=24gb -W umask=0007 sge_option_pda = -l nodes=1:ppn=8,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_pla = -l nodes=1:ppn=2,walltime=96:00:00,mem=26gb -W umask=0007 sge_option_fc = -l nodes=1:ppn=1,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_cns = -l nodes=1:ppn=8,walltime=96:00:00,mem=24gb -W umask=0007
pa_concurrent_jobs = 8 cns_concurrent_jobs = 8 ovlp_concurrent_jobs = 8
pa_HPCdaligner_option = -v -B128 -e.70 -l1000 -s1000 -M24 ovlp_HPCdaligner_option = -v -B128 -h60 -e.96 -l500 -s1000 -M24
pa_DBsplit_option = -x500 -s300 ovlp_DBsplit_option = -x500 -s300
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 0 overlap_filtering_setting = --max_diff 80 --max_cov 80 --min_cov 2 --bestn 10 --n_core 8 |
pc-dunn says "If this is on the latest DALIGNER, it is a serious problem. I suggest providing a test-case to thegenemyers, the owner of DALIGNER." What information can I provide to assist resolve this, keeping in mind that our dataset is over 100Gb.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55, or mute the thread
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub
https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-275031342,
or mute the thread
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-275581906, or mute the thread https://github.com/notifications/unsubscribe-auth/AGkkNjvhZjZrTiAlOMSorjU4oEDljapvks5rWWWZgaJpZM4Ls9rS.
Excellent, thanks. I can provide more blocks if required, just let me know.
Matthew
----- Reply message ----- From: "Eugene W Myers Jr" notifications@github.com To: "thegenemyers/DALIGNER" DALIGNER@noreply.github.com Cc: "Matthew Hodgett" m.hodgett@qut.edu.au, "Author" author@noreply.github.com Subject: [thegenemyers/DALIGNER] Still getting Duplicate overlap (#55) Date: Fri, Jan 27, 2017 6:11 PM
Great, got both blocks. I'll let you know if I can reproduce the problem at my end soon. Cheers, G
On 1/27/17, 4:33 AM, hodgett wrote:
Gene,
I am uploading these blocks as they are from a more current attempt; raw_reads.122.raw_reads.134.C2: Duplicate overlap (3591039 vs 3951857) raw_reads.134.raw_reads.122.C2: Duplicate overlap (3951857 vs 3591039)
https://cloudstor.aarnet.edu.au/plus/index.php/s/vPncp5WgHqbwQuC https://cloudstor.aarnet.edu.au/plus/index.php/s/MclohoOn7WRdjvB
Let me know if you have any trouble getting the files and I hope this helps.
Regards,
Matt
On 25/01/17 16:40, Eugene W Myers Jr wrote:
I should be able to reproduce the error if I have .fasta files for the two read blocks. You can produce these as follows:
DBshow raw_reads.56 >block.56.fasta DBshow raw_reads.213 >block.213.fasta
The files are likely to be 300-400Mbp each depending on how you split the DB, so you'll need to make them available with something like dropbox or a public ftp download.
Best, Gene
On 1/25/17, 1:33 AM, hodgett wrote:
[ from https://github.com/PacificBiosciences/FALCON/issues/515] We are using Falcon-integrate 1.8.6 and for our large sequences we are still getting the occasional Duplicate Overlap that is killing the whole process.
raw_reads.55.raw_reads.213.N3: 324,318 all OK raw_reads.56.raw_reads.213.C0: 332,440 all OK raw_reads.56.raw_reads.213.C1: Duplicate overlap (1573849 vs 6403244) raw_reads.56.raw_reads.213.C2: 325,068 all OK raw_reads.56.raw_reads.213.C3: 336,228 all OK Based on our fc_run.cfg config are there any suggestions? We have tried reducing -s 100 and increasing the cutoff but still get the problem. We actually need to reduce the cutoff.
|[General]
config file for FALCON v1.8.6
job_type = local
job_type = pbs input_fofn = input.fofn input_type = raw length_cutoff = 5000 length_cutoff_pr = 5000 job_queue = lyra
sge_option = -l nodes=1:ppn=2,walltime=128:00:00,mem=20gb -W umask=0007 sge_option_da = -l nodes=1:ppn=4,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_la = -l nodes=1:ppn=2,walltime=96:00:00,mem=24gb -W umask=0007 sge_option_pda = -l nodes=1:ppn=8,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_pla = -l nodes=1:ppn=2,walltime=96:00:00,mem=26gb -W umask=0007 sge_option_fc = -l nodes=1:ppn=1,walltime=96:00:00,mem=25gb -W umask=0007 sge_option_cns = -l nodes=1:ppn=8,walltime=96:00:00,mem=24gb -W umask=0007
pa_concurrent_jobs = 8 cns_concurrent_jobs = 8 ovlp_concurrent_jobs = 8
pa_HPCdaligner_option = -v -B128 -e.70 -l1000 -s1000 -M24 ovlp_HPCdaligner_option = -v -B128 -h60 -e.96 -l500 -s1000 -M24
pa_DBsplit_option = -x500 -s300 ovlp_DBsplit_option = -x500 -s300
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 0 overlap_filtering_setting = --max_diff 80 --max_cov 80 --min_cov 2 --bestn 10 --n_core 8 |
pc-dunn says "If this is on the latest DALIGNER, it is a serious problem. I suggest providing a test-case to thegenemyers, the owner of DALIGNER." What information can I provide to assist resolve this, keeping in mind that our dataset is over 100Gb.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55, or mute the thread
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub
https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-275031342,
or mute the thread
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-275581906, or mute the thread https://github.com/notifications/unsubscribe-auth/AGkkNjvhZjZrTiAlOMSorjU4oEDljapvks5rWWWZgaJpZM4Ls9rS.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-275610646, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AXi0iSPkm6HjV_VHuxt5jbNU6o0TitSIks5rWaadgaJpZM4Ls9rS.
@hodgett @pb-cdunn So far I have not been able to reproduce the problem. Two questions:
(1) Most importantly I note that y'all are not using the most recent version of daligner as the thread files with extensions .C0, .C1, etc. are no longer produced. This change was made about 4 months ago, and I fixed a "duplicate overlap" problem about 5 months ago. This simply could be because the release of falcon you are using does not include the latest version of daligner. This is why I include Chris Dunn in the discussion. Chris: does falcon now use the version that produces a single .las for a block vs block comparison? I note the repository pacbio/dazzler refers to a quite old version of daligner (from 11 months ago).
(2) I split the blocks with -x500 -s300 and I ran daligner with the defaults save for -s1000 -l1000 and most importantly with -M16 as I only have 16Gb on my Mac. When I get home from my vacation next week, I can run with -M24 and this may make the difference, but seems a bit of a long shot. So I ask, have I got the parameters correct?
Cheers, Gene
Gene, Does it truly produce no .C0/.C1/...? Or does it produce those in /tmp and delete them? The latter is a problem for us, so we have not yet integrated the code using /tmp. We are up-to-date wrt
commit 8f179db7fb0bf59f34c4a073b85137a592336d89
Author: thegenemyers <gene.myers@gmail.com>
Date: Sat Aug 6 06:25:10 2016
Bug fix in "Entwine" manifesting as trace point errors
About a month ago introduced a bug when improving handling of redundant
LAs within daligner. Fixed.
Matthew, you should be using use_tmpdir = true
, to eliminate file-system latency as a culprit. But that's a falcon issue. Are you able to reproduce this problem by running only the daligner job, from the command-line using the generated bash script? You could post that bash-script here. (Gene does not care about Falcon parameters or the Falcon workflow.) Maybe -M24
is somehow the cause. What version of DALIGNER are you running? cd DALIGNER; git rev-parse HEAD
(That will be in the PacBio repo, but we can still use that information.)
Gene,
I did try and make sure that the version was the latest. Although the dates included in github for FALCON-integrate 1.8.6 did appear old I noted that the commits for daligner was more up to date. I followed the instructions for installing FALCON-integrate which I can only assume would pull the latest and greatest for Daligner? Running 'git rev-parse HEAD' I get 7c09c812051983ac269d01840a1e8e8b982a06f6
I can also confirm that I also split the blocks with -x500 -s300 and I ran daligner with the defaults save for -s1000 -l1000. I do use -M24, for some earlier trials I increased it to -M32 but noticed performance appeared to suffer.
Matt
On 30/01/17 18:19, Eugene W Myers Jr wrote:
@hodgett https://github.com/hodgett @pb-cdunn https://github.com/pb-cdunn So far I have not been able to reproduce the problem. Two questions:
(1) Most importantly I note that y'all are not using the most recent version of daligner as the thread files with extensions .C0, .C1, etc. are no longer produced. This change was made about 4 months ago, and I fixed a "duplicate overlap" problem about 5 months ago. This simply could be because the release of falcon you are using does not include the latest version of daligner. This is why I include Chris Dunn in the discussion. Chris: does falcon now use the version that produces a single .las for a block vs block comparison? I note the repository pacbio/dazzler refers to a quite old version of daligner (from 11 months ago).
(2) I split the blocks with -x500 -s300 and I ran daligner with the defaults save for -s1000 -l1000 and most importantly with -M16 as I only have 16Gb on my Mac. When I get home from my vacation next week, I can run with -M24 and this may make the difference, but seems a bit of a long shot. So I ask, have I got the parameters correct?
Cheers, Gene
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-276002411, or mute the thread https://github.com/notifications/unsubscribe-auth/AXi0ifNUIW2CbP3Dpm3rHt5wVPiu08AWks5rXZ0vgaJpZM4Ls9rS.
I did try using |use_tmpdir = true| but it made no improvement. I found it added another layer of complexity in our environment (sometimes the permissions on the storage local to one of the 250 nodes does something unpredictable), and did not alter performance, so I've been leaving it out.
Running 'git rev-parse HEAD' I get 7c09c812051983ac269d01840a1e8e8b982a06f6 which doesn't correlate to Genes DALIGNER commits, but is the latest included with FALCON-integrate. Are there differences between your code and Genes?
I will try find some time over the next day or two to do some more testing wrt running bash script for that individual job.
On 31/01/17 01:41, Christopher Dunn wrote:
Gene, Does it truly produce no .C0/.C1/...? Or does it produce those in /tmp and delete them? The latter is a problem for us, so we have not yet integrated the code using /tmp. We are up-to-date wrt
|commit 8f179db7fb0bf59f34c4a073b85137a592336d89 Author: thegenemyers gene.myers@gmail.com Date: Sat Aug 6 06:25:10 2016 Bug fix in "Entwine" manifesting as trace point errors About a month ago introduced a bug when improving handling of redundant LAs within daligner. Fixed. |
Matthew, you should be using |use_tmpdir = true|, to eliminate file-system latency as a culprit. But that's a falcon issue. Are you able to reproduce this problem by running only the daligner job, from the command-line using the generated bash script? You could post that bash-script here. (Gene does not care about Falcon parameters or the Falcon workflow.) Maybe |-M24| is somehow the cause. What version of DALIGNER are you running? |cd DALIGNER; git rev-parse HEAD| (That will be in the PacBio repo, but we can still use that information.)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-276096240, or mute the thread https://github.com/notifications/unsubscribe-auth/AXi0iTpkvxuHJ5DfB1Jv_2-41SbIQ-ooks5rXgSPgaJpZM4Ls9rS.
use_tmpdir = true
Let's discuss only the DALIGNER issue here. You can re-run daligner
and LAsort/LAmerge
with a version of DALIGNER from this repo. You do not need to run Falcon at all to debug this.
I have to admit that I can find nothing that helps me set up these processes individually. How do I go about achieving what you ask?
I have separately tried to incorporate this version of daligner in to falcon, but it simply does not work when compiled in with the tool chain (i.e. replacing the pacbio fork with this version).
Falcon generates a bash script for each task. You can run the bash script in the task directory for a given daligner job. You can also run the bash script in a merge-task directory. You should be able to reproduce your problem independent of Falcon. Then, Gene can help debug any problem you have with daligner/LAmerge.
I missed a script. From what I can saw there were 3 scripts. run.sh calls task.sh. task.sh uses parameters from task.json. I missed the rj_xxx.sh script. Thanks for that one, I'll try it and see if I still have the same duplicate overlap.
On 08/02/17 03:05, Christopher Dunn wrote:
Falcon generates a bash script for each task. You can run the bash script in the task directory for a given daligner job. You can also run the bash script in a merge-task directory. You should be able to reproduce your problem independent of Falcon. Then, Gene can help debug any problem you have with daligner/LAmerge.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-278066743, or mute the thread https://github.com/notifications/unsubscribe-auth/AXi0icFEbW3XVV3QLRWAOywpLvGtdPbIks5raKRHgaJpZM4Ls9rS.
Ok, I have loaded Genes version and run the rj_xxx.sh script. This time I'm getting a new error on a different pair of files that may or may not be related. I am re-running on a different node under less load to see if I get the same result. The original run was performed using settings;
pa_HPCdaligner_option = -v -B128 -e.70 -l1000 -s1000 -M24
ovlp_HPCdaligner_option = -v -B128 -h60 -e.96 -l500 -s1000 -M24
Output tail from running rj_xxx.sh
LAsort /tmp/raw_reads.213.raw_reads.38.[CN]*.las
LAmerge raw_reads.213.raw_reads.38.las /tmp/raw_reads.213.raw_reads.38.[CN]*.S.las
rm /tmp/raw_reads.213.raw_reads.38.[CN]*.las
LAsort /tmp/raw_reads.38.raw_reads.213.[CN]*.las
LAmerge raw_reads.38.raw_reads.213.las /tmp/raw_reads.38.raw_reads.213.[CN]*.S.las
rm /tmp/raw_reads.38.raw_reads.213.[CN]*.las
Building index for raw_reads.39
Kmer count = 299,638,863
Using 8.93Gb of space
Index occupies 4.46Gb
Comparing raw_reads.213 to raw_reads.39
Capping mutual k-mer matches over 168 (effectively -t12)
Hit count = 621,877,828
Highwater of 23.00Gb space
daligner: Out of memory (Allocating daligner hit vectors)
Try reducing -M24
well under your actual memory limit.
-M24 is already well under the limit. Head nodes have 128Gb, nodes have 256Gb.
-- Matthew Hodgett, M.InfTech, CISSP Research Support Specialist | HPC & Research Support Group Queensland University of Technology (QUT) Tel: (+61) 07 3138 9454 | Mob: 0421 824730 | email: m.hodgett@qut.edu.au CRICOS 00213J
On 10/02/17 07:32, Christopher Dunn wrote:
Try reducing |-M24| well under your actual memory limit.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thegenemyers/DALIGNER/issues/55#issuecomment-278780526, or mute the thread https://github.com/notifications/unsubscribe-auth/AXi0iZzduzLZ7JET8aRr5IRVrJEVJ8z8ks5ra4XtgaJpZM4Ls9rS.
I have re-run rj_0111.sh using the latest daligner from Gene and I still have;
raw_reads.56.raw_reads.213.C1: Duplicate overlap (1573849 vs 6403244)
raw_reads.56.raw_reads.213.C2: 325,068 all OK
raw_reads.56.raw_reads.213.C3: 336,228 all OK
raw_reads.56.raw_reads.213: Duplicate overlap (1573849 vs 6403244)
I have temporarily uploaded the DBshow of these blocks to; https://nextcloud.qriscloud.org.au/index.php/s/Bkkd2Js1C45nPfG https://nextcloud.qriscloud.org.au/index.php/s/k85itrErdK2jSRX
I tried the latest blocks and ran with -M24 and still did not duplicate the error. Could you please send me the script rj_0111.sh so I can be certain that I have duplicated all the flags/options exactly.
Alternatively, could you at your end duplicate what I did here and see if you do or do not get a duplicate overlap error. Specifically, I started from your two files Block.56.fasta and Block.213.fasta and I performed the following commands:
fasta2DB Block56 Block.56.fasta fasta2DB Block213 Block.213.fasta DBsplit -s300 -x500 Block56 DBsplit -s300 -x500 Block213 daligner -v -M24 -e.7 -l1000 -s1000 Block213 Block56 LAcheck -vS Block56 Block213 Block56.Block213.las LAcheck -vS Block213 Block56 Block213.Block56.las
Both calls to LAcheck reported OK. Note that given just the data from the blocks, all I can do is create DB's from each and then run daligner between them, so the context is subtly different than calling daligner on blocks of a much larger DB. Depending on the nature of the problem it may be just enough to change the conditions so the problem does not occur. What we/I need then is a pair of blocks such that the sequence of calls above leads to the problem. If the above is problem-free at your end, then perhaps you could try generating say 20 block .fasta's and trying pairs of them to see if you can get the problem and then send me an offending pair.
Sorry this is not going easy. But keep in mind you we are talking about an event that is occuring once every billion overlaps or so :-)
[ from https://github.com/PacificBiosciences/FALCON/issues/515] We are using Falcon-integrate 1.8.6 and for our large sequences we are still getting the occasional Duplicate Overlap that is killing the whole process.
Based on our fc_run.cfg config are there any suggestions? We have tried reducing -s 100 and increasing the cutoff but still get the problem. We actually need to reduce the cutoff.
pc-dunn says "If this is on the latest DALIGNER, it is a serious problem. I suggest providing a test-case to thegenemyers, the owner of DALIGNER." What information can I provide to assist resolve this, keeping in mind that our dataset is over 100Gb.