search

glennhickey / progressiveCactus

Distribution package for the Prgressive Cactus multiple genome aligner. Dependencies are linked as submodules
Other
83 stars 26 forks source link

.maf files have same sources #11

Closed Mcfoofa closed 10 years ago

Mcfoofa commented 10 years ago

The .maf files that are being produced have the same source for each sequence that is being aligned, even when the headers of the sequences are distinct from one another.

The output for cactus.log is

2013-12-17 16:13:22.287993: Beginning Progressive Cactus Alignment

2013-12-17 16:17:59.041387: Finished Progressive Cactus Alignment

2013-12-17 16:17:59.042974: Beginning HAL Export

hal exception caught: Error opening/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h Command exited with non-zero status 1 0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 10816maxresident)k 0inputs+0outputs (0major+741minor)pagefaults 0swaps rm -f ./namingsample2/b00.hal: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.fa' '(dere:5.0,dmel:5.0)Anc0;' './namingsample2/b00.hal:' Traceback (most recent call last): File "/home/hussainather/progressiveCactus/submodules/cactus2hal/bin/cactus2hal.py", line 122, in main(); File "/home/hussainather/progressiveCactus/submodules/cactus2hal/bin/cactus2hal.py", line 113, in main system(cmdline) File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 171, in system raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) RuntimeError: Command: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.fa' '(dere:5.0,dmel:5.0)Anc0;' './namingsample2/b00.hal:' exited with non-zero status 1 The job seems to have left a log file, indicating failure: /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/job Reporting file: /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/log.txt log.txt: ---JOBTREE SLAVE OUTPUT LOG--- log.txt: FAILURE: fopen_or_die failed to open "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/tmpCqJ2Vt/maskFile.dat" for "wt" log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/bin/cactus_lastzRepeatMask.py", line 119, in log.txt: exit(main()) log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/bin/cactus_lastzRepeatMask.py", line 112, in main log.txt: shutil.rmtree(tempDir) log.txt: File "/usr/local/lib/python2.7/shutil.py", line 236, in rmtree log.txt: onerror(os.listdir, path, sys.exc_info()) log.txt: File "/usr/local/lib/python2.7/shutil.py", line 234, in rmtree log.txt: names = os.listdir(path) log.txt: OSError: [Errno 2] No such file or directory: '/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/tmpCqJ2Vt' log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py", line 271, in main log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth) log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/scriptTree/stack.py", line 153, in execute log.txt: self.target.run() log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/preprocessor/cactus_preprocessor.py", line 53, in run log.txt: system(cmdline) log.txt: File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 171, in system log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) log.txt: RuntimeError: Command: cactus_lastzRepeatMask.py --minPeriod=50 --lastzOpts='--step=2 --ambiguous=iupac --nogapped' "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD0/tmp_aIwWd5g8m1/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksOut/chunk_0" exited with non-zero status 1 log.txt: Exiting the slave because of a failed job on host hahnlab-pc01 log.txt: Due to failure we are reducing the remaining retry count of job /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/job to 0 log.txt: We have set the default memory of the failed job to 2147483648 bytes Got message from job at time: 1387315288.59 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/gTD2/tmp_9jG4B8XwNc/tmp_ZMv0ZgJkOL_kill.txt Got message from job at time: 1387315644.9 : Input-sample: ./namingsample2/sequenceData/dere_2L_new.fasta Total-sequences: 2 Total-length: 2176 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 N50: 0 Median-sequence-length: 1155 Got message from job at time: 1387315644.9 : Input-sample: ./namingsample2/sequenceData/dmel-2L-new-chromosome-r5.54.fasta Total-sequences: 1 Total-length: 23011544 Proportion-repeat-masked: 0.001351 ProportionNs: 0.000009 N50: 31082 Median-sequence-length: 23011544 Got message from job at time: 1387315644.9 : Starting caf phase target with index 0 at 1387315289.03 seconds (recursing = 1) Got message from job at time: 1387315644.9 : Pinch graph component with 1206 nodes and 1392 edges is being split up by breaking 143 edges to reduce size to less than 453 max, but found 1 pointless edges Got message from job at time: 1387315644.9 : Attaching the sequence to the cactus root 5707608840250286501, header dmel_2L with length 23011544 and 1078925 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387315645.04 : Starting bar phase target with index 0 at 1387315644.91 seconds (recursing = 1) Got message from job at time: 1387315646.39 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusBarWrapperLarge'> and stats flower name: 662310620200168968 total bases: 8668724 total-ends: 548 total-caps: 1580 max-end-degree: 25 max-adjacency-length: 215606 total-blocks: 0 total-groups: 1 total-edges: 570 total-free-ends: 0 total-attached-ends: 548 total-chains: 0 total-link groups: 0 Got message from job at time: 1387315646.39 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusBarWrapperLarge'> and stats flower name: 662310620200169100 total bases: 6345520 total-ends: 536 total-caps: 1866 max-end-degree: 45 max-adjacency-length: 183512 total-blocks: 0 total-groups: 1 total-edges: 564 total-free-ends: 0 total-attached-ends: 536 total-chains: 0 total-link groups: 0 Got message from job at time: 1387315646.39 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusBarWrapperLarge'> and stats flower name: 662310620200169104 total bases: 5032522 total-ends: 338 total-caps: 960 max-end-degree: 45 max-adjacency-length: 154183 total-blocks: 0 total-groups: 1 total-edges: 349 total-free-ends: 0 total-attached-ends: 338 total-chains: 0 total-link groups: 0 Got message from job at time: 1387315646.54 : Breaking bar job into 17 separate jobs Got message from job at time: 1387315646.65 : Breaking bar job into 12 separate jobs Got message from job at time: 1387315646.76 : Breaking bar job into 10 separate jobs Got message from job at time: 1387318442.56 : Starting avg phase target with index 0 at 1387318442.48 seconds (recursing = 0) Got message from job at time: 1387318442.56 : Starting reference phase target with index 0 at 1387318442.49 seconds (recursing = 1) Got message from job at time: 1387318452.69 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_ZzMo0UA9N9_kill.txt Got message from job at time: 1387318473.71 : Launching ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_ZzMo0UA9N9_kill.txt Got message from job at time: 1387318474.18 : Killing ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_ZzMo0UA9N9_kill.txt Got message from job at time: 1387318474.18 : Report for hahnlab-pc01:2078: cnt_get: 14107 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 14104 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 14107 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=3 size=269032876 path=: db_total_count: 3 db_total_size: 269032876 serv_conn_count: 1 serv_current_time: 1387318465.910052 serv_proc_id: 15165 serv_running_term: 23.342147 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2990678016 sys_mem_free: 3826733056 sys_mem_peak: 5070925824 sys_mem_rss: 42471424 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.032002 sys_ru_utime: 0.080005

Contents of /home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.634936082665: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 17 17:14 ktout.log

Got message from job at time: 1387318477.79 : Starting reference phase target with index 0 at 1387318474.19 seconds (recursing = 1) Got message from job at time: 1387318477.79 : Starting Reference Extract Phase Got message from job at time: 1387318477.79 : Starting check phase target with index 0 at 1387318477.72 seconds (recursing = 0) Got message from job at time: 1387318487.9 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_CmIs25vOaQ_kill.txt Got message from job at time: 1387318498.91 : Launching ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_CmIs25vOaQ_kill.txt Got message from job at time: 1387318499.57 : Killing ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_CmIs25vOaQ_kill.txt Got message from job at time: 1387318499.57 : Report for hahnlab-pc01:2078: cnt_get: 68678 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 68672 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 68678 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=6 size=271001403 path=: db_total_count: 6 db_total_size: 271001403 serv_conn_count: 1 serv_current_time: 1387318492.298675 serv_proc_id: 15565 serv_running_term: 14.503715 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 3022929920 sys_mem_free: 3668910080 sys_mem_peak: 5137113088 sys_mem_rss: 162009088 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.076004 sys_ru_utime: 0.272017

Contents of /home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.634936082665: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 17 17:14 ktout.log

Got message from job at time: 1387318502.36 : Launching ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/gTD2/tmp_9jG4B8XwNc/tmp_ZMv0ZgJkOL_kill.txt Got message from job at time: 1387318502.89 : Killing ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/gTD2/tmp_9jG4B8XwNc/tmp_ZMv0ZgJkOL_kill.txt Got message from job at time: 1387318502.89 : Report for hahnlab-pc01:1978: cnt_get: 139614 cnt_get_misses: 14681 cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 54584 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=14679 size=282130825 path=: db_total_count: 14679 db_total_size: 282130825 serv_conn_count: 1 serv_current_time: 1387318499.606351 serv_proc_id: 23063 serv_running_term: 3221.154520 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 3022749696 sys_mem_free: 3832893440 sys_mem_peak: 5097254912 sys_mem_rss: 109178880 sys_mem_size: 5082243072 sys_mem_total: 8328691712 sys_ru_stime: 2.384149 sys_ru_utime: 2.332145

Contents of /home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0/Anc0_DB: total 23M -rw-rw-r-- 1 hussainather hussainather 23M Dec 17 17:14 cactusSequences -rw-rw-r-- 1 hussainather hussainather 457 Dec 17 16:21 ktout.log

2013-12-17 17:15:02.981026: Finished Progressive Cactus Alignment

2013-12-17 17:15:02.984004: Beginning HAL Export

3.52user 0.06system 0:03.60elapsed 99%CPU (0avgtext+0avgdata 362336maxresident)k 0inputs+15952outputs (0major+34427minor)pagefaults 0swaps rm -f ./namingsample2/b00.hal: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.fa' '(dere:5.0,dmel:5.0)Anc0;' './namingsample2/b00.hal:' total time: 3.61 total halAppendCactusSubtree time: 3.61

2013-12-17 17:15:06.761692: Finished HAL Export

glennhickey commented 10 years ago

try using the --ucscNames options

On Tue, Dec 17, 2013 at 8:49 PM, Hussain notifications@github.com wrote:

The .maf files that are being produced have the same source for each sequence that is being aligned, even when the headers of the sequences are distinct from one another.

The output for cactus.log is

2013-12-17 16:13:22.287993: Beginning Progressive Cactus Alignment

2013-12-17 16:17:59.041387: Finished Progressive Cactus Alignment

2013-12-17 16:17:59.042974: Beginning HAL Export

hal exception caught: Error opening/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h Command exited with non-zero status 1 0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 10816maxresident)k 0inputs+0outputs (0major+741minor)pagefaults 0swaps rm -f ./namingsample2/b00.hal: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.fa' '(dere:5.0,dmel:5.0)Anc0;' './namingsample2/b00.hal:' Traceback (most recent call last): File "/home/hussainather/progressiveCactus/submodules/cactus2hal/bin/cactus2hal.py", line 122, in main(); File "/home/hussainather/progressiveCactus/submodules/cactus2hal/bin/cactus2hal.py", line 113, in main system(cmdline) File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 171, in system raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) RuntimeError: Command: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.fa' '(dere:5.0,dmel:5.0)Anc0;' './namingsample2/b00.hal:' exited with non-zero status 1 The job seems to have left a log file, indicating failure: /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/job Reporting file: /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/log.txt log.txt: ---JOBTREE SLAVE OUTPUT LOG--- log.txt: FAILURE: fopen_or_die failed to open "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/tmpCqJ2Vt/maskFile.dat" for "wt" log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/bin/cactus_lastzRepeatMask.py", line 119, in log.txt: exit(main()) log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/bin/cactus_lastzRepeatMask.py", line 112, in main log.txt: shutil.rmtree(tempDir) log.txt: File "/usr/local/lib/python2.7/shutil.py", line 236, in rmtree log.txt: onerror(os.listdir, path, sys.exc_info()) log.txt: File "/usr/local/lib/python2.7/shutil.py", line 234, in rmtree log.txt: names = os.listdir(path) log.txt: OSError: [Errno 2] No such file or directory: '/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/tmpCqJ2Vt' log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py", line 271, in main log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth) log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/scriptTree/stack.py", line 153, in execute log.txt: self.target.run() log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/preprocessor/cactus_preprocessor.py", line 53, in run log.txt: system(cmdline) log.txt: File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 171, in system log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) log.txt: RuntimeError: Command: cactus_lastzRepeatMask.py --minPeriod=50 --lastzOpts='--step=2 --ambiguous=iupac --nogapped' "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD0/tmp_aIwWd5g8m1/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksOut/chunk_0" exited with non-zero status 1 log.txt: Exiting the slave because of a failed job on host hahnlab-pc01 log.txt: Due to failure we are reducing the remaining retry count of job /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/job to 0 log.txt: We have set the default memory of the failed job to 2147483648bytes Got message from job at time: 1387315288.59 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/gTD2/tmp_9jG4B8XwNc/tmp_ZMv0ZgJkOL_kill.txt Got message from job at time: 1387315644.9 : Input-sample: ./namingsample2/sequenceData/dere_2L_new.fasta Total-sequences: 2 Total-length: 2176 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 N50: 0 Median-sequence-length: 1155 Got message from job at time: 1387315644.9 : Input-sample: ./namingsample2/sequenceData/dmel-2L-new-chromosome-r5.54.fasta Total-sequences: 1 Total-length: 23011544 Proportion-repeat-masked: 0.001351 ProportionNs: 0.000009 N50: 31082 Median-sequence-length: 23011544 Got message from job at time: 1387315644.9 : Starting caf phase target with index 0 at 1387315289.03 seconds (recursing = 1) Got message from job at time: 1387315644.9 : Pinch graph component with 1206 nodes and 1392 edges is being split up by breaking 143 edges to reduce size to less than 453 max, but found 1 pointless edges Got message from job at time: 1387315644.9 : Attaching the sequence to the cactus root 5707608840250286501, header dmel_2L with length 23011544 and 1078925 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387315645.04 : Starting bar phase target with index 0 at 1387315644.91 seconds (recursing = 1) Got message from job at time: 1387315646.39 : Adding an oversize flower for target class and stats flower name: 662310620200168968 total bases: 8668724 total-ends: 548 total-caps: 1580 max-end-degree: 25 max-adjacency-length: 215606 total-blocks: 0 total-groups: 1 total-edges: 570 total-free-ends: 0 total-attached-ends: 548 total-chains: 0 total-link groups: 0 Got message from job at time: 1387315646.39 : Adding an oversize flower for target class and stats flower name: 662310620200169100 total bases: 6345520 total-ends: 536 total-caps: 1866 max-end-degree: 45 max-adjacency-length: 183512 total-blocks: 0 total-groups: 1 total-edges: 564 total-free-ends: 0 total-attached-ends: 536 total-chains: 0 total-link groups: 0 Got message from job at time: 1387315646.39 : Adding an oversize flower for target class and stats flower name: 662310620200169104 total bases: 5032522 total-ends: 338 total-caps: 960 max-end-degree: 45 max-adjacency-length: 154183 total-blocks: 0 total-groups: 1 total-edges: 349 total-free-ends: 0 total-attached-ends: 338 total-chains: 0 total-link groups: 0 Got message from job at time: 1387315646.54 : Breaking bar job into 17 separate jobs Got message from job at time: 1387315646.65 : Breaking bar job into 12 separate jobs Got message from job at time: 1387315646.76 : Breaking bar job into 10 separate jobs Got message from job at time: 1387318442.56 : Starting avg phase target with index 0 at 1387318442.48 seconds (recursing = 0) Got message from job at time: 1387318442.56 : Starting reference phase target with index 0 at 1387318442.49 seconds (recursing = 1) Got message from job at time: 1387318452.69 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_ZzMo0UA9N9_kill.txt Got message from job at time: 1387318473.71 : Launching ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_ZzMo0UA9N9_kill.txt Got message from job at time: 1387318474.18 : Killing ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_ZzMo0UA9N9_kill.txt Got message from job at time: 1387318474.18 : Report for hahnlab-pc01:2078: cnt_get: 14107 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 14104 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 14107 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=3 size=269032876 path=: db_total_count: 3 db_total_size: 269032876 serv_conn_count: 1 serv_current_time: 1387318465.910052 serv_proc_id: 15165 serv_running_term: 23.342147 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2990678016 sys_mem_free: 3826733056 sys_mem_peak: 5070925824 sys_mem_rss: 42471424 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.032002 sys_ru_utime: 0.080005

Contents of /home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.634936082665: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 17 17:14 ktout.log

Got message from job at time: 1387318477.79 : Starting reference phase target with index 0 at 1387318474.19 seconds (recursing = 1) Got message from job at time: 1387318477.79 : Starting Reference Extract Phase Got message from job at time: 1387318477.79 : Starting check phase target with index 0 at 1387318477.72 seconds (recursing = 0) Got message from job at time: 1387318487.9 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_CmIs25vOaQ_kill.txt Got message from job at time: 1387318498.91 : Launching ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_CmIs25vOaQ_kill.txt Got message from job at time: 1387318499.57 : Killing ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_ksnUe38TG9/tmp_CmIs25vOaQ_kill.txt Got message from job at time: 1387318499.57 : Report for hahnlab-pc01:2078: cnt_get: 68678 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 68672 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 68678 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=6 size=271001403 path=: db_total_count: 6 db_total_size: 271001403 serv_conn_count: 1 serv_current_time: 1387318492.298675 serv_proc_id: 15565 serv_running_term: 14.503715 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 3022929920 sys_mem_free: 3668910080 sys_mem_peak: 5137113088 sys_mem_rss: 162009088 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.076004 sys_ru_utime: 0.272017

Contents of /home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.634936082665: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 17 17:14 ktout.log

Got message from job at time: 1387318502.36 : Launching ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/gTD2/tmp_9jG4B8XwNc/tmp_ZMv0ZgJkOL_kill.txt Got message from job at time: 1387318502.89 : Killing ktserver with killPath /home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/gTD2/tmp_9jG4B8XwNc/tmp_ZMv0ZgJkOL_kill.txt Got message from job at time: 1387318502.89 : Report for hahnlab-pc01:1978: cnt_get: 139614 cnt_get_misses: 14681 cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 54584 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=14679 size=282130825 path=: db_total_count: 14679 db_total_size: 282130825 serv_conn_count: 1 serv_current_time: 1387318499.606351 serv_proc_id: 23063 serv_running_term: 3221.154520 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 3022749696 sys_mem_free: 3832893440 sys_mem_peak: 5097254912 sys_mem_rss: 109178880 sys_mem_size: 5082243072 sys_mem_total: 8328691712 sys_ru_stime: 2.384149 sys_ru_utime: 2.332145

Contents of /home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0/Anc0_DB: total 23M -rw-rw-r-- 1 hussainather hussainather 23M Dec 17 17:14 cactusSequences -rw-rw-r-- 1 hussainather hussainather 457 Dec 17 16:21 ktout.log

2013-12-17 17:15:02.981026: Finished Progressive Cactus Alignment

2013-12-17 17:15:02.984004: Beginning HAL Export

3.52user 0.06system 0:03.60elapsed 99%CPU (0avgtext+0avgdata 362336maxresident)k 0inputs+15952outputs (0major+34427minor)pagefaults 0swaps rm -f ./namingsample2/b00.hal: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/namingsample2/progressiveAlignment/Anc0/Anc0_hal.fa' '(dere:5.0,dmel:5.0)Anc0;' './namingsample2/b00.hal:' total time: 3.61 total halAppendCactusSubtree time: 3.61

2013-12-17 17:15:06.761692: Finished HAL Export

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11 .

Mcfoofa commented 10 years ago

What do you mean? You mean running progressiveCactus with --ucscsNames before the seqFile?

glennhickey commented 10 years ago

Oh, I just went by our message (about mafs) and didn't see the log you attached. Looks like Cactus is crashing (error extracted below). This is almost certainly a problem introduced in a recent change. Will let you know what it's fixed. Thanks for alerting us.

log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) log.txt: RuntimeError: Command: cactus_lastzRepeatMask.py --minPeriod=50 --lastzOpts='--step=2 --ambiguous=iupac --nogapped' "/home/hussainather/

progressiveCactus/namingsample2/jobTree/jobs/t1/gTD0/tmp_aIwWd5g8m1/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksOut/chunk_0" exited with non-zero status 1

On Wed, Dec 18, 2013 at 12:35 PM, Hussain notifications@github.com wrote:

What do you mean? You mean running progressiveCactus with --ucscsNames before the seqFile?

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30862501 .

benedictpaten commented 10 years ago

Is this run with the latest code on the development branch? I recently fixed an issue that caused a crash.

On Wed, Dec 18, 2013 at 9:52 AM, Glenn Hickey notifications@github.comwrote:

Oh, I just went by our message (about mafs) and didn't see the log you attached. Looks like Cactus is crashing (error extracted below). This is almost certainly a problem introduced in a recent change. Will let you know what it's fixed. Thanks for alerting us.

log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) log.txt: RuntimeError: Command: cactus_lastzRepeatMask.py --minPeriod=50 --lastzOpts='--step=2 --ambiguous=iupac --nogapped' "/home/hussainather/

progressiveCactus/namingsample2/jobTree/jobs/t1/gTD0/tmp_aIwWd5g8m1/0" "/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/0"

"/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksOut/chunk_0"

exited with non-zero status 1

On Wed, Dec 18, 2013 at 12:35 PM, Hussain notifications@github.com wrote:

What do you mean? You mean running progressiveCactus with --ucscsNames before the seqFile?

— Reply to this email directly or view it on GitHub< https://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30862501>

.

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30864066 .

glennhickey commented 10 years ago

yeah, best place to start is updating to the latest version. from the manual:

cd progressiveCactus git pull git submodule update --init make ucscClean make

On Wed, Dec 18, 2013 at 1:02 PM, Benedict Paten notifications@github.comwrote:

Is this run with the latest code on the development branch? I recently fixed an issue that caused a crash.

On Wed, Dec 18, 2013 at 9:52 AM, Glenn Hickey notifications@github.comwrote:

Oh, I just went by our message (about mafs) and didn't see the log you attached. Looks like Cactus is crashing (error extracted below). This is almost certainly a problem introduced in a recent change. Will let you know what it's fixed. Thanks for alerting us.

log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts)) log.txt: RuntimeError: Command: cactus_lastzRepeatMask.py --minPeriod=50 --lastzOpts='--step=2 --ambiguous=iupac --nogapped' "/home/hussainather/

progressiveCactus/namingsample2/jobTree/jobs/t1/gTD0/tmp_aIwWd5g8m1/0"

"/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksIn/0"

"/home/hussainather/progressiveCactus/namingsample2/jobTree/jobs/t1/gTD1/tmp_M6g6N20igO/preprocessChunksOut/chunk_0"

exited with non-zero status 1

On Wed, Dec 18, 2013 at 12:35 PM, Hussain notifications@github.com wrote:

What do you mean? You mean running progressiveCactus with --ucscsNames before the seqFile?

— Reply to this email directly or view it on GitHub<

https://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30862501>

.

— Reply to this email directly or view it on GitHub< https://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30864066>

.

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30864918 .

Mcfoofa commented 10 years ago

Alright thanks. I'm trying to run make, but I get this error now IOError: [Errno 26] Text file busy: '/home/hussainather/progressiveCactus/python/bin/python' make[1]: * [virtPyRule] Error 1 make[1]: Leaving directory `/home/hussainather/progressiveCactus/submodules' make: * [all] Error 2

EDIT: Just fixed it by killing the python processes.

Mcfoofa commented 10 years ago

Just got this error during the make:

segment.c: In function ‘read_segment_table’: segment.c:436:59: error: variable ‘qNegEnd’ set but not used [-Werror=unused-but-set-variable] cc1: all warnings being treated as errors make[5]: * [segment.o] Error 1 make[5]: Leaving directory /home/hussainather/progressiveCactus/submodules/cactus/externalTools/lastz-distrib-1.03.46/src' make[4]: *** [install_lastz] Error 2 make[4]: Leaving directory/home/hussainather/progressiveCactus/submodules/cactus/externalTools/lastz-distrib-1.03.46' make[3]: * [../bin//cactus_lastz] Error 2 make[3]: Leaving directory `/home/hussainather/progressiveCactus/submodules/cactus/externalTools' make[2]: * [all.externalTools] Error 2 make[2]: Leaving directory/home/hussainather/progressiveCactus/submodules/cactus' make[1]: **\* [ucsc.cactus] Error 2 make[1]: Leaving directory/home/hussainather/progressiveCactus/submodules' make: * [all] Error 2

glennhickey commented 10 years ago

Should be fixed now (if you re update from github).

On Wed, Dec 18, 2013 at 2:52 PM, Hussain notifications@github.com wrote:

Just got this error during the make:

segment.c: In function ‘read_segment_table’: segment.c:436:59: error: variable ‘qNegEnd’ set but not used [-Werror=unused-but-set-variable] cc1: all warnings being treated as errors make[5]: * [segment.o] Error 1 make[5]: Leaving directory /home/hussainather/progressiveCactus/submodules/cactus/externalTools/lastz-distrib-1.03.46/src' make[4]: * [install_lastz] Error 2 make[4]: Leaving directory /home/hussainather/progressiveCactus/submodules/cactus/externalTools/lastz-distrib-1.03.46' make[3]: * [../bin//cactus_lastz] Error 2 make[3]: Leaving directory /home/hussainather/progressiveCactus/submodules/cactus/externalTools' make[2]: * [all.externalTools] Error 2 make[2]: Leaving directory /home/hussainather/progressiveCactus/submodules/cactus' make[1]: *\ [ucsc.cactus] Error 2

make[1]: Leaving directory `/home/hussainather/progressiveCactus/submodules' make: *\ [all] Error 2

—

Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30874409 .

Mcfoofa commented 10 years ago

Looks like it worked now. Thanks!

benedictpaten commented 10 years ago

Great, sorry about that. Let us know if you see anything else odd. Hoping to get stuff all cleaned up now for a proper release.

On Wed, Dec 18, 2013 at 12:54 PM, Hussain notifications@github.com wrote:

Looks like it worked now. Thanks!

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30879776 .

Mcfoofa commented 10 years ago

I'm still getting some outputs that have the same file being aligned to itself. For example, this is what one of the .maf files will read:

maf version=1 scoring=N/A

hal ((dere:8,dyak:8)Anc2:3,(dmel:5,dsim:5)Anc1:6)Anc0;

a s Anc0refChr1 0 1 + 7374327 n

a s Anc0refChr1 1 1 + 7374327 c s Anc2refChr1 254010 1 + 8466230 t s Anc2refChr1 361156 1 + 8466230 C s dyak_2L 1104077 1 - 22210002 C s dyak_2L 21661811 1 + 22210002 t

a s Anc0refChr1 2 1 + 7374327 t s Anc2refChr1 254011 1 + 8466230 t s Anc2refChr1 361157 1 + 8466230 n s dyak_2L 21661812 1 + 22210002 t

a s Anc0refChr1 3 137 + 7374327 atacccgttactcgtagagtaaaagggtatactagagtcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccgtatagagtataaatattcttgatcaggatcaaaagccaagtcgatctggccatgtc s Anc2refChr1 254012 137 + 8466230 atacccgttactcgtagagtaaaagggtatactagagtcgttgaaaagtatgtaacaggcagaagaaagcgtttccgaccgtatagagtataaatatacttgatagggatcaaaagccaagtcgatctggccatgtc s Anc2refChr1 361158 137 + 8466230 atacccgttactcgtagagtaaaagggtatactagattcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccatataaagtatatatattcttgatcaggatcagtagccgagtcgatctggccatgtc s dyak_2L 1128800 137 - 22210002 atacccgttactcgaagagtaaaagggtatactagattcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccatataaagtatatatattcttgatcaggatcaatagccgagtcgatttggccatgtc s dyak_2L 21384820 137 + 22210002 atacccgttactcgtagagtaaaagggtatactagattcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccatataaagtatatatattcttgatcaggatcagtagccgagtcgatctggccatgtc s dyak_2L 21661813 137 + 22210002 atacccgttactcgtagagtaaaaggatatactaaattcgttgaaaagtatgtaacaggcagaagaatgcgtttgcgaccatataaagtatatatattcttgatcaggatcaaaagccaagtcgatctggccatgtc

Here's the cactus.log:

2013-12-18 20:16:15.361480: Beginning Progressive Cactus Alignment

Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dere_2L_new.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dere_2L_new.fasta Total-sequences: 2 Total-length: 2176 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 N50: 0 Median-sequence-length: 1155 Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dyak_2L_new.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dyak_2L_new.fasta Total-sequences: 1 Total-length: 22210002 Proportion-repeat-masked: 0.000157 ProportionNs: 0.000157 N50: 3480 Median-sequence-length: 22210002 Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dro/dmel-2L-new-chromosome-r5.54.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dro/dmel-2L-new-chromosome-r5.54.fasta Total-sequences: 1 Total-length: 23011544 Proportion-repeat-masked: 0.000009 ProportionNs: 0.000009 N50: 200 Median-sequence-length: 23011544 Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dsim_2L_new.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dsim_2L_new.fasta Total-sequences: 1 Total-length: 20694855 Proportion-repeat-masked: 0.001849 ProportionNs: 0.001849 N50: 38263 Median-sequence-length: 20694855 Got message from job at time: 1387415777.0 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dere_2L_new.fasta Total-sequences: 2 Total-length: 2176 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 N50: 0 Median-sequence-length: 1155 Got message from job at time: 1387415856.39 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dsim_2L_new.fasta Total-sequences: 1 Total-length: 20694855 Proportion-repeat-masked: 0.005836 ProportionNs: 0.001849 N50: 120769 Median-sequence-length: 20694855 Got message from job at time: 1387415857.08 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dyak_2L_new.fasta Total-sequences: 1 Total-length: 22210002 Proportion-repeat-masked: 0.020924 ProportionNs: 0.000157 N50: 464725 Median-sequence-length: 22210002 Got message from job at time: 1387415863.08 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dro_dmel-2L-new-chromosome-r5.54.fasta Total-sequences: 1 Total-length: 23011544 Proportion-repeat-masked: 0.014328 ProportionNs: 0.000009 N50: 329700 Median-sequence-length: 23011544 Got message from job at time: 1387415863.32 : The longest path in the tree is 14.000000 Got message from job at time: 1387415863.32 : Made argument lastzArguments=--step=1 --ambiguous=iupac,100 --ydrop=3000 in tag caf with divergence threshold of 9223372036854775807 for longest path of 14.0 (useDefaultDivergences=False) Got message from job at time: 1387415873.42 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD4/tmp_v2ENgQtjyl/tmp_zKUPY7jlAX_kill.txt Got message from job at time: 1387415874.38 : The blast stage will filter by identity, the calculated minimum identity is 25 from a longest path of 14.0 and an adjusted path of 42.0 Got message from job at time: 1387415874.38 : Starting caf phase target with index 0 at 1387415874.08 seconds (recursing = 1) Got message from job at time: 1387415874.38 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusCafWrapperLarge'> and stats flower name: 0 total bases: 45223722 total-ends: 8 total-caps: 8 max-end-degree: 1 max-adjacency-length: 23011545 total-blocks: 0 total-groups: 1 total-edges: 4 total-free-ends: 8 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1387417126.18 : Pinch graph component with 978 nodes and 1157 edges is being split up by breaking 120 edges to reduce size to less than 432 max, but found 0 pointless edges Got message from job at time: 1387417126.18 : Attaching the sequence to the cactus root 4973381363500545548, header dmel_2L with length 23011544 and 90805 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387417126.18 : Starting bar phase target with index 0 at 1387417125.25 seconds (recursing = 1) Got message from job at time: 1387417126.18 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusBarWrapperLarge'> and stats flower name: 7213218490675588443 total bases: 22699115 total-ends: 533 total-caps: 1566 max-end-degree: 13 max-adjacency-length: 646660 total-blocks: 0 total-groups: 1 total-edges: 579 total-free-ends: 1 total-attached-ends: 532 total-chains: 0 total-link groups: 0 Got message from job at time: 1387417126.18 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusBarWrapperLarge'> and stats flower name: 7213218490675588503 total bases: 19669494 total-ends: 533 total-caps: 1496 max-end-degree: 11 max-adjacency-length: 1308294 total-blocks: 0 total-groups: 1 total-edges: 559 total-free-ends: 0 total-attached-ends: 533 total-chains: 0 total-link groups: 0 Got message from job at time: 1387417126.3 : Breaking bar job into 21 separate jobs Got message from job at time: 1387417134.64 : Breaking bar job into 18 separate jobs Got message from job at time: 1387419376.71 : Starting avg phase target with index 0 at 1387419376.63 seconds (recursing = 0) Got message from job at time: 1387419376.71 : Starting reference phase target with index 0 at 1387419376.64 seconds (recursing = 1) Got message from job at time: 1387419386.86 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_w6pL62aVpY_kill.txt Got message from job at time: 1387419417.86 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_w6pL62aVpY_kill.txt Got message from job at time: 1387419419.38 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_w6pL62aVpY_kill.txt Got message from job at time: 1387419419.38 : Report for hahnlab-pc01:2080: cnt_get: 13653 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 13651 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 13653 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=2 size=274200414 path=: db_total_count: 2 db_total_size: 274200414 serv_conn_count: 1 serv_current_time: 1387419407.801119 serv_proc_id: 29432 serv_running_term: 31.050650 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 1998442496 sys_mem_free: 5061177344 sys_mem_peak: 5137113088 sys_mem_rss: 66117632 sys_mem_size: 5074075648 sys_mem_total: 8328691712 sys_ru_stime: 0.096006 sys_ru_utime: 0.124007

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2/Anc2_DB_tempSecondaryDatabaseDir_0.155984206167: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 21:16 ktout.log

Got message from job at time: 1387419419.56 : Starting reference phase target with index 0 at 1387419419.39 seconds (recursing = 1) Got message from job at time: 1387419419.56 : Adding an oversize flower 2059974617056921301 for target class <class 'cactus.pipeline.cactus_workflow.CactusSetReferenceCoordinatesDownWrapper'> Got message from job at time: 1387419426.34 : Starting Reference Extract Phase Got message from job at time: 1387419426.34 : Starting check phase target with index 0 at 1387419426.25 seconds (recursing = 0) Got message from job at time: 1387419436.48 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_tiMGyzIOSb_kill.txt Got message from job at time: 1387419436.55 : Adding an oversize flower 2059974617056921301 for target class <class 'cactus.pipeline.cactus_workflow.CactusHalGeneratorRecursion'> Got message from job at time: 1387419447.48 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_tiMGyzIOSb_kill.txt Got message from job at time: 1387419448.17 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_tiMGyzIOSb_kill.txt Got message from job at time: 1387419448.17 : Report for hahnlab-pc01:2080: cnt_get: 60013 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 60007 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 60013 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=6 size=275368427 path=: db_total_count: 6 db_total_size: 275368427 serv_conn_count: 1 serv_current_time: 1387419442.911553 serv_proc_id: 29913 serv_running_term: 16.561421 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2081054720 sys_mem_free: 4876894208 sys_mem_peak: 5137113088 sys_mem_rss: 161419264 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.144009 sys_ru_utime: 0.240015

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2/Anc2_DB_tempSecondaryDatabaseDir_0.155984206167: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 21:17 ktout.log

Got message from job at time: 1387419458.06 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD4/tmp_v2ENgQtjyl/tmp_zKUPY7jlAX_kill.txt Got message from job at time: 1387419458.63 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD4/tmp_v2ENgQtjyl/tmp_zKUPY7jlAX_kill.txt Got message from job at time: 1387419458.63 : Report for hahnlab-pc01:1980: cnt_get: 141199 cnt_get_misses: 16416 cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 55217 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=16414 size=298891953 path=: db_total_count: 16414 db_total_size: 298891953 serv_conn_count: 1 serv_current_time: 1387419448.210416 serv_proc_id: 23456 serv_running_term: 3584.885500 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2081177600 sys_mem_free: 5040168960 sys_mem_peak: 5103501312 sys_mem_rss: 125767680 sys_mem_size: 5070147584 sys_mem_total: 8328691712 sys_ru_stime: 2.520157 sys_ru_utime: 3.132195

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2/Anc2_DB: total 52M -rw-rw-r-- 1 hussainather hussainather 52M Dec 18 21:16 cactusSequences -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 20:17 ktout.log

Got message from job at time: 1387419458.88 : The longest path in the tree is 17.000000 Got message from job at time: 1387419458.88 : Made argument lastzArguments=--step=1 --ambiguous=iupac,100 --ydrop=3000 in tag caf with divergence threshold of 9223372036854775807 for longest path of 17.0 (useDefaultDivergences=False) Got message from job at time: 1387419468.99 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD3/tmp_XLF3ev9qdg/tmp_bnHjmrPAqM_kill.txt Got message from job at time: 1387419470.03 : The blast stage will filter by identity, the calculated minimum identity is 25 from a longest path of 17.0 and an adjusted path of 51.0 Got message from job at time: 1387419470.03 : Starting caf phase target with index 0 at 1387419469.69 seconds (recursing = 1) Got message from job at time: 1387419470.03 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusCafWrapperLarge'> and stats flower name: 0 total bases: 43708575 total-ends: 8 total-caps: 8 max-end-degree: 1 max-adjacency-length: 23011545 total-blocks: 0 total-groups: 1 total-edges: 4 total-free-ends: 8 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1387420805.78 : Pinch graph component with 903 nodes and 1087 edges is being split up by breaking 118 edges to reduce size to less than 622 max, but found 0 pointless edges Got message from job at time: 1387420805.78 : Attaching the sequence to the cactus root 1839720447780834549, header dmel_2L with length 23011544 and 16650219 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387420805.78 : Starting bar phase target with index 0 at 1387420805.69 seconds (recursing = 1) Got message from job at time: 1387422097.7 : Starting avg phase target with index 0 at 1387422097.63 seconds (recursing = 0) Got message from job at time: 1387422097.7 : Starting reference phase target with index 0 at 1387422097.63 seconds (recursing = 1) Got message from job at time: 1387422107.85 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_eOhDcbSUVp_kill.txt Got message from job at time: 1387422178.83 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_eOhDcbSUVp_kill.txt Got message from job at time: 1387422179.55 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_eOhDcbSUVp_kill.txt Got message from job at time: 1387422179.55 : Report for hahnlab-pc01:2079: cnt_get: 259450 cnt_get_misses: 1 cnt_misc: 0 cnt_remove: 259448 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 259449 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=1 size=268435759 path=: db_total_count: 1 db_total_size: 268435759 serv_conn_count: 1 serv_current_time: 1387422172.214568 serv_proc_id: 658 serv_running_term: 74.502898 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2180894720 sys_mem_free: 4239634432 sys_mem_peak: 5137113088 sys_mem_rss: 251650048 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.176011 sys_ru_utime: 0.796049

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1/Anc1_DB_tempSecondaryDatabaseDir_0.903517412895: total 13M -rw-rw-r-- 1 hussainather hussainather 13M Dec 18 22:02 Anc1.kch.BIGRECORDFILE__3702803318628653371 -rw-rw-r-- 1 hussainather hussainather 455 Dec 18 22:01 ktout.log

Got message from job at time: 1387422181.85 : Starting reference phase target with index 0 at 1387422179.56 seconds (recursing = 1) Got message from job at time: 1387422181.85 : Adding an oversize flower 2339197793953890069 for target class <class 'cactus.pipeline.cactus_workflow.CactusSetReferenceCoordinatesDownWrapper'> Got message from job at time: 1387422209.66 : Starting Reference Extract Phase Got message from job at time: 1387422209.66 : Starting check phase target with index 0 at 1387422209.57 seconds (recursing = 0) Got message from job at time: 1387422219.78 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_twONwSWPEo_kill.txt Got message from job at time: 1387422219.87 : Adding an oversize flower 2339197793953890069 for target class <class 'cactus.pipeline.cactus_workflow.CactusHalGeneratorRecursion'> Got message from job at time: 1387422260.76 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_twONwSWPEo_kill.txt Got message from job at time: 1387422261.9 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_twONwSWPEo_kill.txt Got message from job at time: 1387422261.9 : Report for hahnlab-pc01:2079: cnt_get: 839669 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 839663 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 839669 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=6 size=286809989 path=: db_total_count: 6 db_total_size: 286809989 serv_conn_count: 1 serv_current_time: 1387422257.669621 serv_proc_id: 1171 serv_running_term: 48.003631 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2301743104 sys_mem_free: 3995082752 sys_mem_peak: 5206835200 sys_mem_rss: 347586560 sys_mem_size: 5079363584 sys_mem_total: 8328691712 sys_ru_stime: 0.364022 sys_ru_utime: 2.576161

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1/Anc1_DB_tempSecondaryDatabaseDir_0.903517412895: total 4.0K -rw-rw-r-- 1 hussainather hussainather 456 Dec 18 22:03 ktout.log

Got message from job at time: 1387422262.52 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD3/tmp_XLF3ev9qdg/tmp_bnHjmrPAqM_kill.txt Got message from job at time: 1387422264.11 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD3/tmp_XLF3ev9qdg/tmp_bnHjmrPAqM_kill.txt Got message from job at time: 1387422264.11 : Report for hahnlab-pc01:1979: cnt_get: 2953891 cnt_get_misses: 263895 cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 1159766 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=263887 size=400792384 path=: db_total_count: 263887 db_total_size: 400792384 serv_conn_count: 1 serv_current_time: 1387422261.941090 serv_proc_id: 30285 serv_running_term: 2803.057079 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2301710336 sys_mem_free: 4345667584 sys_mem_peak: 5092704256 sys_mem_rss: 580976640 sys_mem_size: 5088739328 sys_mem_total: 8328691712 sys_ru_stime: 8.372523 sys_ru_utime: 16.729045

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1/Anc1_DB: total 81M -rw-rw-r-- 1 hussainather hussainather 21M Dec 18 22:03 Anc1.kch.BIGRECORDFILE__2339197793953890069 -rw-rw-r-- 1 hussainather hussainather 60M Dec 18 22:02 cactusSequences -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 21:17 ktout.log

Got message from job at time: 1387422264.31 : Extending longest path Got message from job at time: 1387422264.31 : The longest path in the tree is 6.100000 Got message from job at time: 1387422264.31 : Made argument lastzArguments=--step=1 --ambiguous=iupac,100 --ydrop=3000 in tag caf with divergence threshold of 9223372036854775807 for longest path of 6.1 (useDefaultDivergences=False) Got message from job at time: 1387422274.42 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD2/tmp_IrlmJxCF6H/tmp_hTNSGcAUi3_kill.txt Got message from job at time: 1387422275.07 : The blast stage will filter by identity, the calculated minimum identity is 26 from a longest path of 6.1 and an adjusted path of 18.3 Got message from job at time: 1387422275.07 : Starting caf phase target with index 0 at 1387422274.84 seconds (recursing = 1) Got message from job at time: 1387422275.07 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusCafWrapperLarge'> and stats flower name: 0 total bases: 27601997 total-ends: 6 total-caps: 6 max-end-degree: 1 max-adjacency-length: 19135527 total-blocks: 0 total-groups: 1 total-edges: 3 total-free-ends: 6 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1387422989.07 : Pinch graph component with 654 nodes and 684 edges is being split up by breaking 3 edges to reduce size to less than 574 max, but found 0 pointless edges Got message from job at time: 1387422989.07 : Attaching the sequence to the cactus root 9126685382354600639, header Anc1refChr0 with length 19135526 and 6322551 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387422989.07 : Starting bar phase target with index 0 at 1387422988.98 seconds (recursing = 1) Got message from job at time: 1387423002.28 : Adding an oversize flower for target class <class 'cactus.pipeline.cactus_workflow.CactusBarWrapperLarge'> and stats flower name: 4499518240208160293 total bases: 8442425 total-ends: 577 total-caps: 1406 max-end-degree: 8 max-adjacency-length: 617959 total-blocks: 0 total-groups: 1 total-edges: 604 total-free-ends: 0 total-attached-ends: 577 total-chains: 0 total-link groups: 0 Got message from job at time: 1387423002.36 : Breaking bar job into 8 separate jobs Got message from job at time: 1387423360.29 : Starting avg phase target with index 0 at 1387423360.22 seconds (recursing = 0) Got message from job at time: 1387423360.29 : Starting reference phase target with index 0 at 1387423360.23 seconds (recursing = 1) Got message from job at time: 1387423370.4 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4ITet6blIJ_kill.txt Got message from job at time: 1387423401.39 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4ITet6blIJ_kill.txt Got message from job at time: 1387423402.93 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4ITet6blIJ_kill.txt Got message from job at time: 1387423402.93 : Report for hahnlab-pc01:2078: cnt_get: 98898 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 98896 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 98898 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=2 size=273463480 path=: db_total_count: 2 db_total_size: 273463480 serv_conn_count: 1 serv_current_time: 1387423394.560311 serv_proc_id: 12455 serv_running_term: 34.257573 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2336792576 sys_mem_free: 4374437888 sys_mem_peak: 5137113088 sys_mem_rss: 162426880 sys_mem_size: 5075746816 sys_mem_total: 8328691712 sys_ru_stime: 0.136008 sys_ru_utime: 0.312019

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.694241566489: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 22:22 ktout.log

Got message from job at time: 1387423414.5 : Starting reference phase target with index 0 at 1387423402.94 seconds (recursing = 1) Got message from job at time: 1387423414.5 : Starting Reference Extract Phase Got message from job at time: 1387423414.51 : Starting check phase target with index 0 at 1387423414.43 seconds (recursing = 0) Got message from job at time: 1387423424.62 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4P3w559oxv_kill.txt Got message from job at time: 1387423445.64 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4P3w559oxv_kill.txt Got message from job at time: 1387423446.95 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4P3w559oxv_kill.txt Got message from job at time: 1387423446.95 : Report for hahnlab-pc01:2078: cnt_get: 305982 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 305977 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 305982 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=5 size=275295852 path=: db_total_count: 5 db_total_size: 275295852 serv_conn_count: 1 serv_current_time: 1387423438.567249 serv_proc_id: 12885 serv_running_term: 24.054232 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2398117888 sys_mem_free: 4145618944 sys_mem_peak: 5092409344 sys_mem_rss: 313737216 sys_mem_size: 5076869120 sys_mem_total: 8328691712 sys_ru_stime: 0.228014 sys_ru_utime: 0.912057

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.694241566489: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 22:23 ktout.log

Got message from job at time: 1387423456.44 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD2/tmp_IrlmJxCF6H/tmp_hTNSGcAUi3_kill.txt Got message from job at time: 1387423457.39 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD2/tmp_IrlmJxCF6H/tmp_hTNSGcAUi3_kill.txt Got message from job at time: 1387423457.39 : Report for hahnlab-pc01:1978: cnt_get: 1129999 cnt_get_misses: 102964 cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 445086 cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=102962 size=324520860 path=: db_total_count: 102962 db_total_size: 324520860 serv_conn_count: 1 serv_current_time: 1387423446.984993 serv_proc_id: 1461 serv_running_term: 1182.671286 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2398187520 sys_mem_free: 4462174208 sys_mem_peak: 5137113088 sys_mem_rss: 365584384 sys_mem_size: 5095223296 sys_mem_total: 8328691712 sys_ru_stime: 3.536221 sys_ru_utime: 6.820426

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0/Anc0_DB: total 34M -rw-rw-r-- 1 hussainather hussainather 34M Dec 18 22:23 cactusSequences -rw-rw-r-- 1 hussainather hussainather 456 Dec 18 22:04 ktout.log

2013-12-18 22:24:17.475387: Finished Progressive Cactus Alignment

2013-12-18 22:24:17.478569: Beginning HAL Export

3.36user 0.07system 0:03.53elapsed 97%CPU (0avgtext+0avgdata 452832maxresident)k 6416inputs+25720outputs (21major+48566minor)pagefaults 0swaps 7.75user 0.13system 0:07.91elapsed 99%CPU (0avgtext+0avgdata 648496maxresident)k 56inputs+42016outputs (0major+66756minor)pagefaults 0swaps 3.50user 0.09system 0:03.60elapsed 99%CPU (0avgtext+0avgdata 410976maxresident)k 0inputs+17480outputs (0major+38523minor)pagefaults 0swaps rm -f ./2L_new2/b00.hal: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0_hal.fa' '(Anc2:3.0,Anc1:6.0)Anc0;' './2L_new2/b00.hal:' time halAppendCactusSubtree '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1_hal.c2h' '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1_hal.fa' '(dmel:5.0,dsim:5.0,dere:17.0)Anc1;' './2L_new2/b00.hal:' --outgroups dere time halAppendCactusSubtree '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2_hal.c2h' '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2_hal.fa' '(dere:8.0,dyak:8.0,dmel:14.0)Anc2;' './2L_new2/b00.hal:' --outgroups dmel total time: 15.09 total halAppendCactusSubtree time: 15.09

2013-12-18 22:24:33.726012: Finished HAL Export

glennhickey commented 10 years ago

yeah, cactus alignments contain duplications. there's an option in hal2maf to filter them out of the maf output if desired.

On Thu, Dec 19, 2013 at 7:10 AM, Hussain notifications@github.com wrote:

I'm still getting some outputs that have the same file being aligned to itself. For example, this is what one of the .maf files will read:

maf version=1 scoring=N/A

hal ((dere:8,dyak:8)Anc2:3,(dmel:5,dsim:5)Anc1:6)Anc0;

a s Anc0refChr1 0 1 + 7374327 n

a s Anc0refChr1 1 1 + 7374327 c s Anc2refChr1 254010 1 + 8466230 t s Anc2refChr1 361156 1 + 8466230 C s dyak_2L 1104077 1 - 22210002 C s dyak_2L 21661811 1 + 22210002 t

a s Anc0refChr1 2 1 + 7374327 t s Anc2refChr1 254011 1 + 8466230 t s Anc2refChr1 361157 1 + 8466230 n s dyak_2L 21661812 1 + 22210002 t

a s Anc0refChr1 3 137 + 7374327 atacccgttactcgtagagtaaaagggtatactagagtcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccgtatagagtataaatattcttgatcaggatcaaaagccaagtcgatctggccatgtc s Anc2refChr1 254012 137 + 8466230 atacccgttactcgtagagtaaaagggtatactagagtcgttgaaaagtatgtaacaggcagaagaaagcgtttccgaccgtatagagtataaatatacttgatagggatcaaaagccaagtcgatctggccatgtc s Anc2refChr1 361158 137 + 8466230 atacccgttactcgtagagtaaaagggtatactagattcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccatataaagtatatatattcttgatcaggatcagtagccgagtcgatctggccatgtc s dyak_2L 1128800 137 - 22210002 atacccgttactcgaagagtaaaagggtatactagattcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccatataaagtatatatattcttgatcaggatcaatagccgagtcgatttggccatgtc s dyak_2L 21384820 137 + 22210002 atacccgttactcgtagagtaaaagggtatactagattcgttgaaaagtatgtaacaggcagaaggaagcgtttccgaccatataaagtatatatattcttgatcaggatcagtagccgagtcgatctggccatgtc s dyak_2L 21661813 137 + 22210002 atacccgttactcgtagagtaaaaggatatactaaattcgttgaaaagtatgtaacaggcagaagaatgcgtttgcgaccatataaagtatatatattcttgatcaggatcaaaagccaagtcgatctggccatgtc

Here's the cactus.log:

2013-12-18 20:16:15.361480: Beginning Progressive Cactus Alignment

Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dere_2L_new.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dere_2L_new.fasta Total-sequences: 2 Total-length: 2176 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 N50: 0 Median-sequence-length: 1155 Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dyak_2L_new.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dyak_2L_new.fasta Total-sequences: 1 Total-length: 22210002 Proportion-repeat-masked: 0.000157 ProportionNs: 0.000157 N50: 3480 Median-sequence-length: 22210002 Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dro/dmel-2L-new-chromosome-r5.54.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dro/dmel-2L-new-chromosome-r5.54.fasta Total-sequences: 1 Total-length: 23011544 Proportion-repeat-masked: 0.000009 ProportionNs: 0.000009 N50: 200 Median-sequence-length: 23011544 Got message from job at time: 1387415776.59 : Before running any preprocessing on the assembly: /home/hussainather/dsim_2L_new.fasta got following stats (assembly may be listed as temp file if input sequences from a directory): Input-sample: /home/hussainather/dsim_2L_new.fasta Total-sequences: 1 Total-length: 20694855 Proportion-repeat-masked: 0.001849 ProportionNs: 0.001849 N50: 38263 Median-sequence-length: 20694855 Got message from job at time: 1387415777.0 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dere_2L_new.fasta Total-sequences: 2 Total-length: 2176 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 N50: 0 Median-sequence-length: 1155 Got message from job at time: 1387415856.39 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dsim_2L_new.fasta Total-sequences: 1 Total-length: 20694855 Proportion-repeat-masked: 0.005836 ProportionNs: 0.001849 N50: 120769 Median-sequence-length: 20694855 Got message from job at time: 1387415857.08 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dyak_2L_new.fasta Total-sequences: 1 Total-length: 22210002 Proportion-repeat-masked: 0.020924 ProportionNs: 0.000157 N50: 464725 Median-sequence-length: 22210002 Got message from job at time: 1387415863.08 : After preprocessing assembly we got the following stats: Input-sample: ./2L_new2/sequenceData/_home_hussainather_dro_dmel-2L-new-chromosome-r5.54.fasta Total-sequences: 1 Total-length: 23011544 Proportion-repeat-masked: 0.014328 ProportionNs: 0.000009 N50: 329700 Median-sequence-length: 23011544 Got message from job at time: 1387415863.32 : The longest path in the tree is 14.000000 Got message from job at time: 1387415863.32 : Made argument lastzArguments=--step=1 --ambiguous=iupac,100 --ydrop=3000 in tag caf with divergence threshold of 9223372036854775807 for longest path of 14.0 (useDefaultDivergences=False) Got message from job at time: 1387415873.42 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD4/tmp_v2ENgQtjyl/tmp_zKUPY7jlAX_kill.txt Got message from job at time: 1387415874.38 : The blast stage will filter by identity, the calculated minimum identity is 25 from a longest path of 14.0 and an adjusted path of 42.0 Got message from job at time: 1387415874.38 : Starting caf phase target with index 0 at 1387415874.08 seconds (recursing = 1) Got message from job at time: 1387415874.38 : Adding an oversize flower for target class and stats flower name: 0 total bases: 45223722 total-ends: 8 total-caps: 8 max-end-degree: 1 max-adjacency-length: 23011545 total-blocks: 0 total-groups: 1 total-edges: 4 total-free-ends: 8 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1387417126.18 : Pinch graph component with 978 nodes and 1157 edges is being split up by breaking 120 edges to reduce size to less than 432 max, but found 0 pointless edges Got message from job at time: 1387417126.18 : Attaching the sequence to the cactus root 4973381363500545548, header dmel_2L with length 23011544 and 90805 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387417126.18 : Starting bar phase target with index 0 at 1387417125.25 seconds (recursing = 1) Got message from job at time: 1387417126.18 : Adding an oversize flower for target class and stats flower name: 7213218490675588443 total bases: 22699115 total-ends: 533 total-caps: 1566 max-end-degree: 13 max-adjacency-length: 646660 total-blocks: 0 total-groups: 1 total-edges: 579 total-free-ends: 1 total-attached-ends: 532 total-chains: 0 total-link groups: 0 Got message from job at time: 1387417126.18 : Adding an oversize flower for target class and stats flower name: 7213218490675588503 total bases: 19669494 total-ends: 533 total-caps: 1496 max-end-degree: 11 max-adjacency-length: 1308294 total-blocks: 0 total-groups: 1 total-edges: 559 total-free-ends: 0 total-attached-ends: 533 total-chains: 0 total-link groups: 0 Got message from job at time: 1387417126.3 : Breaking bar job into 21 separate jobs Got message from job at time: 1387417134.64 : Breaking bar job into 18 separate jobs Got message from job at time: 1387419376.71 : Starting avg phase target with index 0 at 1387419376.63 seconds (recursing = 0) Got message from job at time: 1387419376.71 : Starting reference phase target with index 0 at 1387419376.64 seconds (recursing = 1) Got message from job at time: 1387419386.86 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_w6pL62aVpY_kill.txt Got message from job at time: 1387419417.86 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_w6pL62aVpY_kill.txt Got message from job at time: 1387419419.38 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_w6pL62aVpY_kill.txt Got message from job at time: 1387419419.38 : Report for hahnlab-pc01:2080: cnt_get: 13653 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 13651 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 13653

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=2 size=274200414 path=: db_total_count: 2 db_total_size: 274200414 serv_conn_count: 1 serv_current_time: 1387419407.801119 serv_proc_id: 29432 serv_running_term: 31.050650 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 1998442496 sys_mem_free: 5061177344 sys_mem_peak: 5137113088 sys_mem_rss: 66117632 sys_mem_size: 5074075648 sys_mem_total: 8328691712 sys_ru_stime: 0.096006 sys_ru_utime: 0.124007

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2/Anc2_DB_tempSecondaryDatabaseDir_0.155984206167: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 21:16 ktout.log

Got message from job at time: 1387419419.56 : Starting reference phase target with index 0 at 1387419419.39 seconds (recursing = 1) Got message from job at time: 1387419419.56 : Adding an oversize flower 2059974617056921301 for target class Got message from job at time: 1387419426.34 : Starting Reference Extract Phase Got message from job at time: 1387419426.34 : Starting check phase target with index 0 at 1387419426.25 seconds (recursing = 0) Got message from job at time: 1387419436.48 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_tiMGyzIOSb_kill.txt Got message from job at time: 1387419436.55 : Adding an oversize flower 2059974617056921301 for target class Got message from job at time: 1387419447.48 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_tiMGyzIOSb_kill.txt Got message from job at time: 1387419448.17 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_K5eacNH8nh/tmp_tiMGyzIOSb_kill.txt Got message from job at time: 1387419448.17 : Report for hahnlab-pc01:2080: cnt_get: 60013 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 60007 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 60013

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=6 size=275368427 path=: db_total_count: 6 db_total_size: 275368427 serv_conn_count: 1 serv_current_time: 1387419442.911553 serv_proc_id: 29913 serv_running_term: 16.561421 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2081054720 sys_mem_free: 4876894208 sys_mem_peak: 5137113088 sys_mem_rss: 161419264 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.144009 sys_ru_utime: 0.240015

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2/Anc2_DB_tempSecondaryDatabaseDir_0.155984206167: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 21:17 ktout.log

Got message from job at time: 1387419458.06 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD4/tmp_v2ENgQtjyl/tmp_zKUPY7jlAX_kill.txt Got message from job at time: 1387419458.63 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD4/tmp_v2ENgQtjyl/tmp_zKUPY7jlAX_kill.txt Got message from job at time: 1387419458.63 : Report for hahnlab-pc01:1980: cnt_get: 141199 cnt_get_misses: 16416

cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 55217

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=16414 size=298891953 path=: db_total_count: 16414 db_total_size: 298891953 serv_conn_count: 1 serv_current_time: 1387419448.210416 serv_proc_id: 23456 serv_running_term: 3584.885500 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2081177600 sys_mem_free: 5040168960 sys_mem_peak: 5103501312 sys_mem_rss: 125767680 sys_mem_size: 5070147584 sys_mem_total: 8328691712 sys_ru_stime: 2.520157 sys_ru_utime: 3.132195

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2/Anc2_DB: total 52M -rw-rw-r-- 1 hussainather hussainather 52M Dec 18 21:16 cactusSequences -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 20:17 ktout.log

Got message from job at time: 1387419458.88 : The longest path in the tree is 17.000000 Got message from job at time: 1387419458.88 : Made argument lastzArguments=--step=1 --ambiguous=iupac,100 --ydrop=3000 in tag caf with divergence threshold of 9223372036854775807 for longest path of 17.0 (useDefaultDivergences=False) Got message from job at time: 1387419468.99 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD3/tmp_XLF3ev9qdg/tmp_bnHjmrPAqM_kill.txt Got message from job at time: 1387419470.03 : The blast stage will filter by identity, the calculated minimum identity is 25 from a longest path of 17.0 and an adjusted path of 51.0 Got message from job at time: 1387419470.03 : Starting caf phase target with index 0 at 1387419469.69 seconds (recursing = 1) Got message from job at time: 1387419470.03 : Adding an oversize flower for target class and stats flower name: 0 total bases: 43708575 total-ends: 8 total-caps: 8 max-end-degree: 1 max-adjacency-length: 23011545 total-blocks: 0 total-groups: 1 total-edges: 4 total-free-ends: 8 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1387420805.78 : Pinch graph component with 903 nodes and 1087 edges is being split up by breaking 118 edges to reduce size to less than 622 max, but found 0 pointless edges Got message from job at time: 1387420805.78 : Attaching the sequence to the cactus root 1839720447780834549, header dmel_2L with length 23011544 and 16650219 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387420805.78 : Starting bar phase target with index 0 at 1387420805.69 seconds (recursing = 1) Got message from job at time: 1387422097.7 : Starting avg phase target with index 0 at 1387422097.63 seconds (recursing = 0) Got message from job at time: 1387422097.7 : Starting reference phase target with index 0 at 1387422097.63 seconds (recursing = 1) Got message from job at time: 1387422107.85 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_eOhDcbSUVp_kill.txt Got message from job at time: 1387422178.83 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_eOhDcbSUVp_kill.txt Got message from job at time: 1387422179.55 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_eOhDcbSUVp_kill.txt Got message from job at time: 1387422179.55 : Report for hahnlab-pc01:2079: cnt_get: 259450 cnt_get_misses: 1 cnt_misc: 0 cnt_remove: 259448 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 259449

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=1 size=268435759 path=: db_total_count: 1 db_total_size: 268435759 serv_conn_count: 1 serv_current_time: 1387422172.214568 serv_proc_id: 658 serv_running_term: 74.502898 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2180894720 sys_mem_free: 4239634432 sys_mem_peak: 5137113088 sys_mem_rss: 251650048 sys_mem_size: 5070004224 sys_mem_total: 8328691712 sys_ru_stime: 0.176011 sys_ru_utime: 0.796049

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1/Anc1_DB_tempSecondaryDatabaseDir_0.903517412895: total 13M -rw-rw-r-- 1 hussainather hussainather 13M Dec 18 22:02 Anc1.kch.BIGRECORDFILE__3702803318628653371 -rw-rw-r-- 1 hussainather hussainather 455 Dec 18 22:01 ktout.log

Got message from job at time: 1387422181.85 : Starting reference phase target with index 0 at 1387422179.56 seconds (recursing = 1) Got message from job at time: 1387422181.85 : Adding an oversize flower 2339197793953890069 for target class Got message from job at time: 1387422209.66 : Starting Reference Extract Phase Got message from job at time: 1387422209.66 : Starting check phase target with index 0 at 1387422209.57 seconds (recursing = 0) Got message from job at time: 1387422219.78 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_twONwSWPEo_kill.txt Got message from job at time: 1387422219.87 : Adding an oversize flower 2339197793953890069 for target class Got message from job at time: 1387422260.76 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_twONwSWPEo_kill.txt Got message from job at time: 1387422261.9 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_62yvw6OpEp/tmp_twONwSWPEo_kill.txt Got message from job at time: 1387422261.9 : Report for hahnlab-pc01:2079: cnt_get: 839669 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 839663 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 839669

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=6 size=286809989 path=: db_total_count: 6 db_total_size: 286809989 serv_conn_count: 1 serv_current_time: 1387422257.669621 serv_proc_id: 1171 serv_running_term: 48.003631 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2301743104 sys_mem_free: 3995082752 sys_mem_peak: 5206835200 sys_mem_rss: 347586560 sys_mem_size: 5079363584 sys_mem_total: 8328691712 sys_ru_stime: 0.364022 sys_ru_utime: 2.576161

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1/Anc1_DB_tempSecondaryDatabaseDir_0.903517412895: total 4.0K -rw-rw-r-- 1 hussainather hussainather 456 Dec 18 22:03 ktout.log

Got message from job at time: 1387422262.52 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD3/tmp_XLF3ev9qdg/tmp_bnHjmrPAqM_kill.txt Got message from job at time: 1387422264.11 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD3/tmp_XLF3ev9qdg/tmp_bnHjmrPAqM_kill.txt Got message from job at time: 1387422264.11 : Report for hahnlab-pc01:1979: cnt_get: 2953891 cnt_get_misses: 263895

cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 1159766

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=263887 size=400792384 path=: db_total_count: 263887 db_total_size: 400792384 serv_conn_count: 1 serv_current_time: 1387422261.941090 serv_proc_id: 30285 serv_running_term: 2803.057079 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2301710336 sys_mem_free: 4345667584 sys_mem_peak: 5092704256 sys_mem_rss: 580976640 sys_mem_size: 5088739328 sys_mem_total: 8328691712 sys_ru_stime: 8.372523 sys_ru_utime: 16.729045

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1/Anc1_DB: total 81M -rw-rw-r-- 1 hussainather hussainather 21M Dec 18 22:03 Anc1.kch.BIGRECORDFILE__2339197793953890069 -rw-rw-r-- 1 hussainather hussainather 60M Dec 18 22:02 cactusSequences -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 21:17 ktout.log

Got message from job at time: 1387422264.31 : Extending longest path Got message from job at time: 1387422264.31 : The longest path in the tree is 6.100000 Got message from job at time: 1387422264.31 : Made argument lastzArguments=--step=1 --ambiguous=iupac,100 --ydrop=3000 in tag caf with divergence threshold of 9223372036854775807 for longest path of 6.1 (useDefaultDivergences=False) Got message from job at time: 1387422274.42 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD2/tmp_IrlmJxCF6H/tmp_hTNSGcAUi3_kill.txt Got message from job at time: 1387422275.07 : The blast stage will filter by identity, the calculated minimum identity is 26 from a longest path of 6.1 and an adjusted path of 18.3 Got message from job at time: 1387422275.07 : Starting caf phase target with index 0 at 1387422274.84 seconds (recursing = 1) Got message from job at time: 1387422275.07 : Adding an oversize flower for target class and stats flower name: 0 total bases: 27601997 total-ends: 6 total-caps: 6 max-end-degree: 1 max-adjacency-length: 19135527 total-blocks: 0 total-groups: 1 total-edges: 3 total-free-ends: 6 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1387422989.07 : Pinch graph component with 654 nodes and 684 edges is being split up by breaking 3 edges to reduce size to less than 574 max, but found 0 pointless edges Got message from job at time: 1387422989.07 : Attaching the sequence to the cactus root 9126685382354600639, header Anc1refChr0 with length 19135526 and 6322551 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1387422989.07 : Starting bar phase target with index 0 at 1387422988.98 seconds (recursing = 1) Got message from job at time: 1387423002.28 : Adding an oversize flower for target class and stats flower name: 4499518240208160293 total bases: 8442425 total-ends: 577 total-caps: 1406 max-end-degree: 8 max-adjacency-length: 617959 total-blocks: 0 total-groups: 1 total-edges: 604 total-free-ends: 0 total-attached-ends: 577 total-chains: 0 total-link groups: 0 Got message from job at time: 1387423002.36 : Breaking bar job into 8 separate jobs Got message from job at time: 1387423360.29 : Starting avg phase target with index 0 at 1387423360.22 seconds (recursing = 0) Got message from job at time: 1387423360.29 : Starting reference phase target with index 0 at 1387423360.23 seconds (recursing = 1) Got message from job at time: 1387423370.4 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4ITet6blIJ_kill.txt Got message from job at time: 1387423401.39 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4ITet6blIJ_kill.txt Got message from job at time: 1387423402.93 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4ITet6blIJ_kill.txt Got message from job at time: 1387423402.93 : Report for hahnlab-pc01:2078: cnt_get: 98898 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 98896 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 98898

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=2 size=273463480 path=: db_total_count: 2 db_total_size: 273463480 serv_conn_count: 1 serv_current_time: 1387423394.560311 serv_proc_id: 12455 serv_running_term: 34.257573 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2336792576 sys_mem_free: 4374437888 sys_mem_peak: 5137113088 sys_mem_rss: 162426880 sys_mem_size: 5075746816 sys_mem_total: 8328691712 sys_ru_stime: 0.136008 sys_ru_utime: 0.312019

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.694241566489: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 22:22 ktout.log

Got message from job at time: 1387423414.5 : Starting reference phase target with index 0 at 1387423402.94 seconds (recursing = 1) Got message from job at time: 1387423414.5 : Starting Reference Extract Phase Got message from job at time: 1387423414.51 : Starting check phase target with index 0 at 1387423414.43 seconds (recursing = 0) Got message from job at time: 1387423424.62 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4P3w559oxv_kill.txt Got message from job at time: 1387423445.64 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4P3w559oxv_kill.txt Got message from job at time: 1387423446.95 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/t1/gTD1/tmp_6QChxYbZkb/tmp_4P3w559oxv_kill.txt Got message from job at time: 1387423446.95 : Report for hahnlab-pc01:2078: cnt_get: 305982 cnt_get_misses: 0 cnt_misc: 0 cnt_remove: 305977 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 305982

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=5 size=275295852 path=: db_total_count: 5 db_total_size: 275295852 serv_conn_count: 1 serv_current_time: 1387423438.567249 serv_proc_id: 12885 serv_running_term: 24.054232 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2398117888 sys_mem_free: 4145618944 sys_mem_peak: 5092409344 sys_mem_rss: 313737216 sys_mem_size: 5076869120 sys_mem_total: 8328691712 sys_ru_stime: 0.228014 sys_ru_utime: 0.912057

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0/Anc0_DB_tempSecondaryDatabaseDir_0.694241566489: total 4.0K -rw-rw-r-- 1 hussainather hussainather 457 Dec 18 22:23 ktout.log

Got message from job at time: 1387423456.44 : Launching ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD2/tmp_IrlmJxCF6H/tmp_hTNSGcAUi3_kill.txt Got message from job at time: 1387423457.39 : Killing ktserver with killPath /home/hussainather/progressiveCactus/2L_new2/jobTree/jobs/gTD2/tmp_IrlmJxCF6H/tmp_hTNSGcAUi3_kill.txt Got message from job at time: 1387423457.39 : Report for hahnlab-pc01:1978: cnt_get: 1129999 cnt_get_misses: 102964

cnt_misc: 0 cnt_remove: 0 cnt_remove_misses: 0 cnt_script: 0 cnt_set: 445086

cnt_set_misses: 0 conf_kc_features: (atomic)(zlib) conf_kc_version: 1.2.76 (16.13) conf_kt_features: (epoll) conf_kt_version: 0.9.56 (2.19) conf_os_name: Linux db_0: count=102962 size=324520860 path=: db_total_count: 102962 db_total_size: 324520860 serv_conn_count: 1 serv_current_time: 1387423446.984993 serv_proc_id: 1461 serv_running_term: 1182.671286 serv_task_count: 0 serv_thread_count: 64 sys_mem_cached: 2398187520 sys_mem_free: 4462174208 sys_mem_peak: 5137113088 sys_mem_rss: 365584384 sys_mem_size: 5095223296 sys_mem_total: 8328691712 sys_ru_stime: 3.536221 sys_ru_utime: 6.820426

Contents of /home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0/Anc0_DB: total 34M -rw-rw-r-- 1 hussainather hussainather 34M Dec 18 22:23 cactusSequences -rw-rw-r-- 1 hussainather hussainather 456 Dec 18 22:04 ktout.log

2013-12-18 22:24:17.475387: Finished Progressive Cactus Alignment

2013-12-18 22:24:17.478569: Beginning HAL Export

3.36user 0.07system 0:03.53elapsed 97%CPU (0avgtext+0avgdata 452832maxresident)k 6416inputs+25720outputs (21major+48566minor)pagefaults 0swaps 7.75user 0.13system 0:07.91elapsed 99%CPU (0avgtext+0avgdata 648496maxresident)k 56inputs+42016outputs (0major+66756minor)pagefaults 0swaps 3.50user 0.09system 0:03.60elapsed 99%CPU (0avgtext+0avgdata 410976maxresident)k 0inputs+17480outputs (0major+38523minor)pagefaults 0swaps rm -f ./2L_new2/b00.hal: time halAppendCactusSubtree '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0_hal.c2h' '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc0/Anc0_hal.fa' '(Anc2:3.0,Anc1:6.0)Anc0;' './2L_new2/b00.hal:' time halAppendCactusSubtree '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1_hal.c2h' '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc1/Anc1_hal.fa' '(dmel:5.0,dsim:5.0,dere:17.0)Anc1;' './2L_new2/b00.hal:' --outgroups dere time halAppendCactusSubtree '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2_hal.c2h' '/home/hussainather/progressiveCactus/2L_new2/progressiveAlignment/Anc2/Anc2_hal.fa' '(dere:8.0,dyak:8.0,dmel:14.0)Anc2;' './2L_new2/b00.hal:' --outgroups dmel total time: 15.09 total halAppendCactusSubtree time: 15.09

2013-12-18 22:24:33.726012: Finished HAL Export

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30924033 .

Mcfoofa commented 10 years ago

okay, so if I filter out with --noDupes and --noAncestors, I get this

Segmentation fault (core dumped) Traceback (most recent call last): File "/home/hussainather/progressiveCactus/submodules/hal/bin/hal2mafMP.py", line 309, in sys.exit(main()) File "/home/hussainather/progressiveCactus/submodules/hal/bin/hal2mafMP.py", line 306, in main runParallelSlices(args) File "/home/hussainather/progressiveCactus/submodules/hal/bin/hal2mafMP.py", line 171, in runParallelSlices runParallelShellCommands(sliceCmds, options.numProc) File "/home/hussainather/progressiveCactus/submodules/hal/stats/halStats.py", line 34, in runParallelShellCommands map(runShellCommand, cmdList) File "/home/hussainather/progressiveCactus/submodules/hal/stats/halStats.py", line 27, in runShellCommand (command, sts)) RuntimeError: Command: hal2maf ./2L_new/b00.hal: ./2L_new/b02.hal.maf --unique --noDupes --noAncestors --start 0 --length 7374327 exited with non-zero status 139

glennhickey commented 10 years ago

By default, hal2maf outputs a maf using the root of the tree (Anc0 or Ancestor0) as the reference. It looks like generating the maf with the ancestor reference in conjunction with the --noAncestors option causes a crash. I have two suggestions for an immediate fix: 1) Use the --refGenome option to specify a non-ancestral genome as the reference for hal2maf (you can get a list of the genomes in your alignment with halStats) 2) Do not use the --noAncestors option.

Otherwise, you can wait for me to fix it but that probably won't be till next week. Thanks for bringing this bug to our attention.

On Thu, Dec 19, 2013 at 9:13 AM, Hussain notifications@github.com wrote:

okay, so if I filter out with --noDupes and --noAncestors, I get this

Segmentation fault (core dumped)

Traceback (most recent call last): File "/home/hussainather/progressiveCactus/submodules/hal/bin/hal2mafMP.py", line 309, in sys.exit(main()) File "/home/hussainather/progressiveCactus/submodules/hal/bin/hal2mafMP.py", line 306, in main runParallelSlices(args) File "/home/hussainather/progressiveCactus/submodules/hal/bin/hal2mafMP.py", line 171, in runParallelSlices runParallelShellCommands(sliceCmds, options.numProc) File "/home/hussainather/progressiveCactus/submodules/hal/stats/halStats.py", line 34, in runParallelShellCommands map(runShellCommand, cmdList) File "/home/hussainather/progressiveCactus/submodules/hal/stats/halStats.py", line 27, in runShellCommand (command, sts)) RuntimeError: Command: hal2maf ./2L_new/b00.hal: ./2L_new/b02.hal.maf --unique --noDupes --noAncestors --start 0 --length 7374327 exited with non-zero status 139

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-30930978 .

Mcfoofa commented 10 years ago

Okay, I'm just waiting until the bug is fixed

Mcfoofa commented 10 years ago

Have you gotten around to fixing it yet?

Mcfoofa commented 10 years ago

Any update on it?

glennhickey commented 10 years ago

should be fixed now. Thanks for bringing this to our attention.

On Sat, Jan 4, 2014 at 10:43 AM, Hussain notifications@github.com wrote:

Any update on it?

— Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/11#issuecomment-31580998 .

Mcfoofa commented 10 years ago

Thank you. It works fine when I run --noDupes --refGenome --noAncestors. Could you explain how I get the list of genomes from halStats?

joelarmstrong commented 10 years ago

To get a (space-separated) list of genomes, use "halStats --genomes "