nzmacalasdair
commented
4 years ago Hello,
Thanks for bringing this to our attention, and sorry that you're having trouble getting a core genome alignment from panaroo. From the logfile it looks like the pan-genome inference step of panaroo has completed successfully, but something might have gone wrong in the gene alignment step.
Fortunately, you don't need to run the entire algorithm again to debug this step, as you should be able to use the command panaroo-msa to just rerun the pan or core gene alignment step.
It's hard to tell exactly what went wrong just from the log file, could you please check, for either of your runs, if the following has occured:
- Does the temporary directory "tmp" followed by some string of random charecters still exist for either run? If so, are there are
.fastafiles for each gene in that directory? - Has an
aligned_gene_sequencesdirectory been created for either run, and if so, does it contain aligned FASTA files,.aln.fasfor each gene?
Also, I see that you're running this on a cluster with some kind of job scheduler. MAFFT output relies on STDOUT redirection using the >redirection operator, so I wonder if that might be causing some issues? It may be worth trying to performing the alignment using clustal. You could do this quickly for either run you've already done, using panaroo-msa. The options should be more or less identical to the main panaroo executable, but it also has its own help message.
Thanks!
martinastoycheva
gtonkinhill
Hello,
I installed panaroo through conda and I ran it twice for pan and core alignemnts but the pan genome does not produce any different output except the lack of core gene alignment file and there is no error in the log file. See command and output below command and log file copied.
Command used: $ panaroo -i ./UK_first/*.gff -o results_panaroo -t 18 --clean-mode strict -a pan --aligner mafft
Log file:
pre-processing gff3 files...
================================================================ Program: CD-HIT, V4.8.1 (+OpenMP), Oct 26 2019, 14:51:47 Command: cd-hit -T 18 -i results_panaroo/combined_protein_CDS.fasta -o results_panaroo/combined_protein_cdhit_out.txt -c 0.98 -s 0.98 -aL 0.0 -AL 99999999 -aS 0.0 -AS 99999999 -M 0 -d 999 -g 1 -n 2
Started: Thu Aug 20 11:46:24 2020
================================================================
Output
Your word length is 2, using 5 may be faster! total seq: 763230 longest and shortest : 9624 and 29 Total letters: 253246411 Sequences have been sorted
Approximated minimal memory consumption: Sequence : 347M Buffer : 18 X 21M = 390M Table : 2 X 12M = 24M Miscellaneous : 9M Total : 771M
Table limit with the given memory limit: Max number of representatives: 582515 Max number of word counting entries: 37326436
comparing sequences from 0 to 1474 .---------- new table with 18 representatives comparing sequences from 1474 to 39561 .......... 10000 finished 79 clusters ---------- 23599 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 15962 to 53325 .......... 20000 finished 147 clusters .......... 30000 finished 215 clusters ---------- 23148 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 30177 to 66829 .......... 40000 finished 286 clusters ---------- 22265 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 44564 to 80497 .......... 50000 finished 352 clusters .......... 60000 finished 416 clusters ---------- 20383 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 60114 to 95269 .......... 70000 finished 479 clusters ---------- 19428 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 75841 to 110210 .......... 80000 finished 544 clusters .......... 90000 finished 607 clusters ---------- 19099 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 91111 to 124716 .......... 100000 finished 672 clusters ---------- 17431 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 107285 to 140082 .......... 110000 finished 739 clusters .......... 120000 finished 799 clusters ---------- 17494 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 122588 to 154620 .......... 130000 finished 861 clusters ---------- 16062 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 138558 to 169791 .......... 140000 finished 926 clusters .......... 150000 finished 992 clusters ---------- 15375 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 154416 to 184856 .......... 160000 finished 1053 clusters ---------- 15499 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 169357 to 199050 .......... 170000 finished 1118 clusters .......... 180000 finished 1183 clusters ---------- 13172 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 185878 to 214745 .......... 190000 finished 1248 clusters .......... 200000 finished 1310 clusters ---------- 14360 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 200385 to 228527 .......... 210000 finished 1376 clusters ---------- 10677 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 217850 to 245119 .......... 220000 finished 1429 clusters .......... 230000 finished 1497 clusters ---------- 11501 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 233618 to 260098 .......... 240000 finished 1561 clusters ---------- 11433 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 248665 to 274393 .......... 250000 finished 1628 clusters .......... 260000 finished 1688 clusters ---------- 9970 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 264423 to 289363 .......... 270000 finished 1757 clusters .......... 280000 finished 1815 clusters ---------- 9251 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 280112 to 304267 .......... 290000 finished 1884 clusters ---------- 8041 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 296226 to 319576 .......... 300000 finished 1945 clusters .......... 310000 finished 2009 clusters ---------- 7069 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 312507 to 335043 .......... 320000 finished 2067 clusters ---------- 7078 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 327965 to 349728 .......... 330000 finished 2135 clusters .......... 340000 finished 2204 clusters ---------- 7018 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 342710 to 363736 .......... 350000 finished 2254 clusters ---------- 4894 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 358842 to 379061 .......... 360000 finished 2319 clusters .......... 370000 finished 2374 clusters ---------- 2220 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 376841 to 396160 .......... 380000 finished 2436 clusters .......... 390000 finished 2500 clusters ---------- 3807 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 392353 to 410896 .......... 400000 finished 2567 clusters ---------- 2626 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 408270 to 426018 .......... 410000 finished 2631 clusters .......... 420000 finished 2693 clusters ---------- 1137 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 424881 to 441798 .......... 430000 finished 2746 clusters ---------- 1915 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 439883 to 456050 .......... 440000 finished 2824 clusters .......... 450000 finished 2885 clusters ---------- 1200 remaining sequences to the next cycle ---------- new table with 100 representatives comparing sequences from 454850 to 470269 .......... 460000 finished 2947 clusters .......... 470000 finished 3012 clusters ---------- new table with 94 representatives comparing sequences from 470269 to 484917 .......... 480000 finished 3074 clusters ....---------- new table with 98 representatives comparing sequences from 484917 to 498832 .......... 490000 finished 3141 clusters ........---------- new table with 86 representatives comparing sequences from 498832 to 512051 .......... 500000 finished 3207 clusters .......... 510000 finished 3264 clusters ..---------- new table with 99 representatives comparing sequences from 512051 to 524609 .......... 520000 finished 3330 clusters ....---------- new table with 62 representatives comparing sequences from 524609 to 536540 .......... 530000 finished 3393 clusters ......---------- new table with 75 representatives comparing sequences from 536540 to 547874 .......... 540000 finished 3458 clusters .......---------- new table with 77 representatives comparing sequences from 547874 to 558641 .......... 550000 finished 3516 clusters ........---------- new table with 70 representatives comparing sequences from 558641 to 568870 .......... 560000 finished 3588 clusters ........---------- new table with 62 representatives comparing sequences from 568870 to 578588 .......... 570000 finished 3652 clusters ........---------- new table with 62 representatives comparing sequences from 578588 to 587820 .......... 580000 finished 3715 clusters .......---------- new table with 59 representatives comparing sequences from 587820 to 596590 .......... 590000 finished 3780 clusters ......---------- new table with 65 representatives comparing sequences from 596590 to 604922 .......... 600000 finished 3842 clusters ....---------- new table with 50 representatives comparing sequences from 604922 to 612837 .......... 610000 finished 3911 clusters ..---------- new table with 60 representatives comparing sequences from 612837 to 620356 .......... 620000 finished 3977 clusters ---------- new table with 40 representatives comparing sequences from 620356 to 627499 ...........................---------- new table with 60 representatives comparing sequences from 627499 to 634285 .......... 630000 finished 4046 clusters ....---------- new table with 42 representatives comparing sequences from 634285 to 640732 .......... 640000 finished 4110 clusters ---------- new table with 31 representatives comparing sequences from 640732 to 646856 ..........................---------- new table with 40 representatives comparing sequences from 646856 to 652674 .......... 650000 finished 4170 clusters ..---------- new table with 42 representatives comparing sequences from 652674 to 658201 ..........................---------- new table with 30 representatives comparing sequences from 658201 to 663452 .......... 660000 finished 4239 clusters ...---------- new table with 36 representatives comparing sequences from 663452 to 668440 .........................---------- new table with 36 representatives comparing sequences from 668440 to 673179 .......... 670000 finished 4300 clusters ...---------- new table with 30 representatives comparing sequences from 673179 to 677681 ........................---------- new table with 32 representatives comparing sequences from 677681 to 681958 .......... 680000 finished 4371 clusters .---------- new table with 26 representatives comparing sequences from 681958 to 686021 .........................---------- new table with 25 representatives comparing sequences from 686021 to 689881 .......................---------- new table with 22 representatives comparing sequences from 689881 to 693548 .......... 690000 finished 4429 clusters ...---------- new table with 26 representatives comparing sequences from 693548 to 697032 ........................---------- new table with 24 representatives comparing sequences from 697032 to 700341 .......... 700000 finished 4496 clusters ---------- new table with 18 representatives comparing sequences from 700341 to 703485 .......................---------- new table with 28 representatives comparing sequences from 703485 to 706472 .......................---------- new table with 17 representatives comparing sequences from 706472 to 709309 .......................---------- new table with 22 representatives comparing sequences from 709309 to 712005 .......... 710000 finished 4564 clusters ..---------- new table with 24 representatives comparing sequences from 712005 to 714566 ......................---------- new table with 12 representatives
763230 finished 4929 clusters
Approximated maximum memory consumption: 771M writing new database writing clustering information program completed !
Total CPU time 145.12 running cmd: cd-hit -T 18 -i results_panaroo/combined_protein_CDS.fasta -o results_panaroo/combined_protein_cdhit_out.txt -c 0.98 -s 0.98 -aL 0.0 -AL 99999999 -aS 0.0 -AS 99999999 -M 0 -d 999 -g 1 -n 2 generating initial network... Processing paralogs... collapse mistranslations... Processing depth: 1 Iteration: 1 Iteration: 2 Iteration: 3 Iteration: 4 Processing depth: 2 Iteration: 1 Iteration: 2 Processing depth: 3 Iteration: 1 collapse gene families... Processing depth: 1 Iteration: 1 Iteration: 2 Iteration: 3 Iteration: 4 Processing depth: 2 Iteration: 1 Processing depth: 3 Iteration: 1 trimming contig ends... refinding genes... Number of searches to perform: 50633 Searching... translating hits... removing by consensus... Updating output... Number of refound genes: 388 collapse gene families with refound genes... Processing depth: 1 Iteration: 1 Processing depth: 2 Iteration: 1 Processing depth: 3 Iteration: 1 writing output... generating pan genome MSAs...
============================
Job utilisation efficiency
============================
Job ID: 8978244 Cluster: viking User/Group: ----/clusterusers State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 20 CPU Utilized: 3-02:22:29 CPU Efficiency: 88.45% of 3-12:05:00 core-walltime Job Wall-clock time: 04:12:15 Memory Utilized: 2.77 GB Memory Efficiency: 1.39% of 200.00 GB Requested wall clock time: 1-00:00:00 Actual wall clock time: 04:12:15 Wall clock time efficiency: 17.5% Job queued time: 00:00:25