gjospin
commented
9 years ago Hello, First off, the taxa.csv ffle was part of a feature that did not work properly and that we ended up not fully developing. So you can ignore that part. I should spend some time to either remove or skip that part of the code so people don't get confused.
The PD number seems rather high but I am not a tree expert so maybe it will work for your purpose. It is intended to trim the resulting tree so that there are no branches smaller than the number. By default we use 0.01 I think. This ends up removing leaves that are really similar to reduce the ambiguity of the placements.
You will want to look at the reps file after the marker build which are picked to maximize the PD of the representatives. Those will be the sequences used in the Lastal matching before aligning to the HMM.
I would suggest to dial down the tree_pd value and see how it goes. Your output seems fairly normal otherwise.
Let us know if you have more questions.
On Tue, Mar 31, 2015 at 12:52 PM, KevinAMeyer notifications@github.com wrote:
Hello,
I'm looking to create a custom database to run in Phylosift and keep getting error messages when trying to build the database using The Monkey. I can index the created database, but the taxa file (taxa.csv) is empty, almost all of my tax_ids have been removed, and when I run phylosift all --custom using the database created by The Monkey it returns no hits whatsoever. Including hits to reference genomes that I used to build the database.
Is this a file formatting issue? I'm not quite sure why I'm getting these error messages as all of my TaxonIDs are in the ncbi database files with the correct taxonomy.
Any assistance would be appreciated.
Cheers, Kevin A. Meyer
Here is the command and output from one such run in the terminal.
phylosift build_marker --force --alignment gene_alignment.fasta --taxonmap gi_mapping_file.txt --tree_pd 1
Checking integrity on File gene_alignment.fasta Using /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.aln ID_map is 28 long FastTree Version 2.1.3 SSE3 Alignment: /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.clean Amino acid distances: BLOSUM45 Joins: balanced Support: SH-like 1000 Search: Normal +NNI +SPR (2 rounds range 10) +ML-NNI opt-each=1 TopHits: 1.00*sqrtN close=default refresh=0.80 ML Model: Jones-Taylor-Thorton, CAT approximation with 20 rate categories Initial topology in 0.01 seconds Refining topology: 18 rounds ME-NNIs, 2 rounds ME-SPRs, 9 rounds ML-NNIs Total branch-length 1.798 after 0.07 sec ML-NNI round 1: LogLk = -3459.471 NNIs 4 max delta 3.18 Time 0.27 Switched to using 20 rate categories (CAT approximation)1 of 20
Rate categories were divided by 0.839 so that average rate = 1.0 CAT-based log-likelihoods may not be comparable across runs Use -gamma for approximate but comparable Gamma(20) log-likelihoods ML-NNI round 2: LogLk = -3079.023 NNIs 1 max delta 0.03 Time 0.55 Turning off heuristics for final round of ML NNIs (converged) ML-NNI round 3: LogLk = -3078.406 NNIs 1 max delta 0.02 Time 0.80 (final) Optimize all lengths: LogLk = -3078.386 Time 0.86 Total time: 1.08 seconds Unique: 24/28 Bad splits: 0/21 PDA - Phylogenetic Diversity Analyzer version 0.5.2 Copyright (c) 2006-2007 Bui Quang Minh, Steffen Klaere and Arndt von Haeseler.
Reading tree file /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tree ... Tree contains 28 taxa and 53 branches
Running PD algorithm on UNROOTED tree...
Greedy Algorithm... Time used: 0.000000 seconds.
Results are summarized in /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tree.pruning.log
FastTree Version 2.1.3 SSE3 Alignment: /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.fasta Amino acid distances: BLOSUM45 Joins: balanced Support: SH-like 1000 Search: Normal +NNI +SPR (2 rounds range 10) +ML-NNI opt-each=1 TopHits: 1.00*sqrtN close=default refresh=0.80 ML Model: Jones-Taylor-Thorton, CAT approximation with 20 rate categories Initial topology in 0.00 seconds Refining topology: 6 rounds ME-NNIs, 2 rounds ME-SPRs, 3 rounds ML-NNIs Total branch-length 0.598 after 0.00 sec ML-NNI round 1: LogLk = -1231.497 NNIs 0 max delta 0.00 Time 0.00 Switched to using 20 rate categories (CAT approximation) Rate categories were divided by 0.735 so that average rate = 1.0 CAT-based log-likelihoods may not be comparable across runs Use -gamma for approximate but comparable Gamma(20) log-likelihoods ML-NNI round 2: LogLk = -1180.745 NNIs 0 max delta 0.00 Time 0.01 Turning off heuristics for final round of ML NNIs (converged) ML-NNI round 3: LogLk = -1180.745 NNIs 0 max delta 0.00 Time 0.01 (final) Optimize all lengths: LogLk = -1180.368 Time 0.02 Total time: 0.02 seconds Unique: 3/3 Bad splits: 0/0 CLEAN_ALN : /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.cleanLooking for representatives Reading NCBI taxonomy at /geomicro/data21/kevmey/share/phylosift/ncbi Reading merged ncbi nodes Done reading merged Reading deleted ncbi nodes Done reading deleted Running taxit taxtable -d /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/../ncbi_taxonomy.db -t /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/tax_ids.txt -o /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/taxa.csv Taxid 74547 not found in taxonomy. Taxid 449447 not found in taxonomy. Taxid 316278 not found in taxonomy. Taxid 103690 not found in taxonomy. Taxid 32051 not found in taxonomy. Taxid 84588 not found in taxonomy. Taxid 1148 not found in taxonomy. Taxid 64471 not found in taxonomy. Taxid 59919 not found in taxonomy. Taxid 59922 not found in taxonomy. Taxid 43989 not found in taxonomy. Taxid 329726 not found in taxonomy. Taxid 321327 not found in taxonomy. Taxid 251221 not found in taxonomy. Taxid 180281 not found in taxonomy. Taxid 696747 not found in taxonomy. Taxid 93059 not found in taxonomy. Taxid 197221 not found in taxonomy. Taxid 93060 not found in taxonomy. Taxid 167539 not found in taxonomy. Taxid 167555 not found in taxonomy. Taxid 146891 not found in taxonomy. Taxid 167546 not found in taxonomy. Taxid 321332 not found in taxonomy. Taxid 167542 not found in taxonomy. Some taxids were invalid. Exiting. Using the mapping stuff taxit create -a "Guillaume Jospin" -d "simple package for reconciliation only" -l temp -f /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.clean -t /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.tree -s /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.log -P /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/temp_refcreating tmpread /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tmpread.fasta MAPFILE : gi_mapping_file.txt Return_hash has : 24 taxons TAXON_ARRAY 24 ncbi tree has lots of nodes Read a bunch of merged nodes NCBI COUNT : 24 Making 53 Nodes AFTER ncbi_subtree /opt/packages/PhyloSift/1.0.1/bin/readconciler /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.subtree /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tmpread.mangled /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.gene_map /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.taxonmap The reference tree has 53 nodes The read tree has 4 nodes Done with gene tree splits Making gene tree map 3 genes mapped Making species to gene tree map Error no mapping found for 25 species mapped rs.size() 25 Finding best edges AFTER readconciler Running cd "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment";taxit create -c -d "Creating a reference package for PhyloSift for the gene_alignment marker" -l "gene_alignment" -f "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.clean" -t "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.tree" -s "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.log" -P "gene_alignment"
— Reply to this email directly or view it on GitHub https://github.com/gjospin/PhyloSift/issues/468.
KevinAMeyer
Hello,
I'm looking to create a custom database to run in Phylosift and keep getting error messages when trying to build the database using The Monkey. I can index the created database, but the taxa file (taxa.csv) is empty, almost all of my tax_ids have been removed, and when I run phylosift all --custom using the database created by The Monkey it returns no hits whatsoever. Including hits to reference genomes that I used to build the database.
Is this a file formatting issue? I'm not quite sure why I'm getting these error messages as all of my TaxonIDs are in the ncbi database files with the correct taxonomy.
Any assistance would be appreciated.
Cheers, Kevin A. Meyer
Here is the command and output from one such run in the terminal.
phylosift build_marker --force --alignment gene_alignment.fasta --taxonmap gi_mapping_file.txt --tree_pd 1
Checking integrity on File gene_alignment.fasta Using /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.aln ID_map is 28 long FastTree Version 2.1.3 SSE3 Alignment: /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.clean Amino acid distances: BLOSUM45 Joins: balanced Support: SH-like 1000 Search: Normal +NNI +SPR (2 rounds range 10) +ML-NNI opt-each=1 TopHits: 1.00*sqrtN close=default refresh=0.80 ML Model: Jones-Taylor-Thorton, CAT approximation with 20 rate categories Initial topology in 0.01 seconds Refining topology: 18 rounds ME-NNIs, 2 rounds ME-SPRs, 9 rounds ML-NNIs Total branch-length 1.798 after 0.07 sec ML-NNI round 1: LogLk = -3459.471 NNIs 4 max delta 3.18 Time 0.27 Switched to using 20 rate categories (CAT approximation)1 of 20
Rate categories were divided by 0.839 so that average rate = 1.0 CAT-based log-likelihoods may not be comparable across runs Use -gamma for approximate but comparable Gamma(20) log-likelihoods ML-NNI round 2: LogLk = -3079.023 NNIs 1 max delta 0.03 Time 0.55 Turning off heuristics for final round of ML NNIs (converged) ML-NNI round 3: LogLk = -3078.406 NNIs 1 max delta 0.02 Time 0.80 (final) Optimize all lengths: LogLk = -3078.386 Time 0.86 Total time: 1.08 seconds Unique: 24/28 Bad splits: 0/21 PDA - Phylogenetic Diversity Analyzer version 0.5.2 Copyright (c) 2006-2007 Bui Quang Minh, Steffen Klaere and Arndt von Haeseler.
Reading tree file /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tree ... Tree contains 28 taxa and 53 branches
Running PD algorithm on UNROOTED tree...
Greedy Algorithm... Time used: 0.000000 seconds.
Results are summarized in /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tree.pruning.log
FastTree Version 2.1.3 SSE3 Alignment: /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.fasta Amino acid distances: BLOSUM45 Joins: balanced Support: SH-like 1000 Search: Normal +NNI +SPR (2 rounds range 10) +ML-NNI opt-each=1 TopHits: 1.00*sqrtN close=default refresh=0.80 ML Model: Jones-Taylor-Thorton, CAT approximation with 20 rate categories Initial topology in 0.00 seconds Refining topology: 6 rounds ME-NNIs, 2 rounds ME-SPRs, 3 rounds ML-NNIs Total branch-length 0.598 after 0.00 sec ML-NNI round 1: LogLk = -1231.497 NNIs 0 max delta 0.00 Time 0.00 Switched to using 20 rate categories (CAT approximation) Rate categories were divided by 0.735 so that average rate = 1.0 CAT-based log-likelihoods may not be comparable across runs Use -gamma for approximate but comparable Gamma(20) log-likelihoods ML-NNI round 2: LogLk = -1180.745 NNIs 0 max delta 0.00 Time 0.01 Turning off heuristics for final round of ML NNIs (converged) ML-NNI round 3: LogLk = -1180.745 NNIs 0 max delta 0.00 Time 0.01 (final) Optimize all lengths: LogLk = -1180.368 Time 0.02 Total time: 0.02 seconds Unique: 3/3 Bad splits: 0/0 CLEAN_ALN : /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.cleanLooking for representatives Reading NCBI taxonomy at /geomicro/data21/kevmey/share/phylosift/ncbi Reading merged ncbi nodes Done reading merged Reading deleted ncbi nodes Done reading deleted Running taxit taxtable -d /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/../ncbi_taxonomy.db -t /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/tax_ids.txt -o /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/taxa.csv Taxid 74547 not found in taxonomy. Taxid 449447 not found in taxonomy. Taxid 316278 not found in taxonomy. Taxid 103690 not found in taxonomy. Taxid 32051 not found in taxonomy. Taxid 84588 not found in taxonomy. Taxid 1148 not found in taxonomy. Taxid 64471 not found in taxonomy. Taxid 59919 not found in taxonomy. Taxid 59922 not found in taxonomy. Taxid 43989 not found in taxonomy. Taxid 329726 not found in taxonomy. Taxid 321327 not found in taxonomy. Taxid 251221 not found in taxonomy. Taxid 180281 not found in taxonomy. Taxid 696747 not found in taxonomy. Taxid 93059 not found in taxonomy. Taxid 197221 not found in taxonomy. Taxid 93060 not found in taxonomy. Taxid 167539 not found in taxonomy. Taxid 167555 not found in taxonomy. Taxid 146891 not found in taxonomy. Taxid 167546 not found in taxonomy. Taxid 321332 not found in taxonomy. Taxid 167542 not found in taxonomy. Some taxids were invalid. Exiting. Using the mapping stuff taxit create -a "Guillaume Jospin" -d "simple package for reconciliation only" -l temp -f /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.clean -t /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.tree -s /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.log -P /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/temp_refcreating tmpread /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tmpread.fasta MAPFILE : gi_mapping_file.txt Return_hash has : 24 taxons TAXON_ARRAY 24 ncbi tree has lots of nodes Read a bunch of merged nodes NCBI COUNT : 24 Making 53 Nodes AFTER ncbi_subtree /opt/packages/PhyloSift/1.0.1/bin/readconciler /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.subtree /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.tmpread.mangled /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.gene_map /geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.taxonmap The reference tree has 53 nodes The read tree has 4 nodes Done with gene tree splits Making gene tree map 3 genes mapped Making species to gene tree map Error no mapping found for 25 species mapped rs.size() 25 Finding best edges AFTER readconciler Running cd "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment";taxit create -c -d "Creating a reference package for PhyloSift for the gene_alignment marker" -l "gene_alignment" -f "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.clean" -t "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.tree" -s "/geomicro/data21/kevmey/share/phylosift/markers/gene_alignment/gene_alignment.pruned.log" -P "gene_alignment"