Closed lplough closed 2 years ago
Python versions is unlikely the issue. Make sure that the ACC.taxonomy files matches the NT database, i.e., restrict the blast database to your custom records. If you are getting blast hits outside this custom database you will likely experience those errors.
Thanks @shump2 . I can check to see if all seq IDs in the Taxonomy file (top 3 rows shown above) match the seq Ids from my blastn output. I would imagine that they do since I blasted my OTUs of interest (fasta file) against the custom database that corresponds to the taxonomy file. In other words, I didnt do a blast search against e.g. the entire genbank NT database and then use a different taxonomy file for the downstream steps of BLCA. FYI , I am using the Midori CO1 database formatted for BLCA. THe midori Co1 database i am using has ~ 530K fasta records and the taxonomy file has a very similar # of records (slightly diffferent because there were some redundant seq IDs in fasta database that had to be removed in order for blastn to run).
Or am I missing something more fundamental about how BLCA works? I have my fasta file of interest (~ 609 OTUs formated as a fasta file , i.e. 609 different sequences), the blast formated database for the Midori CO1 seqs (530K sequences), the blastn results of the my query (609 seqs) against the midori CO1 seqs, and then the taxonomy file for the Midori CO1 seqs.
Thanks @shump2 for your help! @lplough if you still have trouble, I wonder if you can make your dataset temporarily downloadable for us to give it a try, we will delete your files afterwards, as we are not interested in doing your study :-)
@qunfengdong - can I email you the files directly via dropbox?
FYI - BLASTn seq's appear to be in the taxonomy file.
$cat midori_blca_dedup.fasta | grep -o ">" | wc -l
582591
$cat midori_blca_taxonomy.dedup.txt | wc -l
582591
Same # of Fasta seqeunces in the database and in the taxonomy file
head midori_blca_taxonomy.dedup.txt
AB000675 species:Paralichthys olivaceus;genus:Paralichthys;family:Paralichthyidae;order:Pleuronectiformes;class:Actinopteri;phylum:Chordata;superkingdom:Eukaryota;
AB002173 species:Henosepilachna enneasticta;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002175 species:Henosepilachna boisduvali;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002176 species:Henosepilachna septima;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002177 species:Henosepilachna pusillanima;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002178 species:Epilachna admirabilis;genus:Epilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002180 species:Henosepilachna vigintioctopunctata;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002181 species:Henosepilachna vigintioctopunctata;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002182 species:Henosepilachna vigintioctomaculata;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
AB002183 species:Henosepilachna pustulosa;genus:Henosepilachna;family:Coccinellidae;order:Coleoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
head 97.lulu.centroids.FINAL.blastn
OTU2 HM191376 95.122 41 2 0 257 297 621 661 1.77e-09 65.8 35 plus 1237 313
OTU2 GQ154298 97.297 37 1 0 261 297 586 622 6.36e-09 63.9 34 plus 638 313
OTU2 KT193300 97.143 35 1 0 257 291 598 632 8.23e-08 60.2 32 plus 651 313
OTU2 HE964896 100.000 32 0 0 261 292 605 636 8.23e-08 60.2 32 plus 1143 313
OTU2 GQ341690 100.000 32 0 0 261 292 606 637 8.23e-08 60.2 32 plus 658 313
OTU2 EU311294 100.000 32 0 0 263 294 595 626 8.23e-08 60.2 32 plus 645 313
OTU2 EU311287 100.000 32 0 0 263 294 595 626 8.23e-08 60.2 32 plus 645 313
OTU2 EU311286 100.000 32 0 0 263 294 595 626 8.23e-08 60.2 32 plus 645 313
OTU2 KP113679 100.000 31 0 0 261 291 605 635 2.96e-07 58.4 31 plus 657 313
OTU2 KJ082971 100.000 31 0 0 261 291 605 635 2.96e-07 58.4 31 plus 657 313
first lines of the blastn output and taxonomy file
AND, grepping the first 10 seqs matched for OTU2 in blastn output against the taxonomy file, finds those sequences in the tax file, so I dont think this is an issue of mismatches between the BLAST database and taxonomy files.....
$head 97.lulu.centroids.FINAL.blastn | cut -f2 | grep -f /dev/stdin midori_blca_taxonomy.dedup.txt
EU311286 species:Tubifex blanchardi;genus:Tubifex;family:Tubificidae;order:Haplotaxida;class:Clitellata;phylum:Annelida;superkingdom:Eukaryota;
EU311287 species:Tubifex blanchardi;genus:Tubifex;family:Tubificidae;order:Haplotaxida;class:Clitellata;phylum:Annelida;superkingdom:Eukaryota;
EU311294 species:Tubifex blanchardi;genus:Tubifex;family:Tubificidae;order:Haplotaxida;class:Clitellata;phylum:Annelida;superkingdom:Eukaryota;
GQ154298 species:Dacus humeralis;genus:Dacus;family:Tephritidae;order:Diptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
GQ341690 species:Cymothoe zenkeri;genus:Cymothoe;family:Nymphalidae;order:Lepidoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
HE964896 species:Cymothoe zenkeri;genus:Cymothoe;family:Nymphalidae;order:Lepidoptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
HM191376 species:Stachyris ruficeps;genus:Stachyris;family:Timaliidae;order:Passeriformes;class:Aves;phylum:Chordata;superkingdom:Eukaryota;
KJ082971 species:Tachydromia luang;genus:Tachydromia;family:Hybotidae;order:Diptera;class:Insecta;phylum:Arthropoda;superkingdom:Eukaryota;
KP113679 species:Metapenaeopsis palmensis;genus:Metapenaeopsis;family:Penaeidae;order:Decapoda;class:Malacostraca;phylum:Arthropoda;superkingdom:Eukaryota;
KT193300 species:Malapterurus microstoma;genus:Malapterurus;family:Malapteruridae;order:Siluriformes;class:Actinopteri;phylum:Chordata;superkingdom:Eukaryota;
Hello,
I am a postdoc at Dr. Dong's lab. I am currently maintaining the BLCA application. You may just email me the files via dropbox and I'll take a look at them and try to figure out the problem. My email is @.*** Thank you!
Best, Yue
On Thu, Sep 30, 2021 at 4:05 PM lplough @.***> wrote:
@qunfengdong https://github.com/qunfengdong - can I email you the files directly via dropbox?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/qunfengdong/BLCA/issues/29#issuecomment-931625665, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKABWIRRPAKWXOZ6VP7RJKTUES7HFANCNFSM5FARMLVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
Hi @YJulyXing Thanks! You email is hidden (as is mine I think - standard github). So, I opened a dummy repository (private) and invited you so you can access my files without making them fully public. It has dropbox links to the data. If you have any problems with accessing them, just let me know. Thanks!
@YJulyXing @qunfengdong Did you get my dropbox links to my dataset? Please let me know ifyou had any problems with.
Yes, we have downloaded the 3 files you provided. I'll take a look and see what the issue is.
On Tue, Oct 5, 2021 at 1:59 PM lplough @.***> wrote:
@YJulyXing https://github.com/YJulyXing @qunfengdong https://github.com/qunfengdong Did you get my dropbox links to my dataset? Please let me know ifyou had any problems with.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/qunfengdong/BLCA/issues/29#issuecomment-934638151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKABWIW2P5LQ2AGQTRG5BADUFM4INANCNFSM5FARMLVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
Hello,
I was wondering for the "xxx.ACC.taxonomy" file you created in the database, is it empty or not?
On Tue, Oct 5, 2021 at 3:10 PM Yue "July" Xing @.***> wrote:
Yes, we have downloaded the 3 files you provided. I'll take a look and see what the issue is.
On Tue, Oct 5, 2021 at 1:59 PM lplough @.***> wrote:
@YJulyXing https://github.com/YJulyXing @qunfengdong https://github.com/qunfengdong Did you get my dropbox links to my dataset? Please let me know ifyou had any problems with.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/qunfengdong/BLCA/issues/29#issuecomment-934638151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKABWIW2P5LQ2AGQTRG5BADUFM4INANCNFSM5FARMLVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
Hello,
Please discard my previous message. I was able to run BLCA using your given files (with some modification for the database file) without errors.
Here's what I did:
Finally I ran 2.blca_main.py and it worked without errors.
Please let me know if you have further questions or if you would like the modified db file.
On Thu, Oct 7, 2021 at 9:46 PM Yue "July" Xing @.***> wrote:
Hello,
I was wondering for the "xxx.ACC.taxonomy" file you created in the database, is it empty or not?
On Tue, Oct 5, 2021 at 3:10 PM Yue "July" Xing @.***> wrote:
Yes, we have downloaded the 3 files you provided. I'll take a look and see what the issue is.
On Tue, Oct 5, 2021 at 1:59 PM lplough @.***> wrote:
@YJulyXing https://github.com/YJulyXing @qunfengdong https://github.com/qunfengdong Did you get my dropbox links to my dataset? Please let me know ifyou had any problems with.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/qunfengdong/BLCA/issues/29#issuecomment-934638151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKABWIW2P5LQ2AGQTRG5BADUFM4INANCNFSM5FARMLVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
-- Yue "July" Xing, Ph.D. Postdoctoral research associate Ph.D. in Genetics and MS in Statistics at Texas A&M University Center for Biomedical Informatics Department of Medicine Stritch School of Medicine Loyola University Chicago
Thanks @ YJulyXing! I didn't realize that my taxonomy file was improperly formatted - what looked like a tab was NOT a tab ;). I was able to fix it based on your recommendations (proper spacing and removed unexpected characters/spaces in the species names) and ran BLCA just fine!
Thanks again for looking at my data and for your help! Sorry that I didn't pick up on that formatting error earlier.
Cheers! Louis
I am having a similar issue to @shump2 (#19 and #28) using a custom database for CO1 that looks like this (formatted with tab between sequence id and taxonomy info (as in your 16s example).
Blastn runs fine (blast+ 2.9 or 2.12) then the script fails on the alignment. I am using python 3. In not sure how to install biopython for python 2 in a straigthtforward way - i have seen you suggest to others that this error may be corrected w/ python 2?
sample of Blastn output:
See details here:
same errors generated with clustalo or muscle. Must be a python error...
Any insight is much appreciated!
UPDATE - running with python2 and the current or 2.1 version of the program, I get similar errors:
Thanks, Louis