Installl software on AWS

stevenhwu commented 2 years ago

General AWS Linux

It's is based on CentOS, same family as RedHat. The package management system is 'RPM" via "yum' i.e.

sudo yum list
sudo yum info
sudo yum install

Blast

There are 3 ways to do this,

download the precompiled executable.
Download the *.rpm, and use yum to install it.
Download the source code, and install it from scratch. Not too bad for BLAST, it just takes awhile to run, no major dependency issues.

Prodigal

Instead of downloading the pre-compiled executable, try to clone the repository and compile/install it yourself.

Prokka

Again, instead of using conda, you probably will learn more by installing it from scratch. You want to use the Centos/Fedora/RHEL section in the installing guide.

sudo cpanm Bio::Perl This step will fail.

Configuring Test-Weaken-3.022000 ... OK
Building and testing Test-Weaken-3.022000 ... FAIL

which might be a good exercise (might be painful) to learn to how to resolve this type of issue.

yehzx commented 2 years ago

Blast and Prodigal worked well. But when it came to installing Prokka, the issue you mentioned was just the beginning of the dependency hell. I've tried to solve some of them, but problems still exist. The one I'm dealing with now has a quite ambiguous error log, not explicitly showing which part fails. If I still can't solve it, I'll tell you and hope that you can help me out. (or just use conda enviroment, don't deal with these dependency issues)

stevenhwu commented 2 years ago

Regarding Prokka and Perl dependency, what is the current error message/log? What dependency have you installed so far?

yehzx commented 2 years ago

Here is the packages I additionally installed

In order to use cpanm: sudo yum install perl-ExtUtils-MakeMaker One of the blast program need: sudo cpanm JSON::PP The first dependency issue when installing prokka: Can't locate Fatal.pm in @INC It needs sudo yum install perl-core The next one: sudo yum install perl-LWP-Protocol-https

I tried many other things (but it turned out that those were not necessary), and finally last problem should be Bio::DB::GenPept.

yehzx commented 2 years ago

Error log of Bio::DB::GenPept

Searching Bio::DB::GenPept on cpanmetadb ...
--> Working on Bio::DB::GenPept
Fetching http://www.cpan.org/authors/id/C/CJ/CJFIELDS/Bio-DB-NCBIHelper-1.7.7.tar.gz
-> OK
Unpacking Bio-DB-NCBIHelper-1.7.7.tar.gz
Entering Bio-DB-NCBIHelper-1.7.7
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 0 ... Yes (6.68)
Configuring Bio-DB-NCBIHelper-1.7.7
Running Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Bio::DB::NCBIHelper
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have Test::RequiresInternet 0 ... Yes (0.05)
Checking if you have Bio::WebAgent 0 ... Yes (1.7.8)
Checking if you have Bio::DB::Taxonomy 0 ... Yes (1.7.8)
Checking if you have Bio::DB::WebDBSeqI 0 ... Yes (1.7.8)
Checking if you have CGI 0 ... Yes (4.54)
Checking if you have IPC::Open3 0 ... Yes (1.12)
Checking if you have Bio::DB::Query::WebQuery v1.7.7 ... Yes (1.7.8)
Checking if you have Test::Most 0 ... Yes (0.37)
Checking if you have Getopt::Long 0 ... Yes (2.4)
Checking if you have base 0 ... Yes (2.18)
Checking if you have warnings 0 ... Yes (1.13)
Checking if you have Bio::Root::Root 0 ... Yes (1.7.8)
Checking if you have Test::Exception 0 ... Yes (0.43)
Checking if you have Bio::SeqIO::entrezgene 0 ... Yes (1.73)
Checking if you have ExtUtils::MakeMaker 0 ... Yes (6.68)
Checking if you have vars 0 ... Yes (1.02)
Checking if you have HTTP::Request::Common 0 ... Yes (6.37)
Checking if you have Bio::SeqIO 0 ... Yes (1.7.8)
Checking if you have Test::More 0 ... Yes (1.302191)
Checking if you have strict 0 ... Yes (1.07)
Checking if you have Bio::Taxon 0 ... Yes (1.7.8)
Checking if you have Bio::Root::IO 0 ... Yes (1.7.8)
Checking if you have URI::Escape 0 ... Yes (3.31)
Checking if you have File::Spec 0 ... Yes (3.40)
Checking if you have IO::Handle 0 ... Yes (1.33)
Checking if you have LWP::Protocol::https 0 ... Yes (6.04)
Checking if you have URI 0 ... Yes (1.60)
Checking if you have XML::Twig 0 ... Yes (3.52)
Checking if you have Cache::FileCache 0 ... Yes (undef)
Checking if you have constant 0 ... Yes (1.27)
Checking if you have LWP::UserAgent 0 ... Yes (6.67)
Building and testing Bio-DB-NCBIHelper-1.7.7
cp lib/Bio/DB/GenBank.pm blib/lib/Bio/DB/GenBank.pm
cp lib/Bio/DB/NCBIHelper.pm blib/lib/Bio/DB/NCBIHelper.pm
cp lib/Bio/DB/Taxonomy/entrez.pm blib/lib/Bio/DB/Taxonomy/entrez.pm
cp lib/Bio/DB/GenPept.pm blib/lib/Bio/DB/GenPept.pm
cp lib/Bio/DB/Query/GenBank.pm blib/lib/Bio/DB/Query/GenBank.pm
cp lib/Bio/DB/EntrezGene.pm blib/lib/Bio/DB/EntrezGene.pm
cp bin/bp_query_entrez_taxa blib/script/bp_query_entrez_taxa
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/bp_query_entrez_taxa
cp bin/bp_biofetch_genbank_proxy blib/script/bp_biofetch_genbank_proxy
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/bp_biofetch_genbank_proxy
cp bin/bp_download_query_genbank blib/script/bp_download_query_genbank
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/bp_download_query_genbank
Manifying blib/man1/bp_query_entrez_taxa.1
Manifying blib/man1/bp_biofetch_genbank_proxy.1
Manifying blib/man1/bp_download_query_genbank.1
Manifying blib/man3/Bio::DB::Taxonomy::entrez.3pm
Manifying blib/man3/Bio::DB::NCBIHelper.3pm
Manifying blib/man3/Bio::DB::GenBank.3pm
Manifying blib/man3/Bio::DB::Query::GenBank.3pm
Manifying blib/man3/Bio::DB::GenPept.3pm
Manifying blib/man3/Bio::DB::EntrezGene.3pm
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-compile.t ......... ok
t/author-mojibake.t .... skipped: these tests are for testing by the author
t/author-pod-syntax.t .. skipped: these tests are for testing by the author
t/EntrezGene.t ......... ok
t/GenBank.t ............ ok
t/GenPept.t ............ ok
t/Query-Genbank.t ...... ok
t/RefSeq.t ............. ok

#   Failed test at t/Taxonomy.t line 48.

#   Failed test 'Correct: No hit'
#   at t/Taxonomy.t line 181.
#          got: '2890685'
#     expected: 'No hit'
# Looks like you failed 2 tests of 69.
t/Taxonomy.t ...........
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/69 subtests

Test Summary Report
-------------------
t/Taxonomy.t         (Wstat: 512 (exited 2) Tests: 69 Failed: 2)
  Failed tests:  18, 68
  Non-zero exit status: 2
Files=9, Tests=175, 154 wallclock secs ( 0.06 usr  0.01 sys +  4.95 cusr  0.35 csys =  5.37 CPU)
Result: FAIL
Failed 1/9 test programs. 2/175 subtests failed.
make: *** [test_dynamic] Error 2

yehzx commented 2 years ago

So I'm not sure about the reason why the tests failed. Because It seems that it doesn't miss any other packages.

stevenhwu commented 2 years ago

This one is interesting, and you do encounter this in bioinformatics from time to time.

Essentially, the executed answer is "no hit" and the program matched to something.

Phygadeuon ovatus
Taxonomy ID: 2890685 (for references in articles please use NCBI:txid2890685)

This is due to the database is constantly updating, but the software package didn't keep up to date.

Solution: https://github.com/bioperl/Bio-DB-NCBIHelper/issues/8 There is a pull request to fix this already, but haven't beet merged.

So, currently, there is no easy my around it, the current solution is to force it to install.

yehzx commented 2 years ago

I did a force install last time, and in some steps the prokka failed, so I'm not sure if this time will work but will try it.

stevenhwu commented 2 years ago

If that's the error you send through line last time, since you went through all these crazy stuffs on installing all these dependencies now, you might be able to resolve that by installing another dependence.

yehzx commented 2 years ago

Prokka can run now

Something that I did:

force install Bio::DB::GenPept
sudo cpanm Bio::Perl again
issue when running prokka: Failed to load module Bio::SearchIO::hmmer3 solved: sudo cpanm Bio::SearchIO::hmmer

another issue: Could not run command: tbl2asn

# remove the one in prokka library and download it from ncbi
wget https://ftp.ncbi.nih.gov/toolbox/ncbi_tools/converters/by_program/tbl2asn/linux64.tbl2asn.gz
gunzip ./linux64.tbl2asn.gz
chmod +x linux64.tbl2asn
mv linux64.tbl2asn tbl2asn

Done!

stevenhwu commented 2 years ago

Well done! I know it's a painful process, but these things are like to happen frequently in bioinformatics.

Regarding prokka-uniprot_to_fasta_db from your line message, you need sudo cpanm https://sourceforge.net/projects/swissknife/files/swissknife/1.81/swissknife_1.81.tar.gz/download -n from a third party website.

yehzx commented 2 years ago

Actually I wrote my own version. But since we move to prodigal and blast, currently there's no need to use it.

stevenhwu commented 2 years ago

Ya, there are learning exercises, hopefully, it will pay off one day.

please make sure you create an AIM for that.

yehzx commented 2 years ago

I just created one and thank you for your help!

ComputationalAgronomy / biopathway-prediction