can't fetch files from ftp.geneontology.org

[cwzkevin]

Hi, I am having a problem when "make test" the program. It can't fetch files from the ftp.

Below is the detail log about the issue, I changed the code to DEBUG => 1 in the perl module where defining the $ftp connection.

Please advise, thanks!

$ hmmer2go fetchmap -o pfam2go
Net::FTP>>> Net::FTP(3.09)
Net::FTP>>>   Exporter(5.72)
Net::FTP>>>   Net::Cmd(3.09)
Net::FTP>>>   IO::Socket::SSL(2.033)
Net::FTP>>>     IO::Socket::IP(0.37)
Net::FTP>>>       IO::Socket(1.38)
Net::FTP>>>         IO::Handle(1.35)
Net::FTP=GLOB(0x186ba5c8)<<< 220 (vsFTPd 2.2.2)
Net::FTP=GLOB(0x186ba5c8)>>> USER anonymous
Net::FTP=GLOB(0x186ba5c8)<<< 331 Please specify the password.
Net::FTP=GLOB(0x186ba5c8)>>> PASS ....
Net::FTP=GLOB(0x186ba5c8)<<< 230 Login successful.
Net::FTP=GLOB(0x186ba5c8)>>> CWD /pub/go/external2go
Net::FTP=GLOB(0x186ba5c8)<<< 250 Directory successfully changed.
Net::FTP=GLOB(0x186ba5c8)>>> FEAT
Net::FTP=GLOB(0x186ba5c8)<<< 211-Features:
Net::FTP=GLOB(0x186ba5c8)<<<  EPRT
Net::FTP=GLOB(0x186ba5c8)<<<  EPSV
Net::FTP=GLOB(0x186ba5c8)<<<  MDTM
Net::FTP=GLOB(0x186ba5c8)<<<  PASV
Net::FTP=GLOB(0x186ba5c8)<<<  REST STREAM
Net::FTP=GLOB(0x186ba5c8)<<<  SIZE
Net::FTP=GLOB(0x186ba5c8)<<<  TVFS
Net::FTP=GLOB(0x186ba5c8)<<<  UTF8
Net::FTP=GLOB(0x186ba5c8)<<< 211 End
Net::FTP=GLOB(0x186ba5c8)>>> SIZE pfam2go
Net::FTP=GLOB(0x186ba5c8)<<< 213 715160
Net::FTP=GLOB(0x186ba5c8)>>> PASV
Net::FTP=GLOB(0x186ba5c8)<<< 227 Entering Passive Mode (171,67,205,83,47,133).
Net::FTP=GLOB(0x186ba5c8)>>> RETR pfam2go
Net::FTP=GLOB(0x186ba5c8)<<< 425 Failed to establish connection.
get failed Failed to establish connection.
 will retry. at /home/bma/perl5/lib/perl5/HMMER2GO/Command/fetchmap.pm line 80.
Use of uninitialized value $lsize in numeric eq (==) at /home/bma/perl5/lib/perl5/HMMER2GO/Command/fetchmap.pm line 83.
Use of uninitialized value $lsize in concatenation (.) or string at /home/bma/perl5/lib/perl5/HMMER2GO/Command/fetchmap.pm line 83.
Failed to fetch complete file: pfam2go (local size: , remote size: 715160), will retry. at /home/bma/perl5/lib/perl5/HMMER2GO/Command/fetchmap.pm line 83.
Use of uninitialized value $lsize in numeric eq (==) at /home/bma/perl5/lib/perl5/HMMER2GO/Command/fetchmap.pm line 86.

[sestaton]

Hi,

I think this may have been a temporary network issue or perhaps there was an interruption in the service. Could you try it again and see if it works now? I am not able to reproduce this issue.

Thanks, Evan

[cwzkevin]

Hi Evan, I just tried again and still the same. Will it be something related to the perl and ftp? I used wget as below and the file was downloaded. If that is the case (the network/system at my end doesn't like perl and ftp), is there a way to bypass it? For example, can I download all the files at once and disconnect from internet?

$ wget ftp://anonymous@ftp.geneontology.org/go/external2go/pfam2go
--2016-11-15 05:51:46--  ftp://anonymous@ftp.geneontology.org/go/external2go/pfam2go
           => `pfam2go'
Resolving ftp.geneontology.org... 171.67.205.83
Connecting to ftp.geneontology.org|171.67.205.83|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /go/external2go ... done.
==> SIZE pfam2go ... 715160
==> PASV ... done.    ==> RETR pfam2go ... done.
Length: 715160 (698K)

100%[===================================================================================================================>] 715,160      212K/s   in 3.3s

2016-11-15 05:51:51 (212 KB/s) - `pfam2go' saved [715160]

Thanks, Kevin

[sestaton]

Hi Kevin, If you have the 'pfam2go' mapping file locally then you don't need to use the network for the mapping step. In that case, just supply that file to the 'mapterms' command with the '-p' or '--pfam2go' flag.

I would like to figure this out in case I can add a workaround for future use. It would help to know the output of these commands:

perl -v
perl -e 'print Net::FTP::VERSION'

Also, it would be good to know the operating system and version. Though, since this module is a core module that gets tested routinely against hundreds of operating systems I suspect this is a local firewall or network issue.

Thanks, Evan

[cwzkevin]

Evan,

$ perl -v

This is perl 5, version 22, subversion 2 (v5.22.2) built for x86_64-linux-thread-multi

perl -e 'print Net::FTP::VERSION' returns nothing

 $ perl -e 'print Net::FTP::VERSION'
 $ perl -e 'use Net::FTP 9e9'
Net::FTP version 9000000000 required--this is only version 3.09 at /usr/local/packages/perl-5.22.2/lib/5.22.2/Exporter/Heavy.pm line 125.
BEGIN failed--compilation aborted at -e line 1.

For the OS,

$ uname -mrs
Linux 2.6.18-371.1.2.el5 x86_64
$ cat /etc/*-release
Red Hat Enterprise Linux Server release 5.9 (Tikanga)

Thank you very much! Kevin

[sestaton]

Hi Kevin,

After some reading up on the issue it appears that this has to do with how older RHEL systems handled FTP connections. I just tried this on a RHEL 5.11 server and it worked as expected. Though, I have added a method that should fix this issue for you, which is simply using what we know to work. If you could try it out on this branch that would be great.

https://github.com/sestaton/HMMER2GO/tree/topic/add_rhel_ftpfix

If that works I'll make a new release with this change. This new method won't be touched unless the normal approach fails. I have not tested Perl 5.22 yet, but I'm going to do that as well.

Thanks, Evan

[cwzkevin]

Here is the log, seems some downloaded on the 2nd try (I think that is your backup method introduced), but still having some download error which looks similar?

$ make test
PERL_DL_NONLAZY=1 "/usr/local/packages/perl-5.22.2/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ........ 1/3 # Testing HMMER2GO 0.17.3, Perl 5.022002, /usr/local/packages/perl-5.22.2/bin/perl
t/00-load.t ........ ok
t/01-fetchmap.t .... 1/2 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 86.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry. at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 97.
t/01-fetchmap.t .... ok
t/02-getorf.t ...... ok
t/03-run.t ......... ok
t/04-mapterms.t .... 3/9 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 174.
Use of uninitialized value $lsize in numeric eq (==) at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 179.
Use of uninitialized value $lsize in concatenation (.) or string at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 179.
Failed to fetch complete file: pfam2go (local size: , remote size: 715160), will retry. at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 179.
Use of uninitialized value $lsize in numeric eq (==) at /local/projects-t2/M8910/software/HMMER2GO/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 182.

ERROR: Could not open file:
t/04-mapterms.t .... 8/9
#   Failed test 'Expected output from hmmer2go mapterms without mapping file'
#   at t/04-mapterms.t line 77.
Can't open 't/test_data/t_long_Pfam-A_mapped_goterms.tsv' for reading: 'No such file or directory' at t/04-mapterms.t line 79
# Looks like your test exited with 2 just after 8.
t/04-mapterms.t .... Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/9 subtests
t/05-map2gaf.t ..... 1/2 get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 28.
# Looks like your test exited with 255 just after 1.
t/05-map2gaf.t ..... Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests
t/06-pfamsearch.t .. ok
t/07-customdb.t .... ok

Test Summary Report
-------------------
t/04-mapterms.t  (Wstat: 512 Tests: 8 Failed: 1)
  Failed test:  8
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 9 tests but ran 8.
t/05-map2gaf.t   (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=8, Tests=52, 241 wallclock secs ( 0.09 usr  0.03 sys + 18.97 cusr  5.95 csys = 25.04 CPU)
Result: FAIL
Failed 2/8 test programs. 1/52 subtests failed.
make: *** [test_dynamic] Error 255

Thanks, Kevin

[sestaton]

Thanks for the patience! I overlooked something so that was all me. The branch I posted above has some updated methods that should work if you want to try it again. If this works I'll polish it up tomorrow and make a release.

[cwzkevin]

Thanks, Evan.

Still one failed. Is it the same network issue or not?

 $ make test
PERL_DL_NONLAZY=1 "/usr/local/packages/perl-5.22.2/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ........ 1/3 # Testing HMMER2GO 0.17.3, Perl 5.022002, /usr/local/packages/perl-5.22.2/bin/perl
t/00-load.t ........ ok
t/01-fetchmap.t .... 1/2 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 86.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
t/01-fetchmap.t .... ok
t/02-getorf.t ...... ok
t/03-run.t ......... ok
t/04-mapterms.t .... 3/9 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 183.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
t/04-mapterms.t .... ok
t/05-map2gaf.t ..... 1/2 get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 28.
# Looks like your test exited with 255 just after 1.
t/05-map2gaf.t ..... Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests
t/06-pfamsearch.t .. ok
t/07-customdb.t .... ok

Test Summary Report
-------------------
t/05-map2gaf.t   (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=8, Tests=53, 240 wallclock secs ( 0.08 usr  0.03 sys + 19.56 cusr  5.98 csys = 25.65 CPU)
Result: FAIL
Failed 1/8 test programs. 0/53 subtests failed.
make: *** [test_dynamic] Error 255

[sestaton]

Hi Kevin,

Thanks for the patience and the report. This was one was me, but I think I got it sorted. Hopefully this will solve it but please don't hesitate if there are other issues.

Thanks, Evan

[cwzkevin]

Hi Evan, I just tried but it seems the same error. Please advise. Thank you very much!

$ make test
PERL_DL_NONLAZY=1 "/usr/local/packages/perl-5.22.2/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib
t/00-load.t ........ 1/3 # Testing HMMER2GO 0.17.3, Perl 5.022002, /usr/local/packages/perl-5.22.2/bin/perl
t/00-load.t ........ ok
t/01-fetchmap.t .... 1/2 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 86.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
t/01-fetchmap.t .... ok
t/02-getorf.t ...... ok
t/03-run.t ......... ok
t/04-mapterms.t .... 3/9 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 183.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
t/04-mapterms.t .... ok
t/05-map2gaf.t ..... 1/2 get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 28.
# Looks like your test exited with 255 just after 1.
t/05-map2gaf.t ..... Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests
t/06-pfamsearch.t .. ok
t/07-customdb.t .... ok

Test Summary Report
-------------------
t/05-map2gaf.t   (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=8, Tests=53, 267 wallclock secs ( 0.07 usr  0.02 sys + 18.61 cusr  5.82 csys = 24.52 CPU)
Result: FAIL
Failed 1/8 test programs. 0/53 subtests failed.
make: *** [test_dynamic] Error 255

[sestaton]

Thanks, there was an incorrect URL that I added but I think it all should be fixed on that same branch if you want to give it another go.

[cwzkevin]

Evan, It is going backward. More fails now.

$ make test
PERL_DL_NONLAZY=1 "/usr/local/packages/perl-5.22.2/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ........ 1/3 # Testing HMMER2GO 0.17.3, Perl 5.022002, /usr/local/packages/perl-5.22.2/bin/perl
t/00-load.t ........ ok
t/01-fetchmap.t .... 1/2 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 80.
Use of uninitialized value $lsize in numeric eq (==) at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 85.
Use of uninitialized value $lsize in concatenation (.) or string at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 85.
Failed to fetch complete file: pfam2go (local size: , remote size: 715160), will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 85.
Use of uninitialized value $lsize in numeric eq (==) at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 88.
t/01-fetchmap.t .... 2/2
#   Failed test 'Successfully fetched pfam2go mappings'
#   at t/01-fetchmap.t line 27.
# Looks like you failed 1 test of 2.
t/01-fetchmap.t .... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
t/02-getorf.t ...... ok
t/03-run.t ......... ok
t/04-mapterms.t .... 1/9 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 174.
Use of uninitialized value $lsize in numeric eq (==) at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 179.
Use of uninitialized value $lsize in concatenation (.) or string at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 179.
Failed to fetch complete file: pfam2go (local size: , remote size: 715160), will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 179.
Use of uninitialized value $lsize in numeric eq (==) at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 182.

ERROR: Could not open file:
t/04-mapterms.t .... 2/9
#   Failed test 'Expected output from hmmer2go mapterms without mapping'
#   at t/04-mapterms.t line 30.
Can't open 't/test_data/t_long_Pfam-A_mapped_goterms.tsv' for reading: 'No such file or directory' at t/04-mapterms.t line 33
# Looks like your test exited with 2 just after 2.
t/04-mapterms.t .... Dubious, test returned 2 (wstat 512, 0x200)
Failed 8/9 subtests
t/05-map2gaf.t ..... 1/2 get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 28.
# Looks like your test exited with 255 just after 1.
t/05-map2gaf.t ..... Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests
t/06-pfamsearch.t .. ok
t/07-customdb.t .... ok

Test Summary Report
-------------------
t/01-fetchmap.t  (Wstat: 256 Tests: 2 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
t/04-mapterms.t  (Wstat: 512 Tests: 2 Failed: 1)
  Failed test:  2
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 9 tests but ran 2.
t/05-map2gaf.t   (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=8, Tests=46, 226 wallclock secs ( 0.06 usr  0.03 sys + 16.70 cusr  5.48 csys = 22.27 CPU)
Result: FAIL
Failed 3/8 test programs. 2/46 subtests failed.
make: *** [test_dynamic] Error 255

[sestaton]

Hi, it's not clear if the latest code is being used based on those line numbers. I've tested on an older RHEL machine and disabled all the other methods and it's passing for me as expected. So, there is something curious going on and I want to make sure we are looking at the same code. I've changed the version on this branch, and if you could try again that would tell me we are looking at the same thing. That way we're not going in different directions.

[cwzkevin]

Here is what I did:

$ rm -rf HMMER2GO/
$ git clone https://github.com/sestaton/HMMER2GO.git
$ git checkout topic/add_rhel_ftpfix

Here is the log,

$ make test
PERL_DL_NONLAZY=1 "/usr/local/packages/perl-5.22.2/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib
t/00-load.t ........ 1/3 # Testing HMMER2GO 0.17.4, Perl 5.022002, /usr/local/packages/perl-5.22.2/bin/perl
t/00-load.t ........ ok
t/01-fetchmap.t .... 1/2 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 86.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
t/01-fetchmap.t .... ok
t/02-getorf.t ...... ok
t/03-run.t ......... ok
t/04-mapterms.t .... 3/9 get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 183.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
t/04-mapterms.t .... ok
t/05-map2gaf.t ..... 1/2 get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 28.
# Looks like your test exited with 255 just after 1.
t/05-map2gaf.t ..... Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests
t/06-pfamsearch.t .. ok
t/07-customdb.t .... ok

Test Summary Report
-------------------
t/05-map2gaf.t   (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=8, Tests=53, 249 wallclock secs ( 0.09 usr  0.03 sys + 19.18 cusr  6.11 csys = 25.41 CPU)
Result: FAIL
Failed 1/8 test programs. 0/53 subtests failed.
make: *** [test_dynamic] Error 255

[sestaton]

Hi Kevin,

Sorry for the hassles. It looks like there is only one small issue, but it's hard to test without access to that specific OS version. If you could, comment out line 29 in "t/05-map2gaf.t" and uncomment line 28 in the same file. Then, run these commands:

prove -bv t/0[1-4]*t
prove -bv t/05*t

That should indicate why the command failed. This issue is particularly difficult because it is an older non-free OS, so it's not possible to create a cloud instance for testing, which I usually do. But, it appears to be almost resolved.

Thanks, Evan

[cwzkevin]

Hi Evan,

I much appreciated your help! First, it complains I don't have IPC::System::Simple. I fixed that. Then, when it looks like

 28 #my @result1 = capture([0..5], "$hmmer2go map2gaf -i $infile -o $outfile -s $species");
 29 system([0..5], "$hmmer2go map2gaf -i $infile -o $outfile -s $species");

the t/0[1-4]*t returns,

$ prove -bv t/0[1-4]*t
t/01-fetchmap.t ..
1..2
ok 1 - Correct number of options for hmmer2go fetchmap
get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/fetchmap.pm line 86.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
ok 2 - Successfully fetched pfam2go mappings
ok
t/02-getorf.t ....
1..13
ok 1 - Correct number of options for hmmer2go getorf
ok 2 - Successfully ran getorf and produced the expected output
ok 3 - Expected number of ORFs found for test data when only keeping longest ORFs
ok 4 - Successfully ran getorf and produced the expected output
ok 5 - Expected number of ORFs found for test data when keeping all ORFs
ok 6 - Successfully ran getorf and produced the expected output
ok 7 - Expected number of ORFs found for test data when only keeping longest ORFs
ok 8 - Successfully ran getorf and produced the expected output
ok 9 - Expected number of ORFs found for test data when keeping all ORFs
ok 10 - Successfully ran getorf and produced the expected output
ok 11 - Expected number of ORFs found for test data when only keeping longest ORFs
ok 12 - Successfully ran getorf and produced the expected output
ok 13 - Expected number of ORFs found for test data when keeping all ORFs
ok
t/03-run.t .......
1..10
ok 1 - Correct number of options for hmmer2go run
ok 2 # skip skip lengthy tests
ok 3 # skip skip lengthy tests
ok 4 # skip skip lengthy tests
ok 5 # skip skip lengthy tests
ok 6 # skip skip lengthy tests
ok 7 # skip skip lengthy tests
ok 8 # skip skip lengthy tests
ok 9 # skip skip lengthy tests
ok 10 # skip skip lengthy tests
ok
t/04-mapterms.t ..
1..9
ok 1 - Correct number of options for hmmer2go mapterms
ok 2 - Expected output from hmmer2go mapterms without mapping
ok 3 - Expected output from hmmer2go mapterms with mapping
ok 4 - Expected GO term mapping file produced with hmmer2go mapterms
ok 5 - Same genes and GO term numbers returned with either mapping or not mapping option
ok 6 - Correct number of GO terms mapped
ok 7 - Correct number of GO terms mapped
get failed Failed to establish connection.
 will retry. at /local/projects-t2/M8910/software/HMMER2GO/blib/lib/HMMER2GO/Command/mapterms.pm line 183.
Failed to fetch complete file: pfam2go (local size: 0, remote size: 715160), will retry.
ok 8 - Expected output from hmmer2go mapterms without mapping file
ok 9 - Same genes and GO term numbers returned with or without user-supplied mapping file
ok
All tests successful.
Files=4, Tests=34, 174 wallclock secs ( 0.09 usr  0.03 sys + 18.71 cusr  6.58 csys = 25.41 CPU)
Result: PASS

the t/05*t returns,

 $ prove -bv t/05*t
t/05-map2gaf.t ..
1..2
ok 1 - Correct number of options for hmmer2go map2gaf
get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 29.
# Looks like your test exited with 255 just after 1.
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests

Test Summary Report
-------------------
t/05-map2gaf.t (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=1, Tests=1, 78 wallclock secs ( 0.05 usr  0.01 sys +  1.32 cusr  0.24 csys =  1.62 CPU)
Result: FAIL

When it looks like

 28 my @result1 = capture([0..5], "$hmmer2go map2gaf -i $infile -o $outfile -s $species");
 29 #system([0..5], "$hmmer2go map2gaf -i $infile -o $outfile -s $species");

the t/05*t returns,

$ prove -bv t/05*t
t/05-map2gaf.t ..
1..2
ok 1 - Correct number of options for hmmer2go map2gaf
get failed Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s Helianthus annuus" unexpectedly returned exit value 255 at t/05-map2gaf.t line 28.
# Looks like your test exited with 255 just after 1.
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests

Test Summary Report
-------------------
t/05-map2gaf.t (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=1, Tests=1, 68 wallclock secs ( 0.05 usr  0.02 sys +  1.34 cusr  0.20 csys =  1.61 CPU)
Result: FAIL

[sestaton]

Hi Kevin,

Can you show me what steps you used to install the dependencies? That was the first step in the install process, which would install those modules. Assuming that's not the issue, the only other thing I can do is see the command output itself. So, after running this command (I don't need to see the output this time):

prove -bv t/0[1-4]*t

It would help to see the output of:

ls -l t/test_data

And then, try to run the command directly:

perl -Mblib blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus'

Thanks, Evan

[cwzkevin]

Hi Evan,

I didn't need to install any dependencies. They are all in my path for binaries. For Perl Module, I usually install them using cpanm. Here is the outputs:

$ ls -l t/test_data
total 152
-rw-r--r-- 1 cwz usr   191 Nov 18 09:33 t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv
-rw-r--r-- 1 cwz usr   4203 Nov 17 16:10 t_orfs_long_Pfam-A.tblout
-rw-r--r-- 1 cwz usr   11668 Nov 17 16:10 t_seqs_nt.fas
-rw-r--r-- 1 cwz usr   3549 Nov 17 16:10 t_seqs_nt_bz.fas.bz2
-rw-r--r-- 1 cwz usr   4112 Nov 17 16:10 t_seqs_nt_gz.fas.gz

$ perl -Mblib blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus'
get failed Failed to establish connection.

[sestaton]

It looks exactly as I would expect. Can you do:

$ ls -l t/test_data

to show the final results? I'm not seeing why the test failed exactly. That warning doesn't mean the process failed. Thanks.

[cwzkevin]

Nothing changed...

$ ls -l t/test_data
total 152
-rw-r--r-- 1 cwz usr 191 Nov 18 09:33 t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv
-rw-r--r-- 1 cwz usr  4203 Nov 17 16:10 t_orfs_long_Pfam-A.tblout
-rw-r--r-- 1 cwz usr 11668 Nov 17 16:10 t_seqs_nt.fas
-rw-r--r-- 1 cwz usr  3549 Nov 17 16:10 t_seqs_nt_bz.fas.bz2
-rw-r--r-- 1 cwz usr  4112 Nov 17 16:10 t_seqs_nt_gz.fas.gz

[sestaton]

If you do:

ls -l

Can you see a file called "GO.terms_alt_ids" in the working directory?

[cwzkevin]

No

[sestaton]

Okay, I know this is frustrating, sorry. Let's try one more time, if this doesn't work I'll just run the analysis if you like or create a server. I added better debugging messages and more error handling, and some of the intermediate files should stay around so I can see what is happening.

After the update from that branch, first just try make test and see what it outputs. If the same test fails again, let's try:

$ prove -bv t/0[1-4]*t

I don't need to see that because it should work, but then:

$ ls -l t/test_data

And,

$ perl -Mblib blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus'

Followed by...

$ ls -l

If that all works, try the test directly:

$ prove -bv t/05*t

That will tell me something, thanks.

[cwzkevin]

Hi Evan,

$ ls -l t/test_data
total 152
-rw-r--r-- 1 cwz usr   191 Nov 18 16:50 t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv
-rw-r--r-- 1 cwz usr   4203 Nov 18 16:42 t_orfs_long_Pfam-A.tblout
-rw-r--r-- 1 cwz usr   11668 Nov 18 16:42 t_seqs_nt.fas
-rw-r--r-- 1 cwz usr   3549 Nov 18 16:42 t_seqs_nt_bz.fas.bz2
-rw-r--r-- 1 cwz usr   4112 Nov 18 16:42 t_seqs_nt_gz.fas.gz

And, then

$ perl -Mblib blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus'
FTP get failed with the message: Failed to establish connection.

And, then

$ ls -l
total 373
-rw-r--r-- 1 cwz usr  4180 Nov 18 16:42 Changes
-rw-r--r-- 1 cwz usr  1086 Nov 18 16:42 LICENSE
-rw-r--r-- 1 cwz usr  467 Nov 18 16:42 MANIFEST
-rw-r--r-- 1 cwz usr  1154 Nov 18 16:43 MYMETA.json
-rw-r--r-- 1 cwz usr  675 Nov 18 16:43 MYMETA.yml
-rw-r--r-- 1 cwz usr  32062 Nov 18 16:43 Makefile
-rw-r--r-- 1 cwz usr  1224 Nov 18 16:42 Makefile.PL
-rw-r--r-- 1 cwz usr  3610 Nov 18 16:42 README.md
drwxr-xr-x 2 cwz usr  26 Nov 18 16:43 bin
drwxr-xr-x 7 cwz usr  108 Nov 18 16:43 blib
drwxr-xr-x 3 cwz usr  20 Nov 18 16:42 build
-rw-r--r-- 1 cwz usr  369 Nov 18 16:42 cpanfile
drwxr-xr-x 3 cwz usr  55 Nov 18 16:43 lib
-rw-r--r-- 1 cwz usr  444 Nov 18 16:43 pb.out
-rw-r--r-- 1 cwz usr  0 Nov 18 16:43 pm_to_blib
drwxr-xr-x 3 cwz usr  265 Nov 18 16:48 t
drwxr-xr-x 2 cwz usr  55 Nov 18 16:42 xt

And then,

$ prove -bv t/05*t
t/05-map2gaf.t ..
1..2
ok 1 - Correct number of options for hmmer2go map2gaf
blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus'
FTP get failed with the message: Failed to establish connection.
"blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus'" unexpectedly returned exit value 255 at t/05-map2gaf.t line 31.
# Looks like your test exited with 255 just after 1.
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/2 subtests

Test Summary Report
-------------------
t/05-map2gaf.t (Wstat: 65280 Tests: 1 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 2 tests but ran 1.
Files=1, Tests=1, 63 wallclock secs ( 0.05 usr  0.02 sys +  1.25 cusr  0.23 csys =  1.55 CPU)
Result: FAIL

Notice that, all the above command is running under the condition as below for t/05-map2gaf.t

28 #my @result1 = capture([0..5], "$hmmer2go map2gaf -i $infile -o $outfile -s $species");
 29 my $cmd = "$hmmer2go map2gaf -i $infile -o $outfile -s '$species'";
 30 say STDERR $cmd;
 31 system([0..5], "$hmmer2go map2gaf -i $infile -o $outfile -s '$species'");

Thank you so much! Kevin

[sestaton]

Hi Kevin,

Sorry for the delay, I had to switch my focus to other matters. I see what the problem is but it is hard to understand why that one command is failing. It could be a network issue, so let's try this command:

wget -q -O GO.terms_alt_ids ftp://ftp.geneontology.org//pub/go/doc/GO.terms_alt_ids

If that passes, please try the latest update on that branch if you can. I have written some external commands to not use the shell for security reasons but I think that was causing problems with the older RHEL version. Maybe, if that's not it I can only think of one more thing it could possibly be (a Perl library being used). I appreciate the time spent, hopefully it pays off.

Thanks, Evan

[cwzkevin]

Hi Evan,

The wget is fine. I have GO.terms_alt_ids in HMMER2GO/ right now.

I have the latest code of branch topic/add_rhel_ftpfix, and GO.terms_alt_ids in HMMER2GO/. But, the make test and/or perl -Mblib blib/bin/hmmer2go map2gaf -i t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.tsv -o t/test_data/t_long_Pfam-A_mapped_goterms_GOterm_mapping.gaf -s 'Helianthus annuus' still failed.

My thought is that your code isn't using your latest map2gaf.pm. B/c the error message is still FTP get failed with the message: Failed to establish connection. And I see that in your latest map2gaf.pm, the error message should be ERROR: 'wget' failed. Cannot fetch map file. Please report this error.

Thanks, Kevin

[sestaton]

We are using a new GO database in the latest version (v0.18.0), and a different method for retrieving the file. Please try the latest release if you can let me know if this solves the issue. I'm going to close this topic as it is not longer something that needs to be explored given the updates.

Please let me know if you have further issues and we can reopen this and take a look.