p5pRT
commented
20 years ago From rwilkinson@star.net.uk
This is a bug report for perl from rwilkinson@star.net.uk\, generated with the help of perlbug 1.34 running under perl v5.8.0.
Hello
I am trying to retrieve a directory listing from an FTP server using LWP. I think that the site may well be sending me non-ascii characters? However\, Net::FTP can handle it\, and from reading LWP::Protocol::ftp.pm\, there is an implication that Net::FTP is used by LWP?
LWP does act as I expect if I use it to get a directory listing of the parent directory of the one I need to list. I could bypass the problem by using Net::FTP for this one ftp server\, however\, since the rest of my code uses LWP\, I'd sooner get to the bottom of the problem.
I have listed the programs here\, followed by their output collected via script. I have tried to reduce the programs to their essential statements. I have changed the ftp address\, username and password because I do not know if I have permission to distribute them. However\, the outputs are those garnered before this change.
The code is ordered:
netftp.pl netftp.op lwpbad.pl lwpbad.op lwpgood.pl lwpgood.op
I suspect that the fact that 'content-type' => 'application/octet-stream' is set in the HTTP::Header of the lwpbad output is wrong. It is set to 'content-type' => 'text/ftp-dir-listing' in the lwpgood output. I think that this is probably due to the fact that LWP::MediaTypes is interpreting the non-ascii character?
Is it possible for me to override the assignation given. I do set the content_type of my request object to be "text/ftp-dir-listing".
Bob
#!/usr/bin/perl -w
use strict; use Net::FTP; use Data::Dumper;
my $ftp = Net::FTP->new("partner.abcde.ru"\, Debug => 0) or die "Cannot connect to partner.abcde.ru: $@";
$ftp->login("xyz"\,'123') or die "Cannot login "\, $ftp->message;
$ftp->cwd("/bases") or die "Cannot change working directory "\, $ftp->message;
my $dir = $ftp->dir or die "dir failed "\, $ftp->message; warn "DIR is "\,Dumper($dir);
$ftp->quit;
Script started on Wed Jun 2 12:25:13 2004 ]0;bob@snoogen:\~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ perl netftp.pl DIR is $VAR1 = [ 'total 3870'\, '-rw-r--r-- 1 2133 4000 15297 May 20 12:39 advware.avc'\, '-rw-r--r-- 1 2133 4000 962750 May 20 12:39 avp0404.avc'\, '-rw-r--r-- 1 2133 4000 286326 May 20 12:39 backdoor.avc'\, '-rw-r--r-- 1 2133 4000 53719 May 20 12:39 ca.avc'\, '-rw-r--r-- 1 2133 4000 32075 Jun 2 14:54 daily.avc'\, '-rw-r--r-- 1 2133 4000 1843 May 20 12:39 eicar.avc'\, '-rw-r--r-- 1 2133 4000 19046 May 20 12:39 extr-cab.avc'\, '-rw-r--r-- 1 2133 4000 211150 May 20 12:39 extract.avc'\, '-rw-r--r-- 1 2133 4000 8401 May 20 12:39 kernel.avc'\, '-rw-r--r-- 1 2133 4000 5275 May 20 12:39 krndos.avc'\, '-rw-r--r-- 1 2133 4000 23515 May 20 12:39 krnengn.avc'\, '-rw-r--r-- 1 2133 4000 64482 May 20 12:39 krnexe.avc'\, '-rw-r--r-- 1 2133 4000 37999 May 20 12:39 krnjava.avc'\, '-rw-r--r-- 1 2133 4000 76929 May 20 12:39 krnmacro.avc'\, '-rw-r--r-- 1 2133 4000 55548 May 20 12:39 krnunp.avc'\, '-rw-r--r-- 1 2133 4000 141049 May 20 12:39 macro.avc'\, '-rw-r--r-- 1 2133 4000 14114 May 20 12:39 mail.avc'\, '-rw-r--r-- 1 2133 4000 115922 May 20 12:39 malware.avc'\, '-rw-r--r-- 1 2133 4000 375390 May 20 12:39 newexe.avc'\, '-rw-r--r-- 1 2133 4000 14920 May 20 12:39 newexeg.avc'\, '-rw-r--r-- 1 2133 4000 3162 May 20 12:39 ocr.avc'\, '-rw-r--r-- 1 2133 4000 27541 May 20 12:39 pornware.avc'\, '-rw-r--r-- 1 2133 4000 28492 May 20 12:39 riskware.avc'\, '-rw-r--r-- 1 2133 4000 99749 May 20 12:39 script.avc'\, '-rw-r--r-- 1 2133 4000 5366 May 20 12:39 smart.avc'\, '-rw-r--r-- 1 2133 4000 356774 May 20 12:39 trojan.avc'\, '-rw-r--r-- 1 2133 4000 569565 May 20 12:39 unpack.avc'\, '-rw-r--r-- 1 2133 4000 15450 May 20 12:39 up040402.avc'\, '-rw-r--r-- 1 2133 4000 18563 May 20 12:39 up040409.avc'\, '-rw-r--r-- 1 2133 4000 23479 May 20 12:39 up040416.avc'\, '-rw-r--r-- 1 2133 4000 40902 May 20 12:39 up040423.avc'\, '-rw-r--r-- 1 2133 4000 27378 May 20 12:39 up040430.avc'\, '-rw-r--r-- 1 2133 4000 19431 May 20 12:39 up040507.avc'\, '-rw-r--r-- 1 2133 4000 35446 May 20 12:39 up040514.avc'\, '-rw-r--r-- 1 2133 4000 23532 May 20 12:39 x-files.avc'\, '-rw-r--r-- 1 2133 4000 2040 Jun 2 14:54 xdaily.avc' ]; ]0;bob@snoogen:\~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ Script done on Wed Jun 2 12:25:22 2004
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
my $url = "ftp://partner.abcde.ru/bases"; my $user = "xyz"; my $pass = "123"; my $content_type = "text/ftp-dir-listing";
my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new('GET'\, $url); $req->authorization_basic($user\, $pass); $req->content_type($content_type); my $res = $ua->request($req); use Data::Dumper; warn "RES is "\,Dumper($res);
Script started on Wed Jun 2 12:34:13 2004 ]0;bob@snoogen:\~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ perl lwpbad.pl RES is $VAR1 = bless( { '_content' => '\, . * .. \, servers.lst \, advware.avc \, avp0404.avc \, backdoor.avc ³+À \, ca.avc Ç \, daily.avc ä \, eicar.avc ä \, extr-cab.avc ³+À \, extract.avc !\, kernel.avc ä.\, krndos.avc ä/\, krnengn.avc 0\, krnexe.avc ä1\, krnjava.avc 2\, krnmacro.avc ³+À3\, krnunp.avc ä4\, macro.avc ä5\, mail.avc ä6\, malware.avc 7\, newexe.avc ä8\, newexeg.avc 9\, ocr.avc :\, pornware.avc ³+À\<\, script.avc ä;\, riskware.avc ³+À=\, smart.avc ä>\, trojan.avc ä?\, unpack.avc ä@\, up040402.avc ³+ÀA\, up040409.avc ³+ÀB\, up040416.avc ³+ÀC\, up040423.avc ³+ÀD\, up040430.avc ³+ÀE\, up040507.avc ³+ÀF\, up040514.avc ³+ÀG\, x-files.avc H\, ð xdaily.avc ä '\, '_rc' => 200\, '_headers' => bless( { 'client-date' => 'Wed\, 02 Jun 2004 11:34:20 GMT'\, 'content-type' => 'application/octet-stream'\, 'server' => ' ###############################################################'\, 'client-request-num' => 1 }\, 'HTTP::Headers' )\, '_msg' => 'OK'\, '_request' => bless( { '_content' => ''\, '_uri' => bless( do{\(my $o = 'ftp://partner.abcde.ru/bases')}\, 'URI::ftp' )\, '_headers' => bless( { 'user-agent' => 'libwww-perl/5.65'\, 'content-type' => 'text/ftp-dir-listing'\, 'authorization' => 'Basic S0FWREI6UG5RR0c4ZThXM2lKWlVTeQ==' }\, 'HTTP::Headers' )\, '_method' => 'GET' }\, 'HTTP::Request' ) }\, 'HTTP::Response' ); ]0;bob@snoogen:\~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ Script done on Wed Jun 2 12:34:29 2004
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
my $url = "ftp://partner.abcde.ru/"; my $user = "xyz"; my $pass = "123"; my $content_type = "text/ftp-dir-listing";
my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new('GET'\, $url); $req->authorization_basic($user\, $pass); $req->content_type($content_type); my $res = $ua->request($req); use Data::Dumper; warn "RES is "\,Dumper($res);
Script started on Wed Jun 2 12:33:32 2004 ]0;bob@snoogen:\~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ perl lwpgood.pl RES is $VAR1 = bless( { '_content' => 'total 2 drwxr-xr-x 2 2133 4000 1024 May 20 12:39 bases '\, '_rc' => 200\, '_headers' => bless( { 'client-date' => 'Wed\, 02 Jun 2004 11:33:40 GMT'\, 'content-type' => 'text/ftp-dir-listing'\, 'content-length' => 58\, 'server' => ' ###############################################################'\, 'client-request-num' => 1 }\, 'HTTP::Headers' )\, '_msg' => 'OK'\, '_request' => bless( { '_content' => ''\, '_uri' => bless( do{\(my $o = 'ftp://partner.abcde.ru/')}\, 'URI::ftp' )\, '_headers' => bless( { 'user-agent' => 'libwww-perl/5.65'\, 'content-type' => 'text/ftp-dir-listing'\, 'authorization' => 'Basic S0FWREI6UG5RR0c4ZThXM2lKWlVTeQ==' }\, 'HTTP::Headers' )\, '_method' => 'GET' }\, 'HTTP::Request' ) }\, 'HTTP::Response' ); ]0;bob@snoogen:\~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ Script done on Wed Jun 2 12:33:45 2004
Flags: category=library severity=medium
Site configuration information for perl v5.8.0:
Configured by bhcompile at Sun Sep 1 23:55:07 EDT 2002.
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration: Platform: osname=linux\, osvers=2.4.18-11smp\, archname=i386-linux-thread-multi uname='linux daffy.perf.redhat.com 2.4.18-11smp #1 smp thu aug 15 06:41:59 edt 2002 i686 i686 i386 gnulinux ' config_args='-des -Doptimize=-O2 -march=i386 -mcpu=i686 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat\, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr' hint=recommended\, useposix=true\, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n\, bincompat5005=undef Compiler: cc='gcc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm'\, optimize='-O2 -march=i386 -mcpu=i686'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -I/usr/include/gdbm' ccversion=''\, gccversion='3.2 20020822 (Red Hat Linux Rawhide 3.2-5)'\, gccosandvers='' intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=1234 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12 ivtype='long'\, ivsize=4\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8 alignbytes=4\, prototype=define Linker and Libraries: ld='gcc'\, ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil libc=/lib/libc-2.2.92.so\, so=so\, useshrplib=true\, libperl=libperl.so gnulibc_version='2.2.92' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-rdynamic -Wl\,-rpath\,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE' cccdlflags='-fpic'\, lddlflags='-shared -L/usr/local/lib'
Locally applied patches:
@INC for perl v5.8.0: /home/bob/lib /usr/lib/perl5/5.8.0/i386-linux-thread-multi /usr/lib/perl5/5.8.0 /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl .
Environment for perl v5.8.0: HOME=/home/bob LANG=en_GB LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/bob/bin:/usr/bin:/bin:/usr/local/bin:/usr/X11R6/bin:/opt/accurev/bin PERL5LIB=/home/bob/lib PERL_BADLANG (unset) SHELL=/bin/bash
Migrated from rt.perl.org#30019 (status was 'rejected')
Searchable as RT30019$