Install hangs on CentOS 6 (untar failed)

[alanmead]

Below is the command and the relevant output. Happens reliably for me. The install just hangs (I waited 10-15 minutes) after the last line. When I hit ^C (twice) it says "untar failed". This server should have GB of free disk space.

$ sudo cpan Paws::MTurk
CPAN: Storable loaded ok (v2.20)
Going to read '/root/.cpan/Metadata'
  Database was generated on Mon, 07 Aug 2017 22:41:03 GMT
Running install for module 'Paws::MTurk'
CPAN: YAML loaded ok (v0.70)
Running make for J/JL/JLMARTIN/Paws-0.34.tar.gz
CPAN: Digest::SHA loaded ok (v5.47)
CPAN: Compress::Zlib loaded ok (v2.052)
Checksum for /root/.cpan/sources/authors/id/J/JL/JLMARTIN/Paws-0.34.tar.gz ok
CPAN: Archive::Tar loaded ok (v1.58)
Paws-0.34
Paws-0.34/TODO
Paws-0.34/Changes
Paws-0.34/LICENSE
[...]
Paws-0.34/lib/Paws/SimpleWorkflow/RequestCancelExternalWorkflowExecutionFailedEventAttributes.pm
Paws-0.34/lib/Paws/SimpleWorkflow/RequestCancelExternalWorkflowExecutionInitiatedEventAttributes.pm
^CCaught SIGINT, trying to continue
^CGot another SIGINTCPAN: File::Temp loaded ok (v0.22)
Failed during this command:
 JLMARTIN/Paws-0.34.tar.gz                    : unwrapped NO -- untar failed

[pplu]

Ẃow! That's quite strange!

What happens if you download https://cpan.metacpan.org/authors/id/J/JL/JLMARTIN/Paws-0.34.tar.gz and untar with te system tar utility? tar xvzf Paws-0.34.tar.gz? If it uncompresses, then try to cd Paws-0.34, perl Makefile.PL, make and make install.

[alanmead]

Yes, I can manually DL, untar, and make it.

You didn't say to "make test" and the tests do not pass (and throw a lot of warnings and errors). Is that surprising? Or should I not bother with the tests? I went ahead and "make install"d it and I can get a simple script working.

[pplu]

I thought that make test would be automatically invoked 😮 . Can you send me the output of make test? I'm really interested in what could be failing.

[alanmead]

In the attached file, there is a 'make test' which suggests that Test::Exception is out-of-date, so I installed 0.43 and performed 'make test' again.

I imagine it's the old Perl version that I'm using on CentOS 6 and also because of missing dependencies. I had to install these modules and I didn't save the 'make test' before I did so:

sudo cpan Paws::MTurk sudo cpan ARGV::Struct sudo cpan Config::INI sudo cpan DateTime::Format::ISO8601 sudo cpan File::HomeDir sudo cpan Hash::Flatten sudo cpan MooseX::ClassAttribute sudo cpan Net::Amazon::Signature::V3 Net::Amazon::Signature::V4 String::CRC32 Throwable::Error URI::Template URL::Encode URL::Encode::XS sudo cpan Class::Unload

Most of the above were listed by perl Makefile.PL, but Class::Unload caused an error in the tests... I thought maybe it was needed only for testing but installing it didn't completely clean up the 'make test'.

The part I found most alarming was the tests that seemed to indicate simple Perl warnings like ""my" variable $known_regions masks earlier declaration in same scope at t/12_regions.t line 285." and "Use of uninitialized value $ENV{"IN_TRAVIS"} in numeric eq (==) at t/14_dns_client_errors.t line 13." And, of course, there are some real failures. test-output2.txt

[jimsmith]

@pplu paws-output_30-01-2018.txt

In the attached file I'm getting similar errors as @alanmead on CentOS 7.4 64bit OS when doing sudo cpan Paws

This is halting me making use of Paws for writing perl script to upload objects into S3 in different regions and also to tap into AWS API's using perl.

[jimsmith]

@pplu paws_installed_30-01-2018.txt

My workaround has got this installed for anyone else getting this issue.

cd /root/.cpan/build/Paws-0.36-0/t/ mv 13_client_connect_errors.t ../ mv 14_dns_client_errors.t ../

make test make install

[root@centos7 Paws-0.36-0]# which paws /root/perl5/bin/paws

[pplu]

@jimsmith thanks for pointing to the failing tests!

[jimsmith]

@pplu any idea why it's installed in /root/perl5 as when I run a sample perl script I get this error:

Can't locate Paws.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./s3-upload.pl line 6.
BEGIN failed--compilation aborted at ./s3-upload.pl line 6.

sudo locate Paws.pm
/root/.cpan/build/Paws-0.36-0/blib/lib/Paws.pm
/root/.cpan/build/Paws-0.36-0/lib/Paws.pm
/root/perl5/lib/perl5/Paws.pm

Update: this resolved it https://stackoverflow.com/a/36691002

FYI after working thru getting this installed, I've noticed that a simple S3 PutObject into a bucket, consumes a tremendous amount of resident memory (4.256Gb to be precise) and observed 100%-57% CPU.

Is this normal expected behaviour ? as a comparison using awscli s3 cp this didn't even register hardly any resource/usage at all.

[pplu]

@jimsmith thanks for the info. We'll leave this case open with the question label so others can find the information.

The memory consumption for S3 is quite expected. You're almost surely uploading a big object to S3. Paws loads the contents in memory before sending the HTTP request, and is maybe copying the contents more times than needed. There is the intention of getting Paws to be able to accept a filehandle, and pass that handle directly to the HTTP client so that the whole contents of the file don't have to be slurped to memory, but I don't know when that will happen. I (and I'm sure that others) would be very happy if you want to make that happen (you would receive help)

[jimsmith]

My perl is rusty would these help the project ?

http://blog.nmcfarl.org/ https://www.example-code.com/perl/amazonS3.asp

[pplu]

Hi,

It doesn't matter if your Perl is a bit rusty: we'll unrust you :)

http://blog.nmcfarl.org/ has the clues, and https://metacpan.org/source/YAPPO/HTTP-Request-StreamingUpload-0.01/lib/HTTP/Request/StreamingUpload.pm has the interesting part (how to configure the user agent to read from file descriptor in chunks).

Basically you want to take https://github.com/pplu/aws-sdk-perl/blob/master/auto-lib/Paws/S3/PutObject.pm#L5 and let it be a filehandle too, then go to https://github.com/pplu/aws-sdk-perl/blob/master/lib/Paws/API/Caller.pm#L41, detect the type and let the filehandle be passed. Then in https://github.com/pplu/aws-sdk-perl/blob/master/lib/Paws/Net/APIRequest.pm and let the content accept a filehandle too. Then, in https://github.com/pplu/aws-sdk-perl/blob/master/lib/Paws/Net/Caller.pm, you want to put the logic to read in chunks from the filehandle if the content property is a filehandle.

Note: the documentation for passing a callback routine to HTTP::Tiny: https://metacpan.org/pod/HTTP::Tiny#request

This should get you going

[joschi99]

I hade the same issue on Paws with and failed tests with 14_dns_client_errors.t. It was a firewall related problem and the checks failed for this reason. Temporary I configured the firewall to allow any and the installation went fine.