Getting Forbidden error when trying a path with a space in it

[amergrgic]

Testing on Ubuntu 18.04, Perl 5.26.1 and Paws 0.41

When I try to do a simple upload to S3 with a small script (see the script below), I get a Forbidden error, when the key doesn't have a space it does work. Don't know if it's related to #323, but it seems to be. This is a huge problem that I really need a workaround for. Anyone have an idea? I have tested with versions > 0.37 and all have this same issue.

use Paws;

$ENV{'AWS_ACCESS_KEY'} = "xxx";
$ENV{'AWS_SECRET_KEY'} = "xxx";

my $s3 = Paws->service('S3', region => "eu-west-1");

$s3->PutObject(
        Bucket  => "bucketname",
        Key     => "text test.txt",
        Body    => "text.txt",
);

[byterock]

Hmm I have been working extensively on S3 paws ofer the past few weeks.

If you have the opportunity.

Try this branch of paws

https://github.com/byterock/aws-sdk-perl/tree/s3ObjectTagging

So you are saying that

key => "text test.txt",

flails

and

key => "texttest.txt",

passes??

I will check that

[amergrgic]

@byterock yes, exactly. I have tried with the Paws CLI as well:

paws S3 --region eu-west-1 PutObject Bucket: bucketname Key: "test text.txt" Body: text.txt

returns:

Paws::S3 is not stable / supported / entirely developed at /usr/local/share/perl/5.26.1/Paws/S3.pm line 2.
The API retured an error: Forbidden(403)

Have tried on multiple different servers, all with the same exact result.

When trying without space in the Key:

paws S3 --region eu-west-1 PutObject Bucket: bucketname Key: "testtext.txt" Body: text.txt

Returns:

Paws::S3 is not stable / supported / entirely developed at /usr/local/share/perl/5.26.1/Paws/S3.pm line 2.
ETag: "034d361a5942e67697d17534f37ed5a9"

Update: I have tried with your branch as well, but still am getting a Forbidden. Output of my testscript:

Paws::S3 is not stable / supported / entirely developed at /usr/local/share/perl/5.26.1/Paws/S3.pm line 2.
Forbidden

Trace begun at /usr/local/share/perl/5.26.1/Paws/Net/RestXMLResponse.pm line 25
Paws::Net::RestXMLResponse::process('Paws::Net::RestXMLResponse=HASH(0x55e924bd0a30)', 'Paws::S3::PutObject=HASH(0x55e924b9e5e0)', 'Paws::Net::APIResponse=HASH(0x55e924d3cad0)') called at /usr/local/share/perl/5.26.1/Paws/Net/Caller.pm line 46
Paws::Net::Caller::caller_to_response('Paws::Net::Caller=HASH(0x55e923a0f598)', 'Paws::S3=HASH(0x55e92466f678)', 'Paws::S3::PutObject=HASH(0x55e924b9e5e0)', 'Paws::Net::APIResponse=HASH(0x55e924d3cad0)') called at /usr/local/share/perl/5.26.1/Paws/Net/RetryCallerRole.pm line 19
Paws::Net::RetryCallerRole::do_call('Paws::Net::Caller=HASH(0x55e923a0f598)', 'Paws::S3=HASH(0x55e92466f678)', 'Paws::S3::PutObject=HASH(0x55e924b9e5e0)') called at /usr/local/share/perl/5.26.1/Paws/S3.pm line 451
Paws::S3::PutObject('Paws::S3=HASH(0x55e92466f678)', 'Bucket', 'bucketname', 'Key', 'text test.txt', 'Body', 'text.txt') called at test_s3.pl line 8

[byterock]

Well first you need Git installed ;)

https://github.com/byterock/aws-sdk-perl.git

git checkout s3ObjectTagging

Then follow the normal install dev steps

https://github.com/pplu/aws-sdk-perl#development-setup

this command will recompile just the S3 bits

carton exec builder-bin/gen_classes.pl --classes botocore/botocore/data/s3/2006-03-01/service-2.json

Alternatively if you really just want to hack about you can just drop these files

https://github.com/byterock/aws-sdk-perl/blob/s3ObjectTagging/lib/Paws/Net/RestXmlCaller.pm https://github.com/byterock/aws-sdk-perl/blob/s3ObjectTagging/lib/Paws/Net/RestXMLResponse.pm

on the correct path of your Paws it should fix it

[amergrgic]

Well first you need Git installed ;)

https://github.com/byterock/aws-sdk-perl.git

git checkout s3ObjectTagging

Then follow the normal install dev steps

https://github.com/pplu/aws-sdk-perl#development-setup

this command will recompile just the S3 bits

carton exec builder-bin/gen_classes.pl --classes botocore/botocore/data/s3/2006-03-01/service-2.json

Alternatively if you really just want to hack about you can just drop these files

https://github.com/byterock/aws-sdk-perl/blob/s3ObjectTagging/lib/Paws/Net/RestXmlCaller.pm https://github.com/byterock/aws-sdk-perl/blob/s3ObjectTagging/lib/Paws/Net/RestXMLResponse.pm

on the correct path of your Paws it should fix it

Yeah, I updated my comment, even your branch didn't fix the issue.

[byterock]

Opps did not see your update!

Sorry for the above.

Normally I hack into the two '.pm' files and dump the 'response' and 'request' objects to see what is really coming back. I will be working on this shortly

[amergrgic]

Opps did not see your update!

Sorry for the above.

Normally I hack into the two '.pm' files and dump the 'response' and 'request' objects to see what is really coming back. I will be working on this shortly

@byterock I tried btw replacing those two .pm files with the ones from your branch and I get a different error on my screen:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

That's with my testscript, Paws CLI gives me back:

The API retured an error: The request signature we calculated does not match the signature you provided. Check your key and signing method.(SignatureDoesNotMatch)

I tried turning on your Data::Dumper inside the files you created on your branch, result is below:

JSP request=$VAR1 = bless( {
                 'method' => 'PUT',
                 'content' => 'text.txt',
                 'parameters' => {
                                   'Body' => 'text.txt',
                                   'Bucket' => 'bucketname',
                                   'Key' => 'text test.txt'
                                 },
                 'url' => 'https://s3-eu-west-1.amazonaws.com/bucketname/text%20test.txt',
                 'content_length' => 8,
                 'headers' => bless( {
                                       'content-length' => 8,
                                       'content-md5' => 'A002GllC5naX0XU0837VqQ=='
                                     }, 'HTTP::Headers' ),
                 'date' => '20191121T134220Z',
                 'uri' => '/bucketname/text%20test.txt'
               }, 'Paws::Net::S3APIRequest' );
The request signature we calculated does not match the signature you provided. Check your key and signing method.

[byterock]

I just ran that test and it worked fine for me ;)

Which is good on my part but not on yours

Is that bucket of yours public. I can try and upload from my end?

My request and response are

request=$VAR1 = bless( { 'content_length' => 8, 'date' => '20191121T134041Z', 'url' => 'https://s3.amazonaws.com/dev.cargotel.paws/text%20test.txt', 'headers' => bless( { 'content-md5' => 'A002GllC5naX0XU0837VqQ==', 'content-length' => 8 }, 'HTTP::Headers' ), 'uri' => '/dev.cargotel.paws/text%20test.txt', 'method' => 'PUT', 'content' => 'text.txt', 'parameters' => { 'Body' => 'text.txt', 'Bucket' => 'dev.cargotel.paws', 'Key' => 'text test.txt' } }, 'Paws::Net::S3APIRequest' ); process response=$VAR1 = bless( { 'headers' => { 'content-length' => '0', 'server' => 'AmazonS3', 'x-amz-version-id' => 'XVHVnuB3ZiO6vYyqaURdc.NveEKz9aGD', 'date' => 'Thu, 21 Nov 2019 13:40:43 GMT', 'etag' => '"034d361a5942e67697d17534f37ed5a9"', 'x-amz-request-id' => 'DFCBC01EA7C0A2CB', 'x-amz-id-2' => 'W259s/K3Na/Mi/Mb9NzSgMqajHcMPKCmlIfpCnCIjZuIq6nicUihsCH4bs0XfXR5kzGDKABdVms=' }, 'status' => '200', 'content' => '' }, 'Paws::Net::APIResponse' ); PutObject =$VAR1 = bless( { '_request_id' => 'DFCBC01EA7C0A2CB', 'ETag' => '"034d361a5942e67697d17534f37ed5a9"', 'VersionId' => 'XVHVnuB3ZiO6vYyqaURdc.NveEKz9aGD' }, 'Paws::S3::PutObjectOutput' );

[amergrgic]

I just ran that test and it worked fine for me ;)

Which is good on my part but not on yours

Is that bucker of yorus public. I can try and upload from my end?

Bucket is not public unfortunately, did you test with a bucket that has a policy on it?

P.s. I added the data::dumper dump in my previous comment.

[byterock]

You are always on step ahead of me ;)

[byterock]

Yeah the code swap out was jsut a Hail Mary Pass on my part :)

You most likly have to rebuild after a checkout of my branch.

Could be a case of the policy being incorrect.

If you can go back to your .41 branch and add in that dumper warning and compair to what we have above

[byterock]

Don't forget if you download my branch you still have to 'checkout' my branch

ie

git branch s3-todo-cleanup

• s3ObjectTagging

git checkout s3ObjectTagging

but you might already knwo that

Btw I am getting an 'Access Denied' which is expected when I try to upload to your bucket

[amergrgic]

@byterock good call, I think I forgot that the first time, but it still didn't help. The test you are doing, is that on Ubuntu 18.04 with Perl 5.26.1? I have a feeling it might be something Ubuntu related, doing a test on Ubuntu 14.04 with 0.41 now as well.

[amergrgic]

Update: Ubuntu 14.04 with Paws 0.41 works, but it encodes my filename to text%20text.txt, but that is something I had before as well and kind of patched (was in a different issue as well, workaround).

[amergrgic]

Ok, I have found the problem. The problem resides in using V4.pm in /usr/local/share/perl/5.26.1/Net/Amazon/Signature.

The new version of that V4.pm (0.19) doesn't seem to be working correctly with Paws > 0.37. When I moved V4.pm from a machine with Ubuntu 14.04 (which is version 0.16), my Paws works.

I don't know why it doesn't work exactly, but it works when I switched these files out. I think this is something for you guys to fix, as it means that anyone on this version of Ubuntu wouldn't be able to work with Paws (if they use spaces)

[byterock]

Glad I could help on that

Will see that bug might be my next blogging project S3 is coming to an end

[galenhuntington]

This looks the same as #323, where the issue is the encoded (rather than original) key being signed. Have you tried my workaround there (altering S3Signature.pm to decode the uri field in-place before signing)?

[castaway]

It looks like we might need this PR applying to Signature::V4.. https://github.com/Grinnz/Net-Amazon-Signature-V4/pull/7

[Grinnz]

Net::Amazon::Signature::V4 0.20 has this change, please try updating. Thanks

[webtoolbox]

Upgrading Net::Amazon::Signature::V4 fixes the issue related to spaces in the object key, but it still fails in some cases. For example, with a file name like ~lkta100.gif.