Closed peppies closed 1 year ago
yenfryherrerafeliz
commented
1 year ago Hi @peppies, I have seen this issue before and the issue sometimes is that some of the expected parameters are not properly provided. I am working on a working example that I can provide you with, and that can also point us where the issue is in your implementation.
I will get back to you soon.
Thanks!
peppies
commented
1 year ago @yenfryherrerafeliz Thanks for looking into this, I greatly appreciate it. My cloud objects hosting service (sizable company) has been troubleshooting this for nearly 2 months but has not been getting anywhere. They were successful in reproducing the issue I'm having, but cannot get the basic example to work. This is just a basic video upload, I would hate to think how bad it could get for more advanced functions, such as large video uploads, live streaming and chunking, which we will likely need in production when our app launches.
yenfryherrerafeliz
commented
1 year ago Hi @peppies, sorry for the delay answering to this. Can you please confirm there is not a proxy that maybe modifying your request?, are you executing this request directly to S3 or are you using a custom endpoint?. By using the example below I got not issues:
<?php
require '../vendor/autoload.php';
use Aws\S3\PostObjectV4;
use Aws\S3\S3Client;
$client = new S3CLient([
'region' => getenv('TEST_REGION'),
'version' => 'latest'
]);
$bucket = getenv('TEST_BUCKET');
$inputs = array(
'acl' => 'private',
'key' => 'test.txt',
'content-type' => 'text/plain'
);
$options = [
['acl' => 'private'],
['bucket' => $bucket],
['content-type' => 'text/plain'],
['key' => 'test.txt']
];
$postForm = new PostObjectV4(
$client,
$bucket,
$inputs,
$options
);
$formAttributes = $postForm->getFormAttributes();
$formInputs = $postForm->getFormInputs();
$formParams = [
'acl' => $formInputs['acl'],
'key' => $formInputs['key'],
'content-type' => $formInputs['content-type'],
'X-Amz-Security-Token' => $formInputs['X-Amz-Security-Token'],
'X-Amz-Credential' => $formInputs['X-Amz-Credential'],
'X-Amz-Algorithm' => $formInputs['X-Amz-Algorithm'],
'X-Amz-Date' => $formInputs['X-Amz-Date'],
'Policy' => $formInputs['Policy'],
'X-Amz-Signature' => $formInputs['X-Amz-Signature'],
'file' => file_get_contents('./test.txt'),
];
$fileContent = $formAttributes['action'] . "\n" . json_encode($formParams);
file_put_contents('./form-post', $fileContent);
Python code to execute the request:
import requests
import json
formFile = open('./form-post', 'r')
formContentByLines = formFile.readlines()
formFile.close()
formAction = formContentByLines[0].strip() # The form url for posting against to
formInputs = json.loads(formContentByLines[1].strip()) # The form inputs
del formInputs['file']
files = {
'file': ('test.txt', open('./test.txt', 'rb'))
}
print('Posting to ' + formAction)
response = requests.post(url=formAction, data=formInputs, files=files)
print(response.text)
print(response.status)
Please let me know.
Thanks!
peppies
commented
1 year ago We are not running a proxy for this. Basically, files should upload from a user's browser directly to the S3 URL provided to us in the $postForm->getFormAttributes() response.
I don't know Python, but I was already successful in doing something similar to your example. I was successful in having a user upload a file to our server, where it gets saved temporarily, and then upload that file to the S3. That all works perfectly.
However, we wish to allow users to upload the file from their browser directly to the S3 and bypass the requirement of having to save the file first on our server. The purpose of doing this is to future-proof our app, in case we have 100 people uploading videos at once, we don't want to overwhelm our server, which has limited space and RAM. Instead, have them upload directly to S3. It seems like it would be extremely simple to do this, but for some reason it's not working.
yenfryherrerafeliz
commented
1 year ago Hi @peppies, the issue that you are facing is because you are not specifying the enctype in the form, which should be set to "multipart/form-data". This type is what allows you to post a file along with form fields. Here is how it should look like:
<form action="<?php echo $formAttributes['action']; ?>" method="POST" enctype="multipart/form-data">
<input type="hidden" name="acl" value="<?php echo $formParams['acl']; ?>" />
<input type="hidden" name="key" value="<?php echo $formParams['key']; ?>" />
<input type="hidden" name="content-type" value="<?php echo $formParams['content-type']; ?>" />
<input type="hidden" name="X-Amz-Security-Token" value="<?php echo $formParams['X-Amz-Security-Token']; ?>" />
<input type="hidden" name="X-Amz-Credential" value="<?php echo $formParams['X-Amz-Credential']; ?>" />
<input type="hidden" name="X-Amz-Algorithm" value="<?php echo $formParams['X-Amz-Algorithm']; ?>" />
<input type="hidden" name="X-Amz-Date" value="<?php echo $formParams['X-Amz-Date']; ?>" />
<input type="hidden" name="Policy" value="<?php echo $formParams['Policy']; ?>" />
<input type="hidden" name="X-Amz-Signature" value="<?php echo $formParams['X-Amz-Signature']; ?>" />
<input type="file" name="file">
<input type="submit">
</form>Please let me know if that helps!
Thanks!
peppies
commented
1 year ago Sorry about that, I had that in my file, but forgot to put it in the example above. I've updated the example. I can confirm that even with the "multipart/form-data" added to the form, it doesn't change the result, the upload is still failing.
yenfryherrerafeliz
commented
1 year ago @peppies can you please share what is the error you get?, still method not allowed?
peppies
commented
1 year ago @yenfryherrerafeliz - With instruction from my cloud storage hosting service, I'm trying a variety of different things, with different errors. Now it's always 403 Access Denied or 403 Forbidden errors.
My cloud storage hosting service is telling me to use the PUT method instead of the POST method. Even though AWS itself says to use POST, apparently the AWS docs are wrong and I'm supposed to use PUT instead.....
They also told me to use lower-case letters in the form input variable names (I've updated the example at the top of page). This seems to have gotten rid of the 405 errors, but now it's all 403 errors. While we've been testing, I set the bucket to public-access override and the policy to complete admin privileges, just to rule out any permissions issues:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "*",
"Resource": "arn:aws:s3:::v.mysite.com"
}
]
}When I attempt to upload a video using the HTML form (direct browser to S3 upload) using the PUT method, nothing gets uploaded. Instead, S3 just returns my policy:
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>**********************************</ID>
<DisplayName>********</DisplayName>
</Owner>
<AccessControlList>
<Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
<ID>*****************************</ID>
<DisplayName>********</DisplayName>
</Grantee>
<Permission>FULL_CONTROL</Permission>
</Grant>
<Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
<URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
</Grantee>
<Permission>READ</Permission>
</Grant>
</AccessControlList>
</AccessControlPolicy>If I use the POST method instead (same credentials and everything else), I get a 403 error:
<Error>
<Code>AccessDenied</Code>
<Message>Invalid according to Policy: Policy Condition failed: ["eq", "$content-type", "video\/mp4"]</Message>
<RequestId>5915E3EDDE3AC2B9:B</RequestId>
<HostId>F3OjIFl3wHkMu8ar3RjJr/Tb7Ki+1PEIScbblTw25xpRofbAO4dOA6/Wh3DZi+I4Bj5YLJSdISPo</HostId>
</Error>If I remove the policy and keep it blank, like I would in production, remove the public-access override and use the PUT method, I get this error:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>48F720363B6F48F1:A</RequestId>
<HostId>sKLnkz8G05Bu4XzGUSxP9roq5RNcxbTPTxY+IjqWYYOuxYCEwozzD/Z2dqsKw3KTQwwEcJdoJApo</HostId>
</Error>If I do what I did above, but use the POST method instead, I don't even get an XML respond. Instead, I get default browser error:
Access to s3.*********.com was denied
You don't have authorization to view this page.
HTTP ERROR 403So it seems like I'm supposed to use the PUT method, against what the AWS guide instructs. But even with full public/admin access, I have not been able to upload a single file from browser directly to S3.
Apparently, my cloud storage hosting service is still investigating this and awaiting news. They were able to reproduce the problem on their end. It doesn't sound like they could figure it out.
yenfryherrerafeliz
commented
1 year ago @peppies, please do not use PUT method when working with PostObjectV4, since the expected method is POST. For the following error that you were getting:
AccessDenied
the problem is that you are not specifying the content-type as part of the inputs when you should to. Here is an example that works for me: Note: I recommend you to create a new bucket for testing, and that has ACL enabled, otherwise you will get AccessDenied.
<?php
require './vendor/autoload.php';
use Aws\S3\PostObjectV4;
use Aws\S3\S3Client;
$client = new S3CLient([
'region' => 'us-east-2',
'version' => 'latest',
'credentials' => [
'key' => 'KEY',
'secret' => 'SECRET',
'token' => 'TOKEN-IF-USED'
]
]);
$bucket = 'YOUR-BUCKET';
$key = 'YOUR-KEY';
$contentType = 'text/plain';
$inputs = array(
'acl' => 'private',
'key' => $key,
'content-type' => $contentType
);
$options = [
['acl' => 'private'],
['bucket' => $bucket],
['content-type' => $contentType],
['key' => $key]
];
$postForm = new PostObjectV4(
$client,
$bucket,
$inputs,
$options
);
$formAttributes = $postForm->getFormAttributes();
$formInputs = $postForm->getFormInputs();
$formParams = [
'acl' => $formInputs['acl'],
'key' => $formInputs['key'],
'content-type' => $formInputs['content-type'],
'X-Amz-Security-Token' => $formInputs['X-Amz-Security-Token'],
'X-Amz-Credential' => $formInputs['X-Amz-Credential'],
'X-Amz-Algorithm' => $formInputs['X-Amz-Algorithm'],
'X-Amz-Date' => $formInputs['X-Amz-Date'],
'Policy' => $formInputs['Policy'],
'X-Amz-Signature' => $formInputs['X-Amz-Signature'],
];
?>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>PHP Test</title>
</head>
<body>
<form action="<?php echo $formAttributes['action']; ?>" method="POST" enctype="multipart/form-data">
<input type="hidden" name="acl" value="<?php echo $formParams['acl']; ?>" />
<input type="hidden" name="key" value="<?php echo $formParams['key']; ?>" />
<input type="hidden" name="content-type" value="<?php echo $formParams['content-type']; ?>" />
<input type="hidden" name="X-Amz-Security-Token" value="<?php echo $formParams['X-Amz-Security-Token']; ?>" />
<input type="hidden" name="X-Amz-Credential" value="<?php echo $formParams['X-Amz-Credential']; ?>" />
<input type="hidden" name="X-Amz-Algorithm" value="<?php echo $formParams['X-Amz-Algorithm']; ?>" />
<input type="hidden" name="X-Amz-Date" value="<?php echo $formParams['X-Amz-Date']; ?>" />
<input type="hidden" name="Policy" value="<?php echo $formParams['Policy']; ?>" />
<input type="hidden" name="X-Amz-Signature" value="<?php echo $formParams['X-Amz-Signature']; ?>" />
<input type="file" name="file">
<input type="submit">
</form>
</body>
</html>For what I can tell there is not issue with the SDK or the service her, so please try with my sample code and let me know.
Thanks!
Describe the bug
I am using PostObjectV4 to get presigned post credentials, so users can upload videos from their browser directly to my S3 bucket. When I add the data from PostObjectV4 to my HTML form fields, and attempt to upload a test mp4 video (1.5MB in size) using the POST method, I get these errors:
I've tweaked all sorts of variables, cut my example down to the simplest example possible, but S3 is just failing constantly. I've set my bucket to complete public access and allowed all actions on the bucket policy for testing purposes - nothing should be blocking upload. I've been doing back and forth with my cloud storage hosting company for a few weeks and we are getting nowhere. Apparently, they cannot reproduce the error, but they are not giving me specific details on what they are seeing either.
My bucket policy:
Expected Behavior
PostObjectV4 should give me presigned post credentials and form input values that are good for 20 hours, which I can insert into HTML form fields and the user can select a file on their device to upload. Using the 'POST' method, a user's file should upload directly to the S3 server without errors. After the video is uploaded, it should be immediately available for the public to view in their browser (public-read).
Current Behavior
I receive the PostObjectV4 form fields and add those to my HTML form. The user selects the file to upload, and when it gets sent, there are 405 errors - Method Not Allowed.
Reproduction Steps
The full PHP/HTML code example:
Possible Solution
Something wrong with or related to my Nginx, PHP or SSL configurations???
Additional Information/Context
I enabled total public/admin access to my bucket policy and bucket. All actions are allowed, there should be nothing restricting uploads for my test. My entire PHP/HTML script consists of this, about as simple as it gets:
Important notes:
My bucket is called: v.mysite.com, and it's also a subdomain that I use to host my images on. The subdomain has a CNAME set up and proxied through CloudFlare to: v.mysite.com.s3.[cloudstorage].com. I have a free Universal Edge certificate set up on CloudFlare, which provides SSL to that subdomain as well. So I can host my videos using something like https://v.mysite.com/video.mp4 , it looks nicer and professional.
Edit: Should mention that I had created a separate non-subdomain-name bucket (no CloudFlare proxy) as a test, and that didn't work either...
Also important: that v.mysite.com bucket works fine if users upload the video to my server and I process it and send it to S3 using the
putObjectmethod. I've been doing it for years now with images/videos. However, to prevent my server from overloading if many users upload videos at the same time, I would rather have them upload directly from client browser to S3, and the PostObjectV4 strategy doesn't work. I tried using a similar getSignedUrl('putObject') strategy as well with the same failed results.Here are my nginx.conf file (using nginx version: nginx/1.23.2 compiled with modsecurity):
I disabled modsecurity, restarted nginx and tested it, with no success.
mysite.com.conf Nginx configs:
I have attached my PHP.ini file. I am using PHP 8.1 FPM:
php.txt
Lastly, my cloud storage hosting support asked me to run a cURL function as a test in the shell and provide the output. This is the cURL response:
I was asked to remove the ACL='public-read', and also try ACL='private', but that didn't have any affect. I'm running out of ideas on what is causing this. I'm guessing maybe my Nginx/PHP/SSL configs are messing with S3. I might need to remove/fix/add something in these configs? Or this could be a bug related to the type of setup I'm using.
SDK version used
3.281.4
Environment details (Version of PHP (
php -v)? OS name and version, etc.)PHP 8.1.23 FPM, Ubuntu 22.04, Nginx 1.23.2