search

aws / aws-sdk-php

Official repository of the AWS SDK for PHP (@awsforphp)
http://aws.amazon.com/sdkforphp
Apache License 2.0
6.03k stars 1.22k forks source link

error on S3 get object url/presigned url (php-7.4) #1889

Closed aradhell closed 4 years ago

aradhell commented 5 years ago

Hello, I am having an issue while trying to get urls for the objects.

Version of AWS SDK for PHP?

aws/aws-sdk-php 3.112.25

Version of PHP (php -v)?

php-7.4.0RC2 / RC3 / RC4

What issue did you see?

Error on S3Client get URL / temporary URL ErrorException: Trying to access array offset on value of type null

/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:146
/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:93
/vendor/aws/aws-sdk-php/src/S3/S3Client.php:535
/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:248
/vendor/guzzlehttp/promises/src/Promise.php:203
/vendor/guzzlehttp/promises/src/Promise.php:156
/vendor/guzzlehttp/promises/src/TaskQueue.php:47
/vendor/guzzlehttp/promises/src/Promise.php:246
/vendor/guzzlehttp/promises/src/Promise.php:223
/vendor/guzzlehttp/promises/src/Promise.php:267
/vendor/guzzlehttp/promises/src/Promise.php:225
/vendor/guzzlehttp/promises/src/Promise.php:62
/vendor/aws/aws-sdk-php/src/functions.php:329
/vendor/aws/aws-sdk-php/src/S3/S3Client.php:354
/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php:548
/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php:523
/tests/ReportRequestTest.php:90
/.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:200
/.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:159

Steps to reproduce

Upload an object and try to get url / a presigned url

Additional context

Using Laravel 6 but I also tried it directly from the SDK

diehlaws commented 5 years ago

Hi @aradhell, thanks for reaching out to us. Can you provide a code sample showing how you're attempting to retrieve your object's URL / presigned URL? Unfortunately I haven't been able to reproduce this from my end on docker containers running the php:7.4.0RC3-fpm and php:7.4.0RC4-fpm-alpine images with the following code.

CODE

<?php

require_once("vendor/autoload.php");

use Aws\S3\S3Client;

$bucket = 'my-bucket';
$key = 'php/1889/test';
$body = fopen('test', 'r');

echo "Using AWS SDK for PHP version " . Aws\Sdk::VERSION.PHP_EOL;

$client = new S3Client([
    'region' => 'us-west-2',
    'version' => 'latest',
    ]);

$resp = $client->putObject([
    'Bucket' => $bucket,
    'Key' => $key,
    'Body' => $body,
]);

$url = $client->getObjectUrl($bucket, $key);

print_r($url.PHP_EOL);

$cmd = $client->getCommand('GetObject', [
    'Bucket' => $bucket,
    'Key' => $key
]);
try {
    $request = $client->createPresignedRequest($cmd, '+20 minutes');
    $url = (string)$request->getUri();
    echo "Presigned request created:\n{$url}\n";
} catch (AwsException $e) {
    echo "Error creating presigned request:\n{$e}\n";
}

OUTPUT

PHP 7.4.0RC3

root@8dfb17c25c18:/app# php -v
PHP 7.4.0RC3 (cli) (built: Oct 17 2019 14:40:29) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0-dev, Copyright (c) Zend Technologies
root@8dfb17c25c18:/app# php putObjGetURL.php 
Using AWS SDK for PHP version 3.112.25
https://my-bucket.s3.us-west-2.amazonaws.com/php/1889/test
Presigned request created:
https://my-bucket.s3.us-west-2.amazonaws.com/php/1889/test?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAEXAMPLEABCDEFGHJ%2F20191021%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20191021T182533Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1200&X-Amz-Signature=[redacted]

PHP 7.4.0RC4

/app # php -v
PHP 7.4.0RC4 (cli) (built: Oct 19 2019 03:51:14) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0-dev, Copyright (c) Zend Technologies
/app # php putObjGetURL.php 
Using AWS SDK for PHP version 3.112.25
https://my-bucket.s3.us-west-2.amazonaws.com/php/1889/test
Presigned request created:
https://my-bucket.s3.us-west-2.amazonaws.com/php/1889/test?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAEXAMPLEABCDEFGHJ%2F20191021%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20191021T182608Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1200&X-Amz-Signature=[redacted]
aradhell commented 5 years ago

Hi @diehlaws, thanks for your help. Have you enabled error displaying on notice level?

edit: I used your code to test again.

My output with php-7.4.0RC4 / RC3 / RC2 

PHP Notice:  Trying to access array offset on value of type null in /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php on line 146
PHP Stack trace:
PHP   1. {main}() /Users/username/Documents/GitHub/my-repo/test.php:0
PHP   2. Aws\S3\S3Client->createPresignedRequest() /Users/username/Documents/GitHub/my-repo/test.php:24
PHP   3. Aws\serialize() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/S3/S3Client.php:354
PHP   4. GuzzleHttp\Promise\Promise->wait() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/functions.php:329
PHP   5. GuzzleHttp\Promise\Promise->waitIfPending() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:62
PHP   6. GuzzleHttp\Promise\Promise->invokeWaitList() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:225
PHP   7. GuzzleHttp\Promise\Promise->waitIfPending() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:267
PHP   8. GuzzleHttp\Promise\Promise->invokeWaitFn() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:223
PHP   9. GuzzleHttp\Promise\TaskQueue->run() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:246
PHP  10. GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure:/Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:154-158}() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/TaskQueue.php:47
PHP  11. GuzzleHttp\Promise\Promise::callHandler() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:156
PHP  12. Aws\RetryMiddleware->Aws\{closure:/Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:221-264}() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:203
PHP  13. Aws\S3\S3Client::Aws\S3\{closure:/Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/S3/S3Client.php:530-560}() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:248
PHP  14. Aws\RetryMiddleware::Aws\{closure:/Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:80-110}() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/S3/S3Client.php:535
PHP  15. Aws\RetryMiddleware::isRetryable() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:93

Notice: Trying to access array offset on value of type null in /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php on line 146

Call Stack:
    0.0008     401504   1. {main}() /Users/username/Documents/GitHub/my-repo/test.php:0
    0.1480    7931032   2. Aws\S3\S3Client->createPresignedRequest() /Users/username/Documents/GitHub/my-repo/test.php:24
    0.1494    8009520   3. Aws\serialize() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/S3/S3Client.php:354
    0.1640    8432232   4. GuzzleHttp\Promise\Promise->wait() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/functions.php:329
    0.1640    8432232   5. GuzzleHttp\Promise\Promise->waitIfPending() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:62
    0.1640    8432232   6. GuzzleHttp\Promise\Promise->invokeWaitList() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:225
    0.1640    8432232   7. GuzzleHttp\Promise\Promise->waitIfPending() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:267
    0.1640    8432232   8. GuzzleHttp\Promise\Promise->invokeWaitFn() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:223
    0.1640    8432232   9. GuzzleHttp\Promise\TaskQueue->run() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:246
    0.1641    8427136  10. GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure:/Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:154-158}() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/TaskQueue.php:47
    0.1641    8427136  11. GuzzleHttp\Promise\Promise::callHandler() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:156
    0.1641    8427136  12. Aws\RetryMiddleware->Aws\{closure:/Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:221-264}() /Users/username/Documents/GitHub/my-repo/vendor/guzzlehttp/promises/src/Promise.php:203
    0.1641    8427944  13. Aws\S3\S3Client::Aws\S3\{closure:/Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/S3/S3Client.php:530-560}() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:248
    0.1641    8427944  14. Aws\RetryMiddleware::Aws\{closure:/Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:80-110}() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/S3/S3Client.php:535
    0.1641    8427944  15. Aws\RetryMiddleware::isRetryable() /Users/username/Documents/GitHub/my-repo/vendor/aws/aws-sdk-php/src/RetryMiddleware.php:93

Presigned request created:
https://my-bucket.s3.eu-west-1.amazonaws.com/creditsafe/reports/file.txt?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAZY3D2OYTCSAPCUTI%2F20191021%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20191021T214925Z&X-Amz-SignedHeaders=host&X-Amz-Expires=12000&X-Amz-Signature=bfb4f466cf76b62bc38c5043bf17df867f069a33950dd2a3c8397df7b09f410b
mwasper commented 5 years ago

@aradhell did you solve this problem? I have same one...

haxmedia commented 5 years ago

"retries" attribute sets to 0 solve the problem for me. Also works in Laravel filesystems configuration.

$client = new S3Client([ 'retries' => 0 ]);

aradhell commented 5 years ago

@aradhell did you solve this problem? I have same one...

I ignored warnings as a temporary solution, couldn't find time to dig it deeper. @haxmedia suggests another workaround.

Demoboy commented 4 years ago

I experienced the same issue. It appears that in some cases, RetryMiddleware::isRetryable is called with an empty result object passed to it where $result['@metadata'] is not set. This quick patch resolves that issue by returning false in those cases.

diff --git a/src/RetryMiddleware.php b/src/RetryMiddleware.php
index 4ad945a9c..d5e65a30d 100644
--- a/src/RetryMiddleware.php
+++ b/src/RetryMiddleware.php
@@ -143,6 +143,10 @@ class RetryMiddleware
         }

         if (!$error) {
+            if (!isset($result['@metadata'])) {
+                return false;
+            }
+            
             return isset($statusCodes[$result['@metadata']['statusCode']]);
         }
jefersonralmeida commented 4 years ago

The final release of PHP7.4 triggers a notice (during RC versions it was a warning), when you try to use array access (like $foo['bar']) on a variable that is not an array (like when $foo value is null). @Demoboy solution works (because there are flows where $result['@metadata'] can be null. I did the same on my env (before seen his message):

<?php
return is_array($result['@metadata']) ? isset($statusCodes[$result['@metadata']['statusCode']]) : false;
georgeboot commented 4 years ago

In my Laravel project, this issue happened somewhere else in the code base: https://github.com/aws/aws-sdk-php/blob/8fa68de81738f851e0aa62fbc0a657f2905aa860/src/S3/PutObjectUrlMiddleware.php#L46-L48

Both provided solutions don't work for me, as the error happens before RetryMiddleware. This also means that the proposed fix in #1912 isn't sufficient.

I'm using php 7.4.0 and sdk verson 3.124.0.

Full error:

{
  "message": "Trying to access array offset on value of type null",
  "exception": "ErrorException",
  "file": "/path/to/project/vendor/aws/aws-sdk-php/src/S3/PutObjectUrlMiddleware.php",
  "line": 48,
  "trace": [
    {
      "file": "/path/to/project/vendor/aws/aws-sdk-php/src/S3/PutObjectUrlMiddleware.php",
      "line": 48,
      "function": "handleError",
      "class": "Illuminate\\Foundation\\Bootstrap\\HandleExceptions",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 203,
      "function": "Aws\\S3\\{closure}",
      "class": "Aws\\S3\\PutObjectUrlMiddleware",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 156,
      "function": "callHandler",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "::"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/TaskQueue.php",
      "line": 47,
      "function": "GuzzleHttp\\Promise\\{closure}",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "::"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 246,
      "function": "run",
      "class": "GuzzleHttp\\Promise\\TaskQueue",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 223,
      "function": "invokeWaitFn",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 267,
      "function": "waitIfPending",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 225,
      "function": "invokeWaitList",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/guzzlehttp/promises/src/Promise.php",
      "line": 62,
      "function": "waitIfPending",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/aws/aws-sdk-php/src/functions.php",
      "line": 328,
      "function": "wait",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/aws/aws-sdk-php/src/S3/S3Client.php",
      "line": 370,
      "function": "Aws\\serialize"
    },
    {
      "file": "/path/to/project/vendor/laravel/vapor-core/src/Http/Controllers/SignedStorageUrlController.php",
      "line": 36,
      "function": "createPresignedRequest",
      "class": "Aws\\S3\\S3Client",
      "type": "->"
    },
    {
      "function": "store",
      "class": "Laravel\\Vapor\\Http\\Controllers\\SignedStorageUrlController",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Controller.php",
      "line": 54,
      "function": "call_user_func_array"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php",
      "line": 45,
      "function": "callAction",
      "class": "Illuminate\\Routing\\Controller",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Route.php",
      "line": 219,
      "function": "dispatch",
      "class": "Illuminate\\Routing\\ControllerDispatcher",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Route.php",
      "line": 176,
      "function": "runController",
      "class": "Illuminate\\Routing\\Route",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Router.php",
      "line": 692,
      "function": "run",
      "class": "Illuminate\\Routing\\Route",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 130,
      "function": "Illuminate\\Routing\\{closure}",
      "class": "Illuminate\\Routing\\Router",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php",
      "line": 41,
      "function": "Illuminate\\Pipeline\\{closure}",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 171,
      "function": "handle",
      "class": "Illuminate\\Routing\\Middleware\\SubstituteBindings",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 105,
      "function": "Illuminate\\Pipeline\\{closure}",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Router.php",
      "line": 694,
      "function": "then",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Router.php",
      "line": 669,
      "function": "runRouteWithinStack",
      "class": "Illuminate\\Routing\\Router",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Router.php",
      "line": 635,
      "function": "runRoute",
      "class": "Illuminate\\Routing\\Router",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Routing/Router.php",
      "line": 624,
      "function": "dispatchToRoute",
      "class": "Illuminate\\Routing\\Router",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php",
      "line": 176,
      "function": "dispatch",
      "class": "Illuminate\\Routing\\Router",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 130,
      "function": "Illuminate\\Foundation\\Http\\{closure}",
      "class": "Illuminate\\Foundation\\Http\\Kernel",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/spatie/laravel-cors/src/Cors.php",
      "line": 28,
      "function": "Illuminate\\Pipeline\\{closure}",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 171,
      "function": "handle",
      "class": "Spatie\\Cors\\Cors",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/fideloper/proxy/src/TrustProxies.php",
      "line": 57,
      "function": "Illuminate\\Pipeline\\{closure}",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 171,
      "function": "handle",
      "class": "Fideloper\\Proxy\\TrustProxies",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php",
      "line": 105,
      "function": "Illuminate\\Pipeline\\{closure}",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php",
      "line": 151,
      "function": "then",
      "class": "Illuminate\\Pipeline\\Pipeline",
      "type": "->"
    },
    {
      "file": "/path/to/project/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php",
      "line": 116,
      "function": "sendRequestThroughRouter",
      "class": "Illuminate\\Foundation\\Http\\Kernel",
      "type": "->"
    },
    {
      "file": "/path/to/project/public/index.php",
      "line": 55,
      "function": "handle",
      "class": "Illuminate\\Foundation\\Http\\Kernel",
      "type": "->"
    },
    {
      "file": "/Users/xxxxx/.composer/vendor/laravel/valet/server.php",
      "line": 158,
      "function": "require"
    }
  ]
}
jefersonralmeida commented 4 years ago

I updated the PR on https://github.com/aws/aws-sdk-php/pull/1912, to to cover @georgeboot issue.

georgeboot commented 4 years ago

This was fixed in #1916

howardlopez commented 4 years ago

As noted by georgeboot, this was resolved in #1916.