search

humanmade / S3-Uploads

The WordPress Plugin to Store Uploads on Amazon S3
1.92k stars 389 forks source link

Error executing "ListObjectsV2" on file uploads #625

Closed maikelmertens closed 1 year ago

maikelmertens commented 1 year ago

I have problems running the S3-Uploads plugin on our Wordpress website. I am running Wordpress on Kubernetes and AWS. I've configured an instance profile / service account for the pods so it has full access to the concerning S3-bucket.

Verified that setup is successful by running:

# php wp-cli.phar --allow-root s3-uploads verify
Attempting to upload file s3://my-s3-bucket/uploads/2804493203.txt
File uploaded to S3 successfully.
Attempting to delete file. s3://my-s3-bucket/uploads/2804493203.txt
File deleted from S3 successfully.
Success: Looks like your configuration is correct.

However, when uploading images to the media library, I'm experiencing problems, on the website the following error occurs: The server cannot process the image. This can happen if the server is busy or does not have enough resources to complete the task. Uploading a smaller image may help. Suggested maximum size is 2560 pixels.

In the logging it shows the following exception:

[16-Mar-2023 10:48:46 UTC] PHP Warning:  Uncaught exception 'Aws\S3\Exception\S3Exception' with message 
'Error executing "ListObjectsV2" on "https://my-s3-bucket.s3.eu-west-1.amazonaws.com/?list-type=2&delimiter=%2F&prefix=uploads%2F2023%2F03%2Ftest"; 
AWS HTTP error: Client error: `GET https://my-s3-bucket.s3.eu-west-1.amazonaws.com/?list-type=2&delimiter=%2F&prefix=uploads%2F2023%2F03%2Ftest` resulted in a `403 Forbidden` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>VK2JWB (truncated...)
 AccessDenied (client): Access Denied - <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>VK2JWBS919CE73J9</RequestId><HostId>2SYNP4kY2RO8XdP0HUd5yivJmPua05vXliqHX6LHEce9q1vuNkMLJ9uPe9SUJYM6ziClhTx0fcY=</HostId></Error>'

GuzzleHttp\Exception\ClientException: Client error: `GET https://my-s3-bucket.s3.eu-west-1.amazonaws.com/?list-type=2&delimiter=%2F&prefix=uploads%2F2023%2F03%2Ftest` resulted in a `403 Forbidden` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>VK2JWB (truncated...)
 in /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
Stack trace:
#0 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Middleware.php(69): GuzzleHttp\Exception\RequestException::create()
#1 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(204): GuzzleHttp\Middleware::GuzzleHttp\{closure}()
#2 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(153): GuzzleHttp\Promise\Promise::callHandler()
#3 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#4 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(159): GuzzleHttp\Promise\TaskQueue->run()
#5 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(184): GuzzleHttp\Handler\CurlMultiHandler->tick()
#6 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(248): GuzzleHttp\Handler\CurlMultiHandler->execute()
#7 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(224): GuzzleHttp\Promise\Promise->invokeWaitFn()
#8 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\Promise\Promise->waitIfPending()
#9 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\Promise\Promise->invokeWaitList()
#10 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\Promise\Promise->waitIfPending()
#11 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\Promise\Promise->invokeWaitList()
#12 /var/www/html/content/plugins/s3-uploads/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#13 /var/www/html/content/plugins/s3-uploads/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
#14 /var/www/html/content/plugins/s3-uploads/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute()
#15 /var/www/html/content/plugins/s3-uploads/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
#16 /var/www/html/content/plugins/s3-uploads/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map()
#17 [internal function]: Aws\flatmap()
#18 /var/www/html/content/plugins/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
#19 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
#20 /var/www/html/content/plugins/s3-uploads/inc/class-plugin.php(668): scandir()
#21 /var/www/html/wp/wp-includes/class-wp-hook.php(308): S3_Uploads\Plugin->get_files_for_unique_filename_file_list()
#22 /var/www/html/wp/wp-includes/plugin.php(205): WP_Hook->apply_filters()
#23 /var/www/html/wp/wp-includes/functions.php(2640): apply_filters()
#24 /var/www/html/wp/wp-admin/includes/file.php(954): wp_unique_filename()
#25 /var/www/html/wp/wp-admin/includes/file.php(1075): _wp_handle_upload()
#26 /var/www/html/wp/wp-admin/includes/media.php(303): wp_handle_upload()
#27 /var/www/html/wp/wp-admin/includes/ajax-actions.php(2598): media_handle_upload()
#28 /var/www/html/wp/wp-admin/async-upload.php(33): wp_ajax_upload_attachment()
#29 {main}

This is the configuration I have applied:

define( 'S3_UPLOADS_BUCKET', 'my-s3-bucket' );
define( 'S3_UPLOADS_REGION', 'eu-west-1' );
define( 'S3_UPLOADS_USE_INSTANCE_PROFILE', true );
define( 'S3_UPLOADS_OBJECT_ACL', 'private' );

Furthermore what I am observing too in the logging is a memory size exhausion: [16-Mar-2023 10:48:47 UTC] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 167772160 bytes) in /var/www/html/content/plugins/s3-uploads/inc/class-plugin.php on line 668

Even though I have configured the memory limit to be raised to:

define( 'WP_MEMORY_LIMIT', '256M' );
define( 'WP_MAX_MEMORY_LIMIT', '256M' );

Additionally I have tested the aws-sdk-php with a small test script:

<?php
require_once __DIR__ . '/inc/namespace.php';
require_once dirname( __FILE__ ) . '/vendor/autoload.php';
use Aws\S3\S3Client;
use Aws\Exception\AwsException;

$s3 = new S3Client([
    'version'     => 'latest',
    'region'      => 'eu-west-1',
])

echo $s3->listObjectsV2(["Bucket" => "my-s3-bucket", "Delimiter" => "/", "Prefix" => "uploads/2023/03/test"]);

?>

This is giving a correct result, namely (I have stripped the contents out of the array for readability sake):

# php test.php
{
    "IsTruncated": false,
    "Contents": [
         (...)
    ],
    "Name": "my-s3-bucket",
    "Prefix": "uploads\/2023\/03\/test",
    "Delimiter": "\/",
    "MaxKeys": 1000,
    "KeyCount": 14,
    "@metadata": {
        "statusCode": 200,
        "effectiveUri": "https:\/\/my-s3-bucket.s3.eu-west-1.amazonaws.com\/?list-type=2&delimiter=%2F&prefix=uploads%2F2023%2F03%2Ftest",
        "headers": {
            "x-amz-id-2": "PWlrXITA9514qT+Pr1VafPDg6GZN\/74cptcyaxDyRd8PBx5Gs3JmPjdIH+clEhBVU1JjmJoUmwM=",
            "x-amz-request-id": "8A0WXDSCGSWE1HRP",
            "date": "Thu, 16 Mar 2023 14:49:49 GMT",
            "x-amz-bucket-region": "eu-west-1",
            "content-type": "application\/xml",
            "transfer-encoding": "chunked",
            "server": "AmazonS3"
        }
    }
}

Do you know what is happening and how to solve this issue?

Thank you in advance!

– Maikel

rmccue commented 1 year ago

Can you confirm you have not set either S3_UPLOADS_KEY or S3_UPLOADS_SECRET? Additionally, can you confirm you don't have any filters set on s3_uploads_s3_client_params that could be affecting this?

While it shouldn't matter, the config that S3 Uploads is generating will also set 'signature' => 'v4' in the SDK config.

Allowed memory size of 536870912 bytes exhausted (tried to allocate 167772160 bytes) in /var/www/html/content/plugins/s3-uploads/inc/class-plugin.php on line 668

Line 668 is a comment, so is it possible you're not running the latest version of S3 Uploads? https://github.com/humanmade/S3-Uploads/blob/4705906f1324fdc0e8849053450d517e1cacc382/inc/class-plugin.php#L668

maikelmertens commented 1 year ago

Thanks for the response, I haven't set the S3_UPLOADS_KEY nor the S3_UPLOADS_SECRET, verified that by running:

# php wp-cli.phar --allow-root config list
+---------------------------------+------------------------------------------------------------------------------------+----------+
| name                            | value                                                                              | type     |
+---------------------------------+------------------------------------------------------------------------------------+----------+
| s                               |                                                                                    | variable |
| sp                              | http/1.0                                                                           | variable |
| protocol                        | http                                                                               | variable |
| port                            |                                                                                    | variable |
| serverName                      |                                                                                    | variable |
| table_prefix                    | wp_                                                                                | variable |
(...)
| WP_CONTENT_DIR                  | /var/www/html/content                                                              | constant |
| WP_DEBUG_LOG                    | 1                                                                                  | constant |
| WP_CONTENT_URL                  | http:///content                                                                    | constant |
| WP_AUTO_UPDATE_CORE             |                                                                                    | constant |
| DISALLOW_FILE_MODS              | 1                                                                                  | constant |
| WP_MEMORY_LIMIT                 | 256M                                                                               | constant |
| WP_MAX_MEMORY_LIMIT             | 256M                                                                               | constant |
| WP_DEBUG                        | 1                                                                                  | constant |
| WP_DEBUG_DISPLAY                |                                                                                    | constant |
| S3_UPLOADS_BUCKET               | my-s3-bucket                                                                       | constant |
| S3_UPLOADS_REGION               | eu-west-1                                                                          | constant |
| S3_UPLOADS_USE_INSTANCE_PROFILE | 1                                                                                  | constant |
| S3_UPLOADS_OBJECT_ACL           | private                                                                            | constant |
(...)
| autoload.php                    | /var/www/html/vendor/autoload.php                                                  | includes |
| autoload_real.php               | /var/www/html/vendor/composer/autoload_real.php                                    | includes |
| autoload_static.php             | /var/www/html/vendor/composer/autoload_static.php                                  | includes |
| settings.local.php              | /var/www/html/config/settings.local.php                                            | includes |

Indeed, I was not running the latest version (due to some issue with version pinning at the 3.0.3 release), but now am running the latest version and the same error appears, with the same exhaustion exception on a different line: [28-Mar-2023 07:15:44 UTC] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 83886080 bytes) in /var/www/html/content/plugins/s3-uploads/inc/class-plugin.php on line 676

Grepping the s3_uploads_s3_client_params filter through the entire Wordpress root folder, reveils only this:

# grep -R "s3_uploads_s3_client_params" .
./content/plugins/s3-uploads/README.md:add_filter( 's3_uploads_s3_client_params', function ( $params ) {
./content/plugins/s3-uploads/README.md:add_filter( 's3_uploads_s3_client_params', function ( $params ) {
./content/plugins/s3-uploads/inc/class-plugin.php:      $params = apply_filters( 's3_uploads_s3_client_params', $params );
maikelmertens commented 1 year ago

I've tracked down the issue I was facing. The problem was because I was running php-fpm which spawns the process on the www-data user.

Therefore all environment variables (including the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE) that were automatically been set by K8s on the environment of the user defined within Docker, were not available for this www-data user.

This also makes perfect sense on why my CLI command was successful, like php wp-cli.phar --allow-root s3-uploads verify, but the web requests like uploading media files were failing, the web requests were handled by the php-fpm process on the www-data users whereas the CLI is using the user defined within Docker.

Solution to my problem is to add the environment variables that AWS needs to use, to the php-fpm pool configuration like so:

[www]
env[AWS_ROLE_ARN] = "arn:aws:iam::1234567890:role/my-s3-bucket-role"
env[AWS_REGION] = "eu-west-1"
env[AWS_DEFAULT_REGION] = "eu-west-1"
env[AWS_WEB_IDENTITY_TOKEN_FILE] = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"

Thank you for the response and apologies that this issue in the end was not related to your plugin!