aws-sdk-php fails for listobjects and deleteobjects on gateways

[poornas]

Mint fails with errors on the listobjects and deleteobjects test when run against server running in gateway mode.

Azure and S3 Log

PHP Fatal error:  Uncaught exception 'Aws\S3\Exception\S3Exception' with message 'Error executing "DeleteBucket" on "http://192.168.1.157:9000/aws-sdk-php-bucket-40321"; AWS HTTP error: Client error: `DELETE http://192.168.1.157:9000/aws-sdk-php-bucket-40321` resulted in a `404 Not Found` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Mes (truncated...)
 NoSuchBucket (client): The specified bucket does not exist - <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><Key></Key><BucketName></BucketName><Resource>/aws-sdk-php-bucket-40321</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>'

GuzzleHttp\Exception\ClientException: Client error: `PUT http://192.168.1.157:9000/aws-sdk-php-bucket-13204/obj1` resulted in a `400 Bad Request` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Your metadata headers are not support in /mint/run/core/aws-sdk-php/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php on line 192

GCS logs

PHP Fatal error:  Uncaught Exception: Expected listObjects to fail with InvalidArgument in /mint/run/core/aws-sdk-php/quick-tests.php:130
Stack trace:
#0 /mint/run/core/aws-sdk-php/quick-tests.php(257): runExceptionalTests(Object(Aws\S3\S3Client), 'listObjects', 'getAwsErrorCode', Array)
#1 /mint/run/core/aws-sdk-php/quick-tests.php(961): testListObjects(Object(Aws\S3\S3Client), Array)
#2 /mint/run/core/aws-sdk-php/quick-tests.php(1050): runTest(Object(Aws\S3\S3Client), 'testListObjects', 'listObjects ( a...', Array)
#3 {main}
  thrown in /mint/run/core/aws-sdk-php/quick-tests.php on line 130

[krisis]

GCS logs are due to https://github.com/minio/minio/issues/4975. Will investigate further on Azure/S3 gateway logs.

[krisis]

Azure/S3 logs are due to https://github.com/minio/minio/issues/4983 and https://github.com/minio/minio/issues/4982.

Issues specific to aws-sdk-php based functional tests are fixed by #156. @nitisht can we close this issue?

[nitisht]

@krisis sure. Just cross checking - does https://github.com/minio/mint/pull/156 fix the logging issues, I mean ideally

PHP Fatal error:  Uncaught exception 'Aws\S3\Exception\S3Exception' with message 'Error executing "DeleteBucket" on "http://192.168.1.157:9000/aws-sdk-php-bucket-40321"; AWS HTTP error: Client error: `DELETE http://192.168.1.157:9000/aws-sdk-php-bucket-40321` resulted in a `404 Not Found` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Mes (truncated...)
 NoSuchBucket (client): The specified bucket does not exist - <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><Key></Key><BucketName></BucketName><Resource>/aws-sdk-php-bucket-40321</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>'

GuzzleHttp\Exception\ClientException: Client error: `PUT http://192.168.1.157:9000/aws-sdk-php-bucket-13204/obj1` resulted in a `400 Bad Request` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Your metadata headers are not support in /mint/run/core/aws-sdk-php/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php on line 192

should be wrapped inside JSON logs.

[krisis]

@nitisht no it doesn't fix the exception being thrown outside of the json log output. I shall send a PR to fix it.

[poornas]

From GCS perspective, maxKeys is just a hint - it is currently ignored if -ve and GCS returns the objects ignoring the maxKeys constraint. For that matter, the upperbound for maxKeys of 1000 is also not enforced. Need discussion on whether to enforce s3 constraints on gcs at the server or have the aws php sdk expect and ignore this behavior.

[krisis]

@nitisht https://github.com/minio/mint/pull/161 should avoid printing the exception message after logging the failure as JSON.

[nitisht]

Verified against

Server: minio/minio:edge running as S3 gateway . Commit - c282c8bb58c5ecf3b7bb5daac98fef1b8ef3c915 Mint: Commit - 393ea47e0d2c32b3a0903e9ebf8c02dcbca29148 (after #161 was merged)

Tests get stuck after sometime, I get the below logs. Probably it gets stuck during the test testBucketPolicy, @krisis can you please take a look

{"name":"aws-sdk-php","function":"getBucketLocation ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123"},"duration":"2094","status":"PASS"}
{"name":"aws-sdk-php","function":"listBuckets ( array $params = [] )","args":[],"duration":"576","status":"PASS"}
{"name":"aws-sdk-php","function":"listObjects ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"11768","status":"PASS"}
{"name":"aws-sdk-php","function":"listMultipartUploads ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"1020","status":"NA","error":"Error executing \"UploadPartCopy\" on \"http:\/\/127.0.0.1:9000\/aws-sdk-php-bucket-40123\/obj1-copy?uploadId=5Rej7oGGizyXGIW.9tUK4msVf8jEOTY5uF3I4ehXk8HketKlbs.uV3_nAbhVdkaDx8nW0BLCxWWFx_C6tNS0hQ--&partNumber=1\"; AWS HTTP error: Server error: `PUT http:\/\/127.0.0.1:9000\/aws-sdk-php-bucket-40123\/obj1-copy?uploadId=5Rej7oGGizyXGIW.9tUK4msVf8jEOTY5uF3I4ehXk8HketKlbs.uV3_nAbhVdkaDx8nW0BLCxWWFx_C6tNS0hQ--&partNumber=1` resulted in a `501 Not Implemented` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NotImplemented<\/Code><Message>A header you provided implies function (truncated...)\n NotImplemented (server): A header you provided implies functionality that is not implemented - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NotImplemented<\/Code><Message>A header you provided implies functionality that is not implemented<\/Message><Key><\/Key><BucketName><\/BucketName><Resource>\/aws-sdk-php-bucket-40123\/obj1-copy<\/Resource><RequestId>3L137<\/RequestId><HostId>3L137<\/HostId><\/Error>","message":"Not Implemented"}
{"name":"aws-sdk-php","function":"headBucket ( array $params = [] )","args":["aws-sdk-php-bucket-40123","aws-sdk-php-bucket-21034"],"duration":"22646","status":"PASS"}
{"name":"aws-sdk-php","function":"headObject ( array $params = [] )","args":{"aws-sdk-php-bucket-40123":"obj1","aws-sdk-php-bucket-21034":"obj2"},"duration":"2954","status":"PASS"}
{"name":"aws-sdk-php","function":"getObject ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"1232","status":"PASS"}
{"name":"aws-sdk-php","function":"copyObject ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"1797","status":"PASS"}
{"name":"aws-sdk-php","function":"deleteObjects (array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"4552","status":"PASS"}
{"name":"aws-sdk-php","function":"anonDeleteObjects ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"5835","status":"PASS"}
{"name":"aws-sdk-php","function":"createMultipartUpload ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"14443","status":"PASS"}
{"name":"aws-sdk-php","function":"uploadPart ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"7024","status":"NA","error":"Error executing \"CompleteMultipartUpload\" on \"http:\/\/127.0.0.1:9000\/aws-sdk-php-bucket-40123\/obj1?uploadId=hPUTAP6iefBhN6pF9y0QVW2wRgBSBUHrOcYAiZcq2LwOYS1ILAmuzJH10A0TGcT00QwzqlT6_wLsXT_Wko5Gjg--\"; AWS HTTP error: Server error: `POST http:\/\/127.0.0.1:9000\/aws-sdk-php-bucket-40123\/obj1?uploadId=hPUTAP6iefBhN6pF9y0QVW2wRgBSBUHrOcYAiZcq2LwOYS1ILAmuzJH10A0TGcT00QwzqlT6_wLsXT_Wko5Gjg--` resulted in a `500 Internal Server Error` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError<\/Code><Message>We encountered an internal error, pleas (truncated...)\n InternalError (server): We encountered an internal error, please try again. - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError<\/Code><Message>We encountered an internal error, please try again.<\/Message><Key><\/Key><BucketName><\/BucketName><Resource>\/aws-sdk-php-bucket-40123\/obj1<\/Resource><RequestId>3L137<\/RequestId><HostId>3L137<\/HostId><\/Error>","message":"Not Implemented"}
{"name":"aws-sdk-php","function":"abortMultipartupload ( array $params = [] )","args":{"Bucket":"aws-sdk-php-bucket-40123","Object":"obj1"},"duration":"2899","status":"NA","error":"Error executing \"AbortMultipartUpload\" on \"http:\/\/127.0.0.1:9000\/aws-sdk-php-bucket-40123\/obj1?uploadId=non-existent\"; AWS HTTP error: Server error: `DELETE http:\/\/127.0.0.1:9000\/aws-sdk-php-bucket-40123\/obj1?uploadId=non-existent` resulted in a `500 Internal Server Error` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError<\/Code><Message>We encountered an internal error, pleas (truncated...)\n InternalError (server): We encountered an internal error, please try again. - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError<\/Code><Message>We encountered an internal error, please try again.<\/Message><Key><\/Key><BucketName><\/BucketName><Resource>\/aws-sdk-php-bucket-40123\/obj1<\/Resource><RequestId>3L137<\/RequestId><HostId>3L137<\/HostId><\/Error>","message":"Not Implemented"}

[krisis]

@nitisht https://github.com/minio/mint/pull/177 should fix this hang. You can verify it once the PR is merged.

[nitisht]

sure will do @krisis

[nitisht]

this is fixed by #177