aws-sdk-php tests fail for server/gateway

Mint test on latest rc failed. Reproducible with both Server and Gateway modes

Mint logs:

To get logs, run 'sudo docker cp 7bf35828e9e5:/mint/log /tmp/mint-logs'
(1/9) Running aws-sdk-php tests ... FAILED in 1 seconds
{
  "name": "aws-sdk-php",
  "function": "presignedPut ( array $params = [] )",
  "args": {
    "Bucket": "aws-sdk-php-5a0a92dc59ad6",
    "Object": "obj1"
  },
  "duration": "13",
  "status": "FAIL",
  "error": "Server error: `PUT http://127.0.0.1:9000/aws-sdk-php-5a0a92dc59ad6/obj1?X-Amz-Content-Sha256=invalid-sha256&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171114T065317Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1200&X-Amz-Signature=6f0bc30857ea246de1d84b92d91238d9aa83b21168727017e844b820e6a523c8` resulted in a `500 Internal Server Error` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, pleas (truncated...)\n"
}

Server logs:

root@nitish-mint-test:~# ./minio server /tmp/test
Endpoint:  http://147.75.94.237:9000  http://10.64.0.129:9000  http://172.17.0.1:9000  http://127.0.0.1:9000
AccessKey: minio 
SecretKey: minio123 
Region:    us-east-1

Browser Access:
   http://147.75.94.237:9000  http://10.64.0.129:9000  http://172.17.0.1:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://147.75.94.237:9000 minio minio123

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

Drive Capacity: 99 GiB Free, 102 GiB Total
ERRO[0007] Unable to fetch bucket info.                  cause=Bucket name invalid: aws-sdk-php-5a0a92dc59ad6-- source=[bucket-handlers.go:130:objectAPIHandlers.GetBucketLocationHandler()] stack=fs-v1.go:193:fsObjects.getBucketDir fs-v1.go:201:fsObjects.statBucketDir fs-v1.go:229:fsObjects.GetBucketInfo <autogenerated>:1:(*fsObjects).GetBucketInfo bucket-handlers.go:129:objectAPIHandlers.GetBucketLocationHandler api-router.go:69:(objectAPIHandlers).GetBucketLocationHandler-fm
ERRO[0007] Unable to fetch bucket info.                  cause=Bucket name invalid: aws-sdk-php-5a0a92dc59b12-- source=[bucket-handlers.go:659:objectAPIHandlers.HeadBucketHandler()] stack=fs-v1.go:193:fsObjects.getBucketDir fs-v1.go:201:fsObjects.statBucketDir fs-v1.go:229:fsObjects.GetBucketInfo <autogenerated>:1:(*fsObjects).GetBucketInfo bucket-handlers.go:658:objectAPIHandlers.HeadBucketHandler api-router.go:89:(objectAPIHandlers).HeadBucketHandler-fm
ERRO[0007] Unable to delete object. obj1-copy0           cause=Prefix access is denied: aws-sdk-php-5a0a92dc59ad6/obj1-copy0 source=[bucket-handlers.go:337:objectAPIHandlers.DeleteMultipleObjectsHandler()]
ERRO[0007] Unable to delete object. obj1-copy1           cause=Prefix access is denied: aws-sdk-php-5a0a92dc59ad6/obj1-copy1 source=[bucket-handlers.go:337:objectAPIHandlers.DeleteMultipleObjectsHandler()]
ERRO[0007] Unable to delete object. obj1-copy2           cause=Prefix access is denied: aws-sdk-php-5a0a92dc59ad6/obj1-copy2 source=[bucket-handlers.go:337:objectAPIHandlers.DeleteMultipleObjectsHandler()]
ERRO[0008] Unable to complete multipart upload.          cause=Part size for 1 should be at least 5MB source=[object-handlers.go:1061:objectAPIHandlers.CompleteMultipartUploadHandler()] stack=fs-v1-multipart.go:806:fsObjects.CompleteMultipartUpload <autogenerated>:1:(*fsObjects).CompleteMultipartUpload object-handlers.go:1059:objectAPIHandlers.CompleteMultipartUploadHandler api-router.go:52:(objectAPIHandlers).CompleteMultipartUploadHandler-fm

Findings from investigation

aws-sdk-php doesn't include x-amz-contentsha256 in signed headers

http://s3.amazonaws.com/aws-sdk-php-5a0aeaaedaad0/obj1?X-Amz-Content-Sha256=invalid-sha256&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=**REDACTED**%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171114T130808Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1200&X-Amz-Signature=**REDACTED**

Presigned PUT URL with invalid ContentSha256 completes successfully on AWS S3, i.e sha256 sum is not validated in a presigned PUT request. This can be verified using the following sample php program.
```
<?php
require 'vendor/autoload.php';
```

use Aws\S3\S3Client; use Aws\Credentials; use Aws\Exception\AwsException; use GuzzleHttp\Psr7; use GuzzleHttp\Client;

$s3Client = new S3Client([ 'credentials' => [ 'key' => 'accessKey', 'secret' => 'secretKey', ], 'scheme' => 'http', 'use_path_style_endpoint' => true, 'region' => 'us-east-1', 'version' => '2006-03-01' ]); $bucket = 'mybucket'; $object = 'myobject'; $sha256 = "invalid-sha256";

$cmd = $s3Client->getCommand('PutObject', [ 'Bucket' => $bucket, 'Key' => $object, 'ContentSHA256' => $sha256, ]);

$request = $s3Client->createPresignedRequest($cmd, '+20 minutes'); $presignedUrl = (string) $request->getUri(); echo $presignedUrl;

$client = new Client(['debug' => true]); $body = "Hello, World"; $res = $client->request('PUT', $presignedUrl, ['body' => $body]); echo 'statusCode: ' . $res->getStatusCode();



### Resolution
1. aws-sdk-php mint test
This test doesn't capture a [valid use-case](http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html), nor is it a negative test-case. `ContentSha256` is ignored in a presigned PUT request. To quote from AWS S3 docs,
>You don't include a payload hash in the Canonical Request, because when you create a presigned URL, you don't know the payload content because the URL is used to upload an arbitrary payload. Instead, you use a constant string UNSIGNED-PAYLOAD. 

2. minio-server
Minio server should not validate content-sha256 for presigned requests.

We should change the test to incorporate this perhaps

package main() {
      //creds := credentials.NewStaticCredentials("USWUXHGYZQYFYFFIT3RE", "MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03", "")                                                                                          
        creds := credentials.NewStaticCredentials(os.Getenv("ACCESS_KEY"), os.Getenv("SECRET_KEY"), "")                                                                                                              
        newSession := session.New()                                                                                                                                                                                  
        s3Config := &aws.Config{                                                                                                                                                                                     
                Credentials: creds,                                                                                                                                                                                  
                //Endpoint:         aws.String("http://localhost:9000"),                                                                                                                                             
                Endpoint: aws.String("https://s3.amazonaws.com"),                                                                                                                                                    
                Region:   aws.String("us-east-1"),                                                                                                                                                                   
                //DisableSSL:       aws.Bool(true),                                                                                                                                                                  
                //S3ForcePathStyle: aws.Bool(true),                                                                                                                                                                  
        }                                                                                                                                                                                                            

        // Create an S3 service object in the default region.                                                                                                                                                        
        s3Client := s3.New(newSession, s3Config)                                                                                                                                                                     
        req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{                                                                                                                                                      
                Bucket:      aws.String("vadmeste"),                                                                                                                                                                 
                Key:         aws.String("object"),                                                                                                                                                                   
                ContentType: aws.String("application/octet-stream"),                                                                                                                                                 
        })                                                                                                                                                                                                           

        req.HTTPRequest.Header.Set("Content-Md5", "82bb413746aee42f89dea2b59614f9ef")                                                                                                                                
        url, err := req.Presign(24 * time.Hour)                                                                                                                                                                      
        if err != nil {                                                                                                                                                                                              
                panic(err)                                                                                                                                                                                           
        }                                                                                                                                                                                                            

        rreq, err := http.NewRequest("PUT", url, bytes.NewReader([]byte("")))                                                                                                                                        
        rreq.Header.Add("Content-Type", "application/octet-stream")                                                                                                                                                  
        rreq.Header.Set("Content-Md5", "82bb413746aee42f89dea2b59614f9ef")                                                                                                                                           
        log.Println("REQUEST PUT ", url)                                                                                                                                                                             
        for k, v := range rreq.Header {                                                                                                                                                                              
                log.Printf("%v: %v", k, v)                                                                                                                                                                           
        }                                                                                                                                                                                                            

        resp, err := http.DefaultClient.Do(rreq)                                                                                                                                                                     
        if err != nil {                                                                                                                                                                                              
                panic(err)                                                                                                                                                                                           
        }                                                                                                                                                                                                            
        body, err := ioutil.ReadAll(resp.Body)                                                                                                                                                                       
        resp.Body.Close()                                                                                                                                                                                            
        if err != nil {                                                                                                                                                                                              
                panic(err)                                                                                                                                                                                           
        }                                                                                                                                                                                                            
        log.Println("AMZ RESPONSE")                                                                                                                                                                                  
        for k, v := range resp.Header {                                                                                                                                                                              
                log.Printf("%v: %v", k, v)                                                                                                                                                                           
        }                                                                                                                                                                                                            
        log.Printf("\n%v", string(body))      
}

$ go run put-object-presign-content-type.go
2017/11/14 10:14:17 REQUEST PUT  https://vadmeste.s3.amazonaws.com/object?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJY2EP7JTV2SUOH4Q%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171114T181417Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost&X-Amz-Signature=1c1dec7ce9531106bd437d8850589182ff9ae3b561d654b43133a732130d73e4
2017/11/14 10:14:17 Content-Md5: [82bb413746aee42f89dea2b59614f9ef]
2017/11/14 10:14:17 Content-Type: [application/octet-stream]
2017/11/14 10:14:18 AMZ RESPONSE
2017/11/14 10:14:18 Server: [AmazonS3]
2017/11/14 10:14:18 X-Amz-Request-Id: [2F8B8FA82C2BBECB]
2017/11/14 10:14:18 X-Amz-Id-2: [C8t+C+nOXAaALT+uFgarF6q+ALJJUP8Xz1l0PPCK92vtHWdZRvZd+dwGzd5DZcywS24DTNVwyg0=]
2017/11/14 10:14:18 Content-Type: [application/xml]
2017/11/14 10:14:18 Date: [Tue, 14 Nov 2017 18:14:18 GMT]
2017/11/14 10:14:18 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidDigest</Code><Message>The Content-MD5 you specified was invalid.</Message><Content-MD5>82bb413746aee42f89dea2b59614f9ef</Content-MD5><RequestId>2F8B8FA82C2BBECB</RequestId><HostId>C8t+C+nOXAaALT+uFgarF6q+ALJJUP8Xz1l0PPCK92vtHWdZRvZd+dwGzd5DZcywS24DTNVwyg0=</HostId></Error>

Resolution (Updated)

aws-sdk-php mint test This test does capture a valid yet uncommon use-case where the presigned URL provider wishes to control the content to match the given sha256 sum. But aws-sdk-php doesn't include the x-amz-content-sha256 in signed headers. i.e, not including X-Amz-Content-Sha256 in presigned request header doesn't fail with SignatureDoesNotMatch. I.e, when this test is run on Minio server/gateway it would return different error code than when run on AWS S3. So, it makes sense to add an equivalent test using aws-sdk-go and remove this test.
minio-server Minio server should return ErrContentSHA256Mismatch if an invalid hex value is set in X-Amz-Content-Sha256 header.

This PR https://github.com/minio/mint/pull/222 is needed as well

minio / mint

aws-sdk-php tests fail for server/gateway #216

Findings from investigation

Resolution (Updated)