Signature v4 on a multipart upload

[ctippur]

Hello.

I am glad to see this plugin. Thanks again. This is more my ignorance question.

I am trying to follow this https://github.com/arnemolland/sigv4/blob/master/example/example.dart I am trying to post an video file.

Here is what I get from s3 pre-signed url request: {fields: {AWSAccessKeyId: <>, key: <>, policy:<POLICY>, signature: <>}}

I am constructing my request object like this:

      http.MultipartFile multipartFile = await http.MultipartFile.fromPath('image_file', filepath);
      request.files.add(multipartFile);
      request.fields['key'] = signed_url['fields']['key'];
      request.fields['AWSAccessKeyId'] = signed_url['fields']['AWSAccessKeyId'];
      request.fields['policy'] = signed_url['fields']['policy'];  
      request.fields['signature'] = signed_url['fields']['signature'];
      request.fields['file'] = filepath;

from the example:

final client = Sigv4Client(
    accessKey: 'your_access_key',
    secretKey: 'your_secret_key',
  );

I am following this doc to generate the url - https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html

I have couple of questions:

1. I can see how to get the access key but I dont see a secret key. Is it part of the policy? what field represents the secretKey?
2. the client as defined on the example, comes with a error The named parameter 'accessKey' isn't defined.

Appreciate a response. I have been working on this issue for the past couple of weeks.

Shekar

[ctippur]

I tried using the constructor this way:

final client = Sigv4Client(
        signed_url['fields']['AWSAccessKeyId'],
        signed_url['fields']['key'],
        signed_url['url']
      );

but when I try ```
// A larger request final request = client.request( 'https://service.aws.com/endpoint', method: 'POST', queryParameters: {'key': 'value'}, headers: {'header': 'value'}, body: {'content': 'some-content'}, );

I get an error saying ``` The setter 'request' isn't defined for the class 'Sigv4Client'. ```

[arnemolland]

Hi @ctippur 👋

If anything below is unclear, check out the AWS documentation.

The accessKey parameter refers to the Access Key ID, and the secretKey parameter refers to the secret access key itself. I'll update the API to name these more clearly.

Both the accessKey and secretKey parameters are required. The example provided does not run without providing your own accessKey, secretKey and endpoint. The existing values are placeholders, and the service.aws.com/endpoint path is purely fictional. I'll update the docs to make this clearer.

I'm unsure about the last bit though. It looks like you're trying to set a request field in the client, but there's only a request method. Do you have a complete example of this part?

[ctippur]

@arnemolland thanks for responding. Sorry about the delay. My bad on the request. I was pulling the wrong version.

Now that the request method is sorted, can you please give a little more clarity on the large request object

final largeRequest = client.request(
    path,
    method: 'POST',
    query: {'key': 'value'},
    headers: {'header': 'value'},
    body: {'content': 'some-content'},
  );

  // POST request
  post(largeRequest.url, headers: largeRequest.headers, body: largeRequest.body);

1. path here refers to the path to the local file?
2. query - can you please give a sample for what should go here
3. header - I am guessing we can add the Content-Type here
4. body - what should go here?

Thanks again.

Shekar

[ctippur]

Just to be a little more explicit -

The objective is to upload an image or a video file using flutter.

I have presigned URL object via generate_presigned_post call. The object i have looks like this:

{fields: {AWSAccessKeyId: <>, key: <>, policy:<POLICY>, signature: <>}}

I am struggling to unravel the right mapping to do a POST call.

I have created a client

final  client = Sigv4Client(
      accessKey: accessKeyId, 
      secretKey: _secretKeyId, 
      region: _region,
      );

// Create a local file
var lf1 = await http.MultipartFile.fromPath('image_file', filepath);
var largeRequest;
try{
 largeRequest = client.request(
      _s3Endpoint,
      method: 'POST',
      headers: {'Content-Type': 'Application/octet-stream'},
      body: {'file': lf1}
    );
} catch (e) {
  print ("Large request error: " + e.toString());
}

  // POST request
  try{
    var response = await http.post(largeRequest.url, headers: largeRequest.headers, body: largeRequest.body);
    print (response.body);
  }catch (e){
    print (e.toString());
  }

I get an error

flutter: Large request error: Converting object to an encodable object failed: Instance of 'MultipartFile'

I also tried using

try{
 largeRequest = client.request(
      _s3Endpoint,
      method: 'POST',
      headers: {'Content-Type': 'Application/octet-stream'},
      body: {'file': File(filepath)}
    );
} catch (e) {
  print ("Large request error: " + e.toString());
}

I get the same error.

Also, if I try just the file path, I get a different error - Invalid Argument(s)

try {
      largeRequest = client.request(
        _s3Endpoint,
        method: 'POST',
        headers: {'Content-Type': 'Application/octet-stream'},
        body: {'file': filepath}
      );
    }catch (e) {
      print ("Large request error " + e.toString());
    }

[arnemolland]

Thanks for the details @ctippur

To answer your questions:

1. path refers to the absolute path of your endpoint, e.g. https://service.com/api/list
2. query is a map of query parameters, and is optional
3. headers is the request headers and is optional as well. The Content-Type header defaults to either what you set as default when instantiating your Sigv4Client with the parameter defaultContentType, which again defaults to application/json if omitted
4. body (or payload) is the request message body, and is optional. This is often used with POST or PATCH requests.

For your error, you're providing some object to the body field. This object has to be encodable, which means that its fields need to be directly encodable to a JSON string. These types are encodable:

• number
• boolean
• string
• null
• list
• map with string keys

If your object is non-encodable by default, you can encode it "manually" using dart:convert:

// This is just an example and will not work in your case
final encoded = jsonEncode(value, toEncodable: (value) => value.toString());

If toEncodable is omitted, the JSON encoder tries calling a toJson() method on the object. If none exists, it will throw an error, which is what you're getting.

MultiPartFile is not encodable and does not expose a toJson() method. To solve this, you could pass the data as a byte array (Uint8List is encodable, as it is a list):

// MultiPartFile => Uint8List
final bytes = await file.finalize().toBytes()
...
largeRequest = client.request(
  _s3Endpoint,
  method: 'POST',
  headers: {'Content-Type': 'Application/octet-stream'},
  body: bytes,
);

[ctippur]

@arnemolland

Thanks for the explanation

I followed your instructions.

This is what I have:

final bytes = await lf1.finalize().toBytes();
final  client = Sigv4Client(
      accessKey: accessKeyId, 
      secretKey: _secretKeyId, 
      region: _region,
      );

var largeRequest;
    try {
      largeRequest = client.request(
        _s3Endpoint,
        method: 'POST',
        headers: {'Content-Type': 'Application/octet-stream'},
        body: bytes,
      );
    }catch (e, s) {
      print ("Large request error " + e.toString());
      print ("Large request stack " + s.toString());
    }

Here is the error:

flutter: Large request error Invalid argument(s)
flutter: Large request stack #0      _StringBase.+  (dart:core-patch/string_patch.dart:263:57)
#1      Sigv4.buildAuthorizationHeader 
package:sigv4/src/sigv4.dart:133
#2      Sigv4Client._generateAuthorization 
package:sigv4/src/client.dart:198
#3      Sigv4Client.signedHeaders 
package:sigv4/src/client.dart:116
#4      Sigv4Client.request 
package:sigv4/src/client.dart:149
#5      TestsBloc.update_signed_url 
package:helloworld/…/testsBloc/tests_bloc.dart:361
<asynchronous suspension>

[ctippur]

Something is fishy. I am now getting a different error:

flutter:  SocketException: OS Error: Connection reset by peer, errno = 54, address =my s3 bucket, port = 56079

[arnemolland]

For the first issue, it seems to me like there is an invalid Access Key ID, Secret Access Key or region passed to the client. I’ll improve the error handling for this case.

For the OSException, it seems like you’re making a request to my s3 bucket which is an invalid URL and will not reach a server.

[ctippur]

my s3 bucket is a url of the form https://<my bucket>.s3.amazonaws.com/ You were right on the first issue. I was referencing a wrong accessKeyId

I am now correctly referencing signed_url['fields']['AWSAccessKeyId`]

[ctippur]

Here is the tcpdump from the transaction

23:13:58.283118 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R.], seq 4177, ack 21788, win 287, length 0
23:13:58.284111 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762353556, win 0, length 0
23:13:58.284117 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762353880, win 0, length 0
23:13:58.284118 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762354302, win 0, length 0
23:13:58.284120 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762354333, win 0, length 0
23:13:58.284121 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762354333, win 0, length 0
23:13:58.284122 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762354333, win 0, length 0
23:13:58.284123 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq 3762354333, win 0, length 0
As we can see the server is sending a reset back :/

[ctippur]

Meanwhile, I am trying a put variant of the presigned url Put variant gives a URL

 https://My_bucket.s3.amazonaws.com/?AWSAccessKeyId=<ACCESS_KEY>&Signature=<Signature>&x-amz-security-token=<SEC_TOKEN>&Expires=<Expires>

Here is my code for that:

// Create a client
final client = Sigv4Client(
      secretKey: uri.queryParameters['Signature'],
      accessKey: uri.queryParameters['AWSAccessKeyId'],
      defaultContentType: 'application/octet-stream'
    );

    var lf1 = await http.MultipartFile.fromPath('video_file', filepath);

    final bytes = await lf1.finalize().toBytes();

   // Create a request object
    final request = client.request(
      'https://' + uri.host + '/' + uri.path,
      method: 'PUT',
      //query: {},
      headers: {'Content-Type': 'application/octet-stream', 'x-amz-content-sha256': "UNSIGNED-PAYLOAD"},
      body: {'file': bytes},
    );

   // Make a http put request
   try {

      var response = await http.put(request.url, headers: request.headers, body: request.body);
      print (response.body);

    }catch (e,s){

      print ("**** Error **** " + e.toString());
      print ("**** Stack **** " + s.toString());

    }

Error I am getting is

<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; incorrect service "execute-api". This endpoint belongs to "s3".</Message><RequestId>010975D5EE84ED8B</RequestId><HostId>Host id redacted</HostId></Error>

I am guessing I should be encouraged by this error.

[arnemolland]

I should’ve spotted this earlier. The service parameter of your client should be set to s3. It defaults to execute-api, which is why S3 is closing the connection.

I’ll update the constructor to make it a required parameter without a default value.

[ctippur]

Aha. Thx

Changed that. Now a different error: This is with the PUT Call

<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>REDACTED</AWSAccessKeyId><RequestId>EA78B8A76190CAA9</RequestId><HostId>REDACTED</HostId></Error>

[ctippur]

Meanwhile, Let me switch the call to POST and make the corresponding service change as well and try.

[ctippur]

Just tested POST after making the changes to client

final  client = Sigv4Client(
      accessKey: _accessKeyId, 
      secretKey: _secretKeyId, 
      region: _region,
      defaultContentType: "application/octet-stream",
      serviceName: 's3'
      );

I am still getting the same error:

SocketException: OS Error: Connection reset by peer, errno = 54, address = <MY bucket>, port = 50301

[ctippur]

I should’ve spotted this earlier. The service parameter of your client should be set to s3. It defaults to execute-api, which is why S3 is closing the connection.

I’ll update the constructor to make it a required parameter without a default value.

I agree..

[ctippur]

Could this be a content type issue? for a post request, the content type should be multipart/form-data.

I tried changing the POST call code to

final  client = Sigv4Client(
      accessKey: _accessKeyId, 
      secretKey: _secretKeyId, 
      region: _region,
      defaultContentType: 'multipart/form-data',
      serviceName: 's3'
      );

    var largeRequest;
    try {
      largeRequest = client.request(
        _s3Endpoint,
        method: 'POST',
        body: bytes,
      );
    }catch (e, s) {
      print ("Large request error " + e.toString());
      print ("Large request stack " + s.toString());
    }

Got the same error

[arnemolland]

Since you're making a multipart request, you should try using MultipartRequest, as the request() method only returns a regular Request object.

[ctippur]

To keep this simple, would something like this work?

var config = File('config.txt');
var bytes = await config.readAsBytes();

And use the body as bytes?

[ctippur]

Since you're making a multipart request, you should try using MultipartRequest, as the request() method only returns a regular Request object.

Bit confused with this.

Looking at MultipartRequest, the request object is a http.MultipartRequest type.

I dont need it to be MultipartRequest, I can make do with a simpler solution.

[arnemolland]

Yes, any encodable primitive or object can be used in the body. It’s up to you what request classes you’d like to use, the only thing you need from this library is the signed request headers you can get using the signedHeaders() method. The request() method is just a wrapper that puts them in a Request object along with a canonical URI.

[ctippur]

Yes, any encodable primitive or object can be used in the body. It’s up to you what request classes you’d like to use, the only thing you need from this library is the signed request headers you can get using the getSignedHeaders() method. The request() method is just a wrapper that puts them in a Request object along with a canonical URI.

Possible to give an example? client object doesnt seem to have getSignedHeaders() method.

This is what I have so far:

   var uri_s3 = Uri.parse('https://' + uri.host + '/' + uri.path,);

    var request = new http.MultipartRequest("PUT", uri_s3);

    request.files.add(new http.MultipartFile.fromBytes(
    'Video',
    bytes,
    contentType: new MediaType('application', 'octet-stream')));

   final headers = client.signedHeaders(
      'https://' + uri.host + '/' + uri.path,
    );

    try{
      request.headers.addAll(headers);
    }catch (e) {
      print (e.toString());
    }

At this point, I am getting an exception:

flutter: type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Map<String, String>'

[arnemolland]

I had a typo, sorry, it's signedHeaders(). The error message you're getting is pretty descriptive, as you're probably passing some dynamic value where it's expecting a <String, String> map. An example of how it might look:

  final file = MultipartFile.fromString('field', 'value');
  final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
  final bytes = await file.finalize().toBytes();

  final signed = client.signedHeaders(path, method: 'PUT', body: bytes);

  multipart.headers.addAll(signed);

  final request = await multipart.send();

[ctippur]

I had a typo, sorry, it's signedHeaders(). The error message you're getting is pretty descriptive, as you're probably passing some dynamic value where it's expecting a <String, String> map. An example of how it might look:

  final file = MultipartFile.fromString('field', 'value');
  final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
  final bytes = await file.finalize().toBytes();

  final signed = client.signedHeaders(path, method: 'PUT', body: bytes);

  multipart.headers.addAll(signed);

  final request = await multipart.send();

Getting into some syntax errors at final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);

MultipartFile file
The argument type 'MultipartFile (where MultipartFile is defined in /Users/shekar/playground/flutter/flutter/.pub-cache/hosted/pub.dartlang.org/dio-3.0.7/lib/src/multipart_file.dart)' can't be assigned to the parameter type 'MultipartFile (where MultipartFile is defined in /Users/shekar/playground/flutter/flutter/.pub-cache/hosted/pub.dartlang.org/http-0.12.0+2/lib/src/multipart_file.dart)'.dart

[arnemolland]

You’re importing both dio and http which both expose a MultiPartFile. Add a prefix to resolve any ambiguity.

import ‘package:http/http.dart’ as http;

var file = http.MultiPartFile(...)

[ctippur]

I had a typo, sorry, it's signedHeaders(). The error message you're getting is pretty descriptive, as you're probably passing some dynamic value where it's expecting a <String, String> map. An example of how it might look:

  final file = MultipartFile.fromString('field', 'value');
  final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
  final bytes = await file.finalize().toBytes();

  final signed = client.signedHeaders(path, method: 'PUT', body: bytes);

  multipart.headers.addAll(signed);

  final request = await multipart.send();

My apologies about the syntax error. I tried this and I still get the same error:

type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Map<String, String>'

Map<String, dynamic> is the right data type for this case as the file contents are in bytes. Not sure how to deal with this. I dont think casting signed to Map<String,String> would be correct.

What is the right way to deal with this?

While declaring signed, do we need to have body: bytes? We are already referencing it final multipart = http.MultipartRequest('PUT', uri_s3)..files.add(file);

[arnemolland]

Are you om the latest version of sigv4? signedHeaders() should return a <String, String> map.

HTTP headers always contain string values only, which is why the Request object does not allow any dynamic values. If you want to include a non-encodable object, you need to convert is to a JSON-string, e.g. using jsonEncode from the dart:convert library.

The AWS Signature that this package helps generate is purely headers that contain information about the request itself. The signature is used to check the integrity of the request data. This means that you need to include your other request data like body, headers and query parameters when generating these signed headers.

You can then add these headers to a request, however the request itself must match the request described when generating the signed headers. If you either add or remove some data, you’ll break the integrity of the request, the signature will not match and AWS will reject it.

This is why you will need to include your bytes value both in the request itself and when generating the signed headers. If not, the signature will not match the request.

Here’s a link to the AWS Signature v4 documentation if anything of this was unclear.

[ctippur]

I tried this. Tried to avoid body: bytes from signed

    var url_s3 = 'https://' + uri.host  + uri.path + '/';
    var uri_s3 = Uri.parse('https://' + uri.host + '/' + uri.path,);

    final file = await http.MultipartFile.fromString('videofile',filepath);
    final multipart = http.MultipartRequest('PUT', uri_s3)..files.add(file);
    var bytes;
    try{
      bytes = await file.finalize().toBytes();
    }catch (e){
      print ("**** Exception in getting bytes **** " + e.toString());
    }

    var signed;
    try {
      signed = client.signedHeaders(url_s3, method: 'PUT');
    } catch (e) {
      print ("*** Exception in signing *** " + e.toString());
    }

    print ("**** Add headers to to multipart ****" );
    Map<String,String> signed_map;

    try {
      multipart.headers.addAll(signed);
    }catch (e) {
      print ("*** Exception in adding headers *** " + e.toString());
    }

    print ("**** Send request ****" );

    try{
      final request = await multipart.send();
      //print (request.statusCode);
    }catch (e){
      print ("**** EXCEPTION ON PUSING MULTIPARTREQUEST " + e.toString());
    }

The exception I get is Unhandled Exception: Bad state: Can't finalize a finalized MultipartFile.

[ctippur]

Are you om the latest version of sigv4? signedHeaders() should return a <String, String> map.

HTTP headers always contain string values only, which is why the Request object does not allow any dynamic values. If you want to include a non-encodable object, you need to convert is to a JSON-string, e.g. using jsonEncode from the dart:convert library.

The AWS Signature that this package helps generate is purely headers that contain information about the request itself. The signature is used to check the integrity of the request data. This means that you need to include your other request data like body, headers and query parameters when generating these signed headers.

You can then add these headers to a request, however the request itself must match the request described when generating the signed headers. If you either add or remove some data, you’ll break the integrity of the request, the signature will not match and AWS will reject it.

This is why you will need to include your bytes value both in the request itself and when generating the signed headers. If not, the signature will not match the request.

Here’s a link to the AWS Signature v4 documentation if anything of this was unclear.

Thanks for the detailed explanation. I do have the latest version. With all the changes, I think the last straw in the puzzle maybe to resolve the exception Unhandled Exception: Bad state: Can't finalize a finalized MultipartFile.

[arnemolland]

You could check it before trying to finalize:

var bytes;

if(!file.finalized) {
  bytes = await file.finalize().toBytes();
} else {
  bytes = file.toBytes();
}

I’m on my phone, so take this as pseudocode.

[ctippur]

Thanks @arnemolland. bytes = file.toBytes(); wont work as it is a MultipartFile

[ctippur]

tried to simplify it by taking away multipartfile.

   final client = Sigv4Client(
      keyId: uri.queryParameters['Signature'],
      accessKey: uri.queryParameters['AWSAccessKeyId'],
      defaultContentType: 'application/octet-stream',
      region: 'ap-south-1',
      serviceName: 's3',
    );
  final bytes = File(filepath).readAsBytesSync();
  final  signed = client.signedHeaders(signed_url, method: 'PUT', body: bytes);
  signed.addAll({'x-amz-content-sha256': "UNSIGNED-PAYLOAD"});
  final simpleput = await http.put(
      url_s3,
      headers: signed,
      body: bytes
      );

    print (simpleput.body);

The error I get is

<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the authorization component "Credential=SIGNATURE/20191217/ap-south-1/s3/aws4_request" is malformed.</Message><RequestId>B7F60499DDF13FF3</RequestId><HostId>Host id</HostId></Error>

[ctippur]

I am not sure if I am using keyId correctly.

keyId: uri.queryParameters['Signature'],

Referring back to the signed url structure -

https://My_bucket.s3.amazonaws.com/?AWSAccessKeyId=<ACCESS_KEY>&Signature=<Signature>&x-amz-security-token=<SEC_TOKEN>&Expires=<Expires>

there is no key referred here.

Since keyId is a mandatory field, not sure what to refer here.

final client = Sigv4Client(
      keyId: uri.queryParameters['Signature'],
      accessKey: uri.queryParameters['AWSAccessKeyId'],
      defaultContentType: 'application/octet-stream',
      region: 'ap-south-1',
      serviceName: 's3',
    );

[ctippur]

Just for completion, here is the code I have to generate signed url

s3_client = boto3.client('s3')
        try:
            response = s3_client.generate_presigned_url('put_object',
                                                        Params={'Bucket': bucket_name,
                                                                'Key': object_name},
                                                        ExpiresIn=expiration)
        except ClientError as e:
            logging.error("In client error exception code")
            logging.error(e)
            return None

        # The response contains the presigned URL
        return response

[ctippur]

@arnemolland I got more clarity on this. For .a put call, the code from the server side should generate a url that has all the components in it. As we are using presigned url, the ensuing url can be used as is without any modifications. Generation of SignatureV4 locally is needed for aws api access from other sources. I am sorry to have taken you through this path but it was a learning curve for me. I was able to resolve this issue after adding some additional parameters to the server code that renders the url. Please refer to the stackoverflow link for details. It does expose the documentation deficiencies around this topic. For example, boto3 server code does not do a good job in explaining the v4 signature needs and how to generate a url that can be consumed.

[arnemolland]

Great! The important thing is that you’ve solved it and gained experience.