search

timkay / aws

Easy command line access to Amazon EC2, S3, SQS, ELB, and SDB
http://timkay.com/aws/
426 stars 126 forks source link

How to attach metadata when put to S3? #95

Closed phanan closed 9 years ago

phanan commented 9 years ago

According to S3 put-object documentation, we can attach custom metadata to an object with

--metadata key_name=string,key_name2=string

Is this something aws supports? I've try --metadata, --meta-data, --meta, and --data, all failed miserably with the mispelled meta parameter error.

timkay commented 9 years ago

The documentation http://goo.gl/f1d0QB shows that the --metadata parameters get turned into headers like these (to use the same data as in your example).

x-amz-meta-key_name=string
x-amz-meta-key_name2=string

"aws" allows you to specify arbitrary headers by simply adding them to the command line:

aws put 'x-amz-meta-key_name:string' 'X-AMZ-Meta-key_name2: String' test683 hello.txt

Your example exposed a bug in the code that sorts headers. Please pull the latest version.

I could add a --metadata parameter if you need it for compatibility. Let me know.

phanan commented 9 years ago

This is the most amazing level of support I've ever seen on Github! :+1:

Yes, if you could introduce --metadata, it would be great! I would be happy to contribute, if my Perl knowledge wasn't close to zéro ;)

phanan commented 9 years ago

Hi, any news regarding this?

timkay commented 9 years ago

Ahh sorry! I've been busy with my startup and I didn't get to it. It wasn't too hard. Pull the latest and see if it helps.

Let me know if I can be of any other help!

phanan commented 9 years ago

Thanks. Unfortunately specifying --metadata didn't work for me. The file was uploaded without any error, but without the metadata attached.

Also it's weird that, when aws put 'x-amz-meta-key_name:string' sort of works, the meta key appears as x-amz-meta-key_name as a whole, not key_name as it should be (I think). Any thoughts?

timkay commented 9 years ago

You can use --metadata to add both standard metadata, such as a Content-Type header and also user defined metadata. The latter must start with x-amz-meta-. Thus, you can say

$ aws put test683/ aws --metadata Content-Type=text/plain,x-amz-meta-Tim=Kay
$ aws head test683/aws
HTTP/1.1 200 OK
x-amz-id-2: cWxL30uczXaF7xFEOL0M972BaWj59wkFANX7js8dG1BnJDDWmdOk+sXjdb5rdQVT9+snh79eiB0=
x-amz-request-id: 5F13D73E868FDDD3
Date: Wed, 20 May 2015 13:56:21 GMT
x-amz-meta-tim: Kay
Last-Modified: Wed, 20 May 2015 13:54:38 GMT
ETag: "289f25fb859276e0e62a155bf09dd928"
Accept-Ranges: bytes
Content-Type: text/plain
Content-Length: 119531
Server: AmazonS3

When you look in the S3 console, you'll see both items, and the x-amz-meta- prefix is preserved. I was wondering about the x-amz-meta- part of the metadata. From the S3 docs, it seems that the x-amz-meta- prefix will be there when using REST (which "aws" does), and that the SOAP interface is being retired slowly.

User-Defined Metadata

When uploading an object, you can also assign metadata to the object. You provide this optional information as a name-value pair when you send a PUT or POST request to create the object. When uploading objects using the REST API the optional user-defined metadata names must begin with "x-amz-meta-" to distinguish them from other HTTP headers. When you retrieve the object using the REST API, this prefix is returned. When uploading objects using the SOAP API, the prefix is not required. When you retrieve the object using the SOAP API, the prefix is removed, regardless of which API you used to upload the object.

Note SOAP support over HTTP is deprecated, but it is still available over HTTPS. New Amazon S3 features will not be supported for SOAP. We recommend that you use either the REST API or the AWS SDKs.

phanan commented 9 years ago

Ah, got it. Thanks! :+1: