search

gholt / swiftly

Client for Swift - NOTE: The master branch is current dev work. Depending on how adventurous you are, you might want to switch to the "stable" branch if you're running from a git clone.
http://gholt.github.io/swiftly/
Apache License 2.0
48 stars 27 forks source link

Make segments of segmented object get the extra headers. #40

Closed gholt closed 10 years ago

gholt commented 10 years ago

Specifically x-delete-at/x-delete-after but probably just all of them.

mikemar10 commented 10 years ago

+1

gholt commented 10 years ago

This seems to work already, at least on master which will become release 2.00 soon.

Dynamic Segmented Object:

$ dd if=/dev/urandom bs=524288 count=5 of=2.5m
5+0 records in
5+0 records out
2621440 bytes (2.6 MB) copied, 0.235589 s, 11.1 MB/s

$ swiftly -v put test/object -s 1048576 -hx-delete-after:86400 -i 2.5m
VERBOSE 0.01 1 Read auth response values from cache '/tmp/<redacted>.swiftly'.
VERBOSE 0.01 1 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.02 1 > PUT /v1/MossoCloudFS_<redacted>/test_segments Transfer-Encoding: chunked  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>
VERBOSE 0.05 1 < 201 Created
VERBOSE 0.05 1 > PUT /v1/MossoCloudFS_<redacted>/test_segments/object/1387588694.0/2621440/00000000 Content-Length: 1048576  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400
VERBOSE 0.05 2 Read auth response values from cache '/tmp/<redacted>.swiftly'.
VERBOSE 0.05 2 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.05 3 Read auth response values from cache '/tmp/<redacted>.swiftly'.
VERBOSE 0.05 3 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.06 3 > PUT /v1/MossoCloudFS_<redacted>/test_segments/object/1387588694.0/2621440/00000002 Content-Length: 524288  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400
VERBOSE 0.07 2 > PUT /v1/MossoCloudFS_<redacted>/test_segments/object/1387588694.0/2621440/00000001 Content-Length: 1048576  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400
VERBOSE 0.25 1 < 201 Created
VERBOSE 0.28 2 < 201 Created
VERBOSE 0.38 3 < 201 Created
VERBOSE 0.38 1 > PUT /v1/MossoCloudFS_<redacted>/test/object Content-Length: 0  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400  X-Object-Manifest: test_segments/object/1387588694.0/2621440/  X-Object-Meta-Mtime: 1387588694.000000
VERBOSE 0.48 1 < 201 Created

$ swiftly head test/object | grep ^X-Delete-At:
X-Delete-At:         1387675120

$ swiftly for test_segments do head '<item>' | grep ^X-Delete-At:
X-Delete-At:    1387675120
X-Delete-At:    1387675120
X-Delete-At:    1387675120

Static Segmented Object:

$ dd if=/dev/urandom bs=524288 count=5 of=2.5m
5+0 records in
5+0 records out
2621440 bytes (2.6 MB) copied, 0.235589 s, 11.1 MB/s

$ swiftly -v put test/object -s s1048576 -hx-delete-after:86400 -i 2.5m
VERBOSE 0.01 1 Read auth response values from cache '/tmp/<redacted>.swiftly'.
VERBOSE 0.01 1 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.02 1 > PUT /v1/MossoCloudFS_<redacted>/test_segments Transfer-Encoding: chunked  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>
VERBOSE 0.04 1 < 201 Created
VERBOSE 0.04 1 > PUT /v1/MossoCloudFS_<redacted>/test_segments/object/1387588694.0/2621440/00000000 Content-Length: 1048576  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400
VERBOSE 0.05 2 Read auth response values from cache '/tmp/<redacted>.swiftly'.
VERBOSE 0.05 2 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.05 3 Read auth response values from cache '/tmp/<redacted>.swiftly'.
VERBOSE 0.05 3 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.07 2 > PUT /v1/MossoCloudFS_<redacted>/test_segments/object/1387588694.0/2621440/00000001 Content-Length: 1048576  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400
VERBOSE 0.09 3 > PUT /v1/MossoCloudFS_<redacted>/test_segments/object/1387588694.0/2621440/00000002 Content-Length: 524288  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400
VERBOSE 0.20 1 < 201 Created
VERBOSE 0.28 2 < 201 Created
VERBOSE 0.30 3 < 201 Created
VERBOSE 0.30 1 > PUT /v1/MossoCloudFS_<redacted>/test/object?multipart-manifest=put Content-Length: 395  User-Agent: Swiftly v1.99  X-Auth-Token: <redacted>  X-Delete-After: 86400  X-Object-Meta-Mtime: 1387588694.000000
VERBOSE 0.46 1 < 201 Created

$ swiftly head test/object | grep ^X-Delete-At:
X-Delete-At:           1387675715

$ swiftly for test_segments do head '<item>' | grep ^X-Delete-At:
X-Delete-At:    1387675714
X-Delete-At:    1387675714
X-Delete-At:    1387675714

I also tested with X-Delete-At in place of X-Delete-After and it worked fine too.