search

thenickdude / snap-to-s3

Upload EBS volume snapshots to Amazon S3/Glacier
Other
114 stars 24 forks source link

Any noted limitations on size of EBS Snapshots crossing which can't be copied. #11

Closed santoshkatageri closed 4 years ago

santoshkatageri commented 4 years ago

Hey @thenickdude ,

Thank you so much for creating this tool. This made my half work done so easily.

I came across multiple sizes of snapshots from 8, 10, 50, 100, 200, 350, 500, 850, 960 and 1000 GiBs.

It is working fine for till size of 500 GiBs but when I tried 850 GiB snapshots,

I got a response like below

`[snap-0xxxxxxxx3] Migrating snap-xxxxxxxxf3 to S3 [snap-0xxxxxxxx3] Tagging snapshot with "migrating"... [snap-0xxxxxxxx3] A temporary volume for snap-xxxxxxxx3 already exists, using vol-0xxxxxxxxb1 [snap-0xxxxxxxx3] Attaching vol-0xxxxxxxx1 to this instance (i-0exxxxxxxx2f) at /dev/sdx... [snap-0xxxxxxxx3] Waiting for vol-02xxxxxxxxb1's partitions to become visible to the operating system... [snap-0xxxxxxxx3] 1 partition to upload [snap-0xxxxxxxx3] [snap-0xxxxxxxx3] Uploading partition 1 of 1... [snap-0xxxxxxxx3] Mounting /dev/xvdx at /mnt/snap-0xxxxxxxx3... [snap-0xxxxxxxx3] Computing size of files to upload... [snap-0xxxxxxxx3] 28 KB to compress and upload to s3://xxxxxxxx/vol-0fdc1xxxxxxxx19-09-10T11:21:48+00:00 snap-0xxxxxxxx3 - Created for policy: policy-0xxxxxxxxec7 schedule: Default Schedule.tar.lz4 [snap-0xxxxxxxx3] Progress is based on the pre-compression data size:

[snap-0xxxxxxxx3] Uploaded partition to S3 successfully! [snap-0xxxxxxxx3] Unmounting partition... [snap-0xxxxxxxx3] [snap-0xxxxxxxx3] Detaching vol-02xxxxxxxxb1 [snap-0xxxxxxxx3] Deleting temporary volume vol-xxxxxxxx1 [snap-0xxxxxxxx3] Tagging snapshot with "migrated" [snap-0xxxxxxxx3] Successfully migrated this snapshot!`

Let me know if the issue is caused by my snapshots

thenickdude commented 4 years ago

The only threshold is that at 1024GiB, it switches to using "gp2" volumes instead of "standard" ones (because standard EBS volumes top out at 1024GiB).

Run it with "--keep-temp-volumes" and investigate the volume - mount it and see what's on there to get a clue as to why tar might be choking on it (snap-to-s3 just chdirs into the mountpoint and then calls tar with very straightforward arguments: tar -c . ).

My bet is that your snapshot is of a 850GiB disk - but that disk was empty when the snapshot was taken, so there is only 28KB of files to upload.

santoshkatageri commented 4 years ago

That worked for me and I can see that there is no data in the disk/volume attached with the snapshot.

santoshkatageri commented 4 years ago

This issue can be closed.

thenickdude commented 4 years ago

Sweet, thanks for checking it out