$ time xz -cT 0 newton.tar | wc -c
1453954588 // 1.35GiB
real 6m33.006s
user 13m0.724s
sys 0m1.608s
$ time gzip -c9 newton.tar | wc -c
1701486703 // 1.58GiB
real 1m16.832s
user 1m15.556s
sys 0m2.144s
$ time pigz -c9 newton.tar | wc -c
1702539244 // 1.59GiB
real 0m50.708s
user 1m39.512s
sys 0m1.488s
$ time pigz -c newton.tar | wc -c
1703702154 // 1.59GiB
real 0m44.796s
user 1m27.860s
sys 0m1.480s
pigz with default settings it is. There is enough bandwidth to S3 to upload at >1Gb/s - takes 15s to upload 1.9GiB. Not the bottleneck. Cost of storing 2G in S3 is <$.05/month
pigz
with default settings it is. There is enough bandwidth to S3 to upload at >1Gb/s - takes 15s to upload 1.9GiB. Not the bottleneck. Cost of storing 2G in S3 is <$.05/month