Copy v0.2 training data from S3 into EBS storage

[yellowcap]

For training we are copying all data from S3 into a EBS volume. This will have to change for v1 but is still doable for v0.2.

[yellowcap]

This was some challenge as the downloads are slow for the 10M files and 20TB of data we have. Found a clunky way to speed up

https://gist.github.com/yellowcap/c39a10d19d78833b29a2c4828b47b2ea

[chuckwondo]

@yellowcap, you might want to take a look at s5cmd

[yellowcap]

@yellowcap, you might want to take a look at s5cmd

Oh that would have been helpful indeed! thanks @chuckwondo ! Next time I'll use that for sure.

The clunky transfer script worked too, so now we have 16TB of training data on EBS.