Open laughedelic opened 8 years ago
Done! :shipit: I've checked that all files are in S3. Didn't perform and DB tests though. I'm going to use this issue to modify the docs later.
The problem is that you cannot use aws s3api get-object
to download an S3 folder and you cannot use aws s3 cp --recursive
(or sync
), because they don't have --request-payer
parameter..
Also you cannot use aws s3api get-object
to download a stream and pass it to tar xf -
, which is possible with aws s3 cp
.
I don't see how to make these multifile distributions available for users. Writing a little scala app for this purpose? Could be a solution.
Since yesterday I'm preparing the full distribution that was before in a tar at
s3://eu-west-1.releases.bio4j.com/2014_12_03/bio4j_all_plus_isoforms.tar
. Here are the steps performed, just in case we'll need to do it again:m4.10xlarge
, because of the 10-Gigabit bandwidth)[x] Download the tar
[x] Untar it
[x] Set the S3 multipart upload threshhold to something big:
[x] Upload the folder to S3:
P.S. I don't know how to check that everything went well though..