Full distribution untared

[laughedelic]

Since yesterday I'm preparing the full distribution that was before in a tar at s3://eu-west-1.releases.bio4j.com/2014_12_03/bio4j_all_plus_isoforms.tar. Here are the steps performed, just in case we'll need to do it again:

• [x] Launch a big instance (I'm using m4.10xlarge, because of the 10-Gigabit bandwidth)
• [x] Add a huge EBS to it (3000GB)

• [x] Download the tar

  aws s3 cp \
  --region eu-west-1 \
  s3://eu-west-1.releases.bio4j.com/2014_12_03/bio4j_all_plus_isoforms.tar .

• [x] Untar it

  tar xvf bio4j_all_plus_isoforms.tar

• [x] Set the S3 multipart upload threshhold to something big:

  aws configure set default.s3.multipart_threshold 2TB

• [x] Upload the folder to S3:

  aws s3 sync \
  --region eu-west-1 \
  mnt/disk1/bio4j/ \
  s3://eu-west-1.releases.bio4j.com/2014_12_03/bio4j_all_plus_isoforms/

P.S. I don't know how to check that everything went well though..

[laughedelic]

Done! :shipit: I've checked that all files are in S3. Didn't perform and DB tests though. I'm going to use this issue to modify the docs later.

[laughedelic]

The problem is that you cannot use aws s3api get-object to download an S3 folder and you cannot use aws s3 cp --recursive (or sync), because they don't have --request-payer parameter..

Also you cannot use aws s3api get-object to download a stream and pass it to tar xf -, which is possible with aws s3 cp.

I don't see how to make these multifile distributions available for users. Writing a little scala app for this purpose? Could be a solution.