Closed eparejatobes closed 8 years ago
Just so I don't forget later, to avoid paying a lot due to people downloading TBs of data from who knows where, we have two options
I set the releases.bio4j.com
bucket requester-pays, with public read-only permissions. Right now I'm copying files from that bucket to another one in EU, releases.eu.bio4j.com
. This should be solved as part of the import process.
OK those buckets where in the wrong account. Right now we have
eu-west-1.releases.bio4j.com
eu-west-1.raw.bio4j.com
cc @pablopareja
we should expand this, move to docs after it stabilizes
hi1.4xlarge
spot on eu looks good.eu-west-1.raw.bio4j.com
scoped by data source name and version/dateThis bucket will have convenient lifecycle settings so that old raw data gets pushed to Glacier, we use reduced redundancy, etc. With respect to access, it is public-read with requester pays.
eu-west-1.releases.bio4j.com
eu-west-1.raw.bio4j.com
eu-west-1.releases.bio4j.com
Same as for raw, eu-west-1.releases.bio4j.com
has archiving, public-read requester-pays etc.
ping @pablopareja
What do you mean by
scoped by data source name and version/date
:question:
that the name of the S3 object (including the prefix) should contain that information
something like:
bio4j_0_12_enzyme_12_03_2014.dat
:question:
well the first part should not be there, this has nothing to do with Bio4j (I mean the file itself). I'd
enzymedb_12_03_2014_enzyme.dat
under the folder
s3://eu-west-1.raw.bio4j.com/bio4j_0_12/
:question:
I think, bio4j_0_12/
shouldn't be there (and by the way, what about semantic versioning?). So
s3://eu-west-1.raw.bio4j.com/enzymedb/12_03_2014/<the_original_file_name>.dat
I couldn't care less about it... :smiley: Please let me know the best option so that I can start uploading them :space_invader:
s3://eu-west-1.raw.bio4j.com/<resource-scope>/yyyy-mm-dd/<the_original_file_name>
but what's <resource-scope>
:question:
Could you please write what it would exactly be for this case?
enzymedb
so no trace in the end about the bio4j version where it's used right?
of course not! :hamburger:
This is done. Closing
We need to have this in a minimally organized way.