bio-guoda / preston-brit-2022

experimental image corpus for BRIT
2 stars 0 forks source link

$/TB/year cost estimates #2

Open jhpoelen opened 1 year ago

jhpoelen commented 1 year ago

Michael D. shared a size/cost estimate for image storage - $258 / TB / yr on a estimated corpus of 145 TB

image

@jbest is this along the lines of what you are seeing?

jbest commented 1 year ago

@jhpoelen We're expecting to pay ~$180/TB/yr

jhpoelen commented 1 year ago

@jbest thanks for sharing! Sounds like you got a better deal than Michael D. did. I wonder what makes up for the price difference.

jhpoelen commented 1 year ago

fyi @denslowm - now I am curious about the different rates that folks pay to storage / archive their specimen images. . . are you aware of any other project that may be willing to share their $ / TB / yr?

Also, I am curious whether you locked in a price for a specific amount of time, or whether you are at the mercy of the storage provider.

And, what the mitigation would be in case you'd want to move to a cheaper storage provider.

jbest commented 1 year ago

@jhpoelen I don't know where MD got that rate but perhaps it was a full-fledged image server like BisQue or IIIF compliant? Maybe high uptime SLA? We're getting a very good deal at UT TACC but it's a bare bones file repository and HTTP service.

jhpoelen commented 1 year ago

Many aspects to getting image storage services . . . wow must be tricky to make such a big decision.

denslowm commented 1 year ago

$258 /TB/yr was the rate from TACC / iPlant (now CyVerse) for our original proposal (2014). I know their rates have come down since that time. We have basically negotiated to stay with them until the end of 2023. This included at LOT more than just storage, but SERNEC really didn't leverage all of that. I can't speak for CyVerse, but I'm not sure if this is the kind of thing they want to keep doing going forward. [Edit - Yes , @jbest this includes usage of BisQue. I don't think that CyVerse is IIIF compliant, but IIIF has certainly been on my radar based on information from you and Nelson]

Several years ago, I priced out Amazon and Google and spinning disc estimates were much higher than CyVerse.

@jhpoelen Perhaps we could get iDigBio to do a survey of some sort. I think if we asked the right questions we could get a better comparison for the different offerings, but from my experience most TCNs didn't/don't have formal agreements so it would be interesting to find out.

jhpoelen commented 1 year ago

@denslowm great idea about the survey. Just sent a short message to Gil Nelson and Jill Goodwin. Perhaps they can help forward the idea to interested parties.

jhpoelen commented 1 year ago

from morphosource - via https://www.idigbio.org/wiki/index.php/BioDigiCon_2022

$ 0.17 / GB / year for expected 13.5 TB /yr volume growth

image

jhpoelen commented 1 year ago

Media mobilization at Yale Peabody Museum via Nelson Rios at 2022-09-28 https://www.idigbio.org/wiki/index.php/BioDigiCon_2022

volume - 27 TB / 951k images.

Nelson shared that total estimate costs ~ $ / TB / yr ~ $300 a month at a rate of about $5/TB/Month.

image

image

jhpoelen commented 1 year ago

@mkoo shared o Arctos Media aspects > 12TB of media via https://www.idigbio.org/wiki/index.php/BioDigiCon_2022

from Zoom chat - image

image

image

jhpoelen commented 1 year ago

for @nfranz Symbiota Support Hub -

image

from ASU-Services-and-Pricing-Structure-Research-Computing-Confluence-2022-09-28.pdf

Storage Protocols Purpose Free Starting Amount Cost per additionalTB per year
Project-based Storage NFS, SMB, Globus DTN Long term project file storage 100GB $50

image

image

image

jhpoelen commented 1 year ago

So far, it appears we have the following cost / volume estimates:

location est. volume storage costs
@denslowm @ NEON 145TB $258 /TB/yr
@jbest @ BRIT ~13TB ? $180/TB/yr
@mkoo @ Arctos > 12TB TBA
@nfranz @ SSH 16TB $50 /TB/yr
Nelson Rios @ Yale Peabody 27 TB $5 /TB/yr
Julie Winchester @ Morphosource 76 TB $170 /TB/yr

please feel free to update comment / correct etc.