ManifoldRG / MultiNet

MIT License
11 stars 1 forks source link

Evaluate disk VS buckets #58

Open pranavguru opened 3 months ago

pranavguru commented 3 months ago

Assess whether buckets are a cheaper and more efficient option to store Multinet v0 internally, compared to attached persistent disks

eihli commented 2 months ago

Disk pricing: https://cloud.google.com/compute/disks-image-pricing#persistentdisk Object pricing (Cloud Storage): https://cloud.google.com/storage/pricing#cloud-storage-pricing

Options with disk:

Options with Cloud:

What would that look like if we're serving ~80 TB?

Persistent disk: 80e12 / 1e9 * 0.05 = $4,000 Hyperdisk: more than twice that; we should only use this if we need the performance gain, and I doubt we do. Standard: $1,600 + ... this all depends on how we store stuff. If we store individual images, it will get ridiculously expensive. 80 TB of 500KB images would cost $64,000 to download. But 80 TB of 1GB files would only cost $32 Nearline: Half of standard, but same thing about file sizes applies. Coldline: Same...

As long as we're not paying for throughput (i.e. we don't have millions of individual files), we're fine with Standard Cloud Storage: https://cloud.google.com/storage/pricing#north-america.