Choose object storage provider for haikuports

[kallisti5]

With haikuporter's support of s3, we need to choose a object storage provider. For context, this will be replacing our Digital Ocean volume block attachment which is $25 month / 250GiB

Assuming ~400GiB stored... 2TiB of egress a month (which gives us a lot of head room) Assuming 35 million API ops a month (17M Class A, 17M Class B)

• Wasabi $16-35 / month likely as we grow. Risk of pulling too much egress and getting shut off.
  • Pro: We use them today, "known evil"
  • Pro: Cheap as long as we don't pull more than we store. ($5.99 per TiB)
  • Con: Weird "storage amounts count for 3 months" system.
  • Con: Storage pricing is cheap, but inconsistent. If we use more egress then we store, they get upset and cut us off. (and have done it before). There's a lot of risk here of sudden cut-offs without clear resolution paths.
  • Con: We probably should put them behind some caching CDN. bunny.net is 0.005/per GiB. +$10/mo
• telnyx - Likely $11-20 / month, $ per API operations a big risk. Haikuporter, hpkgbouncer, all hit APIs
  • Pro: Flat $0.006 per GiB stored. At 400 GiB that's $2.4
  • Pro: No egress fees (We started on Wasabi in this model... so I question it long term)
  • Con: Change operations (REST PUT COPY POST LIST) $0.5 per million.
  • Con: Read operations (REST GET SELECT) $0.04 per million
  • Con: New player
• storj.io - Likely $12-20 / month
  • Pro: Flat $0.004 per GiB stored. At 400 GiB that's $1.6/month
  • Pro: Flat $0.006 per GiB egress. At 2TiB that's $12 / month
  • Pro: No per-operation costs
  • Pro: Lots of interesting use cases like hosting our own file gateways
  • Con: Not the cheapest.
  • Con: Would need CDN for + 10 / mo
• Backblaze B2 - Likely $16-24 / month
  • Pro: Flat $6 per TiB
  • Pro: Free egress up to 3x their average monthly storage amount. Egress over average stored is $0.01/GiB. 2TiB - 400GiB = 1600GiB * 0.01 = $16 / month
  • Pro: Big company, been around for a long time
  • Con: Most expensive "cheap" s3 storage provider.
  • Con: Need for CDN likely at scale. Unlimited free egress with CDN partners: Fastly, Cloudflare, bunny.net, CacheFly, Coreweave, Equinix Metal, Vultr, PhoenixNAP. (I've looked at bunny.net before for DNS... they're pretty good. 2TiB $10/mo)
• ~Digital Ocean Spaces - ~$23 per month for 400GiB + 2TiB xfer~ We don't like having to have multiple buckets to get reasonable pricing.
  • Pro: Flat $5 for 250GiB.
  • Pro: Can have multiple buckets for $5 / per 250GiB
  • Pro: 1TiB egress included.
  • Pro: Local to our nodes. May cut down egress bandwidth usage.
  • Con: Expensive 0.01/GiB egress after 1TiB. 2TiB total is $10
  • Con: Would need branch haikuports buckets for reasonable pricing, needing hpkgbouncer changes.
  • Con: Expensive 0.05 per GiB after 250GiB in a single bucket. 400GiB is $12.50/month

Notes: We don't have to go all-in on a single S3 provider. Haiku can remain at wasabi, haikuports can be "where ever". We can run one deployment of hpkgbouncer per repo.

[kallisti5]

My preference:

1. Storj.io - Cheap, does some cool things being based on storj. Would need a CDN for egress.
2. Backblaze B2 - not the cheapest.. but if we grow we can toss a reasonably priced CDN in front of it and get free egress (minus how much the CDN costs... and bunny.net is cheap)
3. Wasabi - Continue with Wasabi... they have served us well. However, we should likely throw a CDN in front of them to reduce risk of getting cut off.
4. Digital Ocean - Risks being expensive, but can't beat "local" object storage access (which will hopefully cut down on bandwidth usage)
5. Telnyx - Cheapest storage at scale... we just have to be mindful of API transaction limits from hpkgbouncer and haikuporter. New provider though.

[kallisti5]

I've reached out to wasabi to try and get "actual" bandwidth utilization numbers. They don't publish it in our portal (but I sure as hell know they look at it since they have cut us off before due to egress)

[waddlesplash]

Why do you have Backblaze as "20-25 a month"? If we factor in the CDN with free egress then shouldn't it be storage costs only, and thus be equivalent to Wasabi + CDN?

[kallisti5]

Why do you have Backblaze as "20-25 a month"? If we factor in the CDN with free egress then shouldn't it be storage costs only, and thus be equivalent to Wasabi + CDN?

"Free egress up to 3x their average monthly storage amount. Egress over average stored is $0.01/GiB. ~2TiB - 400GiB = 1600GiB * 0.01 = $16 / month" 16 + 6 = $21~

EDIT: I did that math wrong. Lets re-run the cost numbers. Assuming 6TiB egress, and 400GiB storage.

Backblaze:

• Storage: $6
• Egress: 6000GiB (haikuports transfer below) - 1200 (400GiB 3) = 4800GiB 0.01 = $48 / month
• Egress + Bunny CDN: 6000*0.005 = $30 / month
• Daily API Call limits. 2,500 Class B free 0.004 per 10,000 after , 2,500 Class C, 0.004 per 1,000 after.
  • If the Wasabi bill is right, we make 900k a month. That's around 30k a day. ~+$4 - $8 / month

Storj:

• Storage: $1.6 / month
• Egress: 6000GiB $36
• CDN: 6000*0.005 $30 (probably not worth it beyond caching down requests to Storj)

Wasabi:

• haha, we'll "cut you off" for using more than 400GiB egress. CDN might cut down bandwidth since it caches popular artifacts........ but man, the risks are big and 400GiB vs 6TiB is a big gap.
• I think we might need to strike off Wasabi for haikuports.

Telnyx:

• Storage: 0.006 * 400 = $2.4 / month
• No egress fees
• $0.5 per million change operations (PUT, COPY, POST, LIST)
• $0.04 per million read operations (GET, SELECT)
  • Big unknown costs since we don't track api call types + counts

Backblaze + bunny.net CDN seems like the best deal tbh with controlled risk. The Bunny.net CDN could cut that 6TiB way down to a "a few TiB or less" on all providers, but it's an unknown how efficient their caching is in our use-case

[kallisti5]

EDIT - Actual worst-case egress bandwidth numbers:

• haiku repo at Wasabi averages 54GiB / month
• haikuports is some percentage of our overall Digital Ocean egress... 5.3 - 5.9 TiB :shaking_face:

[nielx]

For me, while I think reliability is important, it is not the end of the world if we get cut off and need to relocate. However, how do we keep control of our packages? I.e. is there going to be a backup or a primary source for them? Another factor is the odds of hidden surprises, i.e. I do not want to be surprised by a sudden change of rates if we cross some sort of threshold, so any provider with a 'flat' rate that scales linearly is preferred over a provider that requires us to closely monitor some sort of threshold. Finally, I would keep things as easy as possible, so the Digital Ocean Spaces where we will need to do additional data design is off the table for me.

[kallisti5]

For me, while I think reliability is important, it is not the end of the world if we get cut off and need to relocate. However, how do we keep control of our packages? I.e. is there going to be a backup or a primary source for them?

The nice thing about s3 is it actually gets easier to back things up. Today we have the automatic "compress all the artifacts, encrypt them, and upload to an s3 bucket" backup system. That doesn't work for huge things though since I really don't want to work with 300GiB tar delta's :sweat_smile:

In the model where some object storage provider is the source of truth, we really just need to rclone the bucket "somewhere" else. Historically i've just rcloned to a dedicated bit of local storage at my house as a cold backup (you could do the same). rclone works off of deltas like rsync, so it's bandwidth consumption friendly after the initial clone.

rclone also lets you sync between storage providers... and it supports a TON

We actually have an rclone container today ready to go that will do that to storj. We can make some fixes though to make it more generic.

I also have rclonefs which will (theoretically) let us mount s3 buckets as fuse storage mounts on each k8s node so we can (theoretically) offer s3 buckets over rsync to mirrors from pods running on any k8s node. (fuse in k8s is weird though, and we need elevated security context).

Another factor is the odds of hidden surprises, i.e. I do not want to be surprised by a sudden change of rates if we cross some sort of threshold, so any provider with a 'flat' rate that scales linearly is preferred over a provider that requires us to closely monitor some sort of threshold.

Agree. Definitely the biggest pain point of object storage. I really like the pricing of Telnyx, but the whole "per million API hits" thing makes me nervous on something complex and large like haikuports.

Finally, I would keep things as easy as possible, so the Digital Ocean Spaces where we will need to do additional data design is off the table for me.

Agree. Lets strike DO off the list. They had some appealing things to them, but needing a whole gaggle of buckets to groom to get reasonable pricing is too much lift. I'm tired of forming infrastructure "around" providers weird limitations.

[kallisti5]

I updated https://github.com/haiku/infrastructure/issues/141#issuecomment-2383633257 with the pricing based on the actual worst case bandwidth numbers I saw on digital ocean.

[kallisti5]

Oh, and I just looked at the Wasabi bill.. it does list "908.40 API requests" for the month. I'm guessing that's 1000's though given the decimal point.. so 908,400 makes more sense.

[kallisti5]

Here's some data on the bunny.net cdn. It definitely cuts down our bandwidth usage ~50% on a single haiku nightly repo.

I'm sure the savings will be less for haikuports (more random packages, etc)

[nielx]

Looks like the preferred is backblaze + bunny then?

[kallisti5]

Agree. I think backblaze + bunny are going to be the cheapest combo. Bunny will cut down the xfer 50%, so that $30 / month should be "worst case"

[kallisti5]

Ryan went ahead and entered our billing info. I went ahead and deployed a temporary VM @ digital ocean to use to shovel artifacts over to backblaze.

I'm going to start with the Haiku repos themselves since it's an easy (smaller) test of data before moving on to haikuports.

[kallisti5]

~Aaaand.. Backblaze just crapped the bed.~

API calls are NOT free.

EDIT: I guess +$4-8 a month extra for API calls isn't horrible... however it adds risk to Backblaze.

[nielx]

API calls are NOT free.

EDIT: I guess +$4-8 a month extra for API calls isn't horrible... however it adds risk to Backblaze.

That's not great and definitely false advertising...

[kallisti5]

I went ahead and put the haiku repo over onto backblaze. We already blew past the "free tier" of class C api calls during the last sync. :face_exhaling:

I'm about to head out of town and will be back Sunday.. so here are important facts:

• I set a cap of $3/day on API calls just in-case the API calls begin to be too many. If we hit the cap, hpkgbouncer will stop updating and other bad stuff :tm:

If the :hankey: hits the fan, you can take the following actions to undo the migration to backblaze:

• Revert this, and apply via kubectl: https://github.com/haiku/infrastructure/commit/74a09796d2dbbc508d7add6e5617b81d10e56260
• Revert this and apply via concourse: https://github.com/haiku/infrastructure/commit/49cda4522d1d5ce961cf33963187c57a0c2e2dc3