Use other hosts to keep costs down?

[crazy4groovy]

Would using a CDN like cloudflare help reduce your costs? Or DigitalOcean? Zeit? They have been known to help struggling viral projects like this stay afloat, such as jsperf.

Try contacting them.

[customstickers]

FWIW https://customsticke.rs/ runs on a single $5 / mo DigitalOcean box behind a free CloudFlare CDN. But I see almost no traffic there. :)

(see also issues #14 and #16 and #25 for more about this "mirror")

[brunolemos]

Seeing our transactions, everything is super cheap, except for "Cloud Storage Network (Egress) - Americas and EMEA"

A CDN solves that, right? @crazy4groovy @customstickers could you guys help us migrating to a CDN?

[customstickers]

Okay, well, I wouldn't say using just any given CDN would help a lot, for example, looking at Google's pricing at https://cloud.google.com/cdn/pricing their prices aren't so great for this scale, maybe at most 5x cheaper, for traffic over 1000 TB/mo.

My idea of most optimal setup for such website (traffic below 150 TB / month), would be just a couple of Digital Ocean boxes (similarly to https://blog.kapeli.com/a-poor-mans-cdn), they give 1 TB for just $5/mo, which is just $0.005/GB/mo, 16x cheaper than cheapest Google CDN price for below 10 TB/mo and your .gifs - that's now 20x reduction from your current Google Cloud Storage prices.

It's cheaper basically because Digital Ocean doesn't guarantee as good availability, is not spread all over the world, CDNs have cheaper storage, and with DO you need to do load balancing / geo distribution yourself, but I think it's worth trying to invest a bit more effort here to avoid longer term costs.

Regarding cheaper storage, it may be an issue to store 340 GB on DO boxes ($35 / mo * number of boxes), but with some binary diffing for less frequently accessed files (see the simple bsdiff tool, for instance), it could be reduced to 10% of your current storage requirements.

To sum up, assuming 10 TB / mo traffic, i'd estimate the total costs to be around $50 / month with some 10 smallest DO boxes. That's without gif size optimizations, after applying lossy compression from #16 it'd be like $20 / month - 30x reduction from your $600 estimate.

Alternatively, you could first try to do just one box with free CloudFlare CDN, but they are known to take hosts down for using too much traffic (see https://news.ycombinator.com/item?id=5214480)

UPDATEs: some numbers weren't quite right

[brunolemos]

I've just closed https://github.com/ivanseidel/Is-Now-Illegal/issues/16 (reduced the gif size in 4x), I believe it will have a huge impact on server costs!

[ghost]

In case you were going to shut down, I could set up a mirror on my host as long as I don't get too much traffic.

[brunolemos]

We are monitoring the server costs, so far its great. The cost for the last 10 hours was $13, much lower than last week. (~$40).

[jasondinh]

In case you need a shitload of bandwidth, a LeaseWeb dedi or a VPS at scaleway is way cheaper than DO. It might take longer to load, but it's cheap.

Also I think the amount of traffic you get in the first few days won't hold up for too long. Wait for the traffic to stablize to get a better idea of what you need.

[MayaLekova]

My suggestion was also to wait for the traffic to get more stable, since the first few days there was a peak probably. Although if that costs you that much, you can't wait long, of course.

@customstickers Just curious, what's the general idea behind using bsdiff for this task? Storing patches instead of complete gifs for the least accessed files?

[customstickers]

@MayaLekova Yes, I've experimented a bit with the bsdiff command line tool, and managed to get binary patches of 10% the size of the full .gifs.

I've basically guessed the only difference between each file must be in the bytes storing the 'text pixels', and it turns out it's right, given these results.

Patching with bspatch also was taking only about 50ms on my cheapest Digital Ocen box, so it seems like a reasonable idea to reduce the amount of storage required - it sounds like it can be patched on each HTTP request.

[brunolemos]

Do you have a working example?

[customstickers]

Not really, I've just called bsdiff and bspatch manually to see if it could work. Using it together with Google Cloud Storage is not really an option though - you would need some 'smarter' web server which is able to run bspatch to create the final .gifs. (Hence my DigitalOcean boxes suggestion.)

[crazy4groovy]

For bandwidth I recommend looking into cloudflare CDN, it's amazing what they offer for free! Essentially CF acts as your DNS, so the caching config is just a flip switch ("yes" cache all assets from this domain please). I've used it to front a popular blog site, and never had any issues.

[maebert]

Another suggestion:

1. Host the image generator on AWS Lambda (Zappa is your friend).
2. Post all generated GIFs to Imgur (or just redirect there if X is already illegal).
3. Host the actual website on Github pages for free.

Yes, I'm a cheapskate.

[brunolemos]

The website is on gh pages.

Just an update, the costs have been ~$15/day in the last few days.

We had 65k sessions on the last 7 days, 60% lower that the previous period (155k)

[ytsok777]

Welcome Tudorache Constantin

[darthvader45]

I know costs are high, but in this case and many others, Google is your friend. In my search, I found a cheap server host called GoDaddy, which does 1$ a month and free domain name. I'm sure you could manage it with them. https://www.godaddy.com/offers/web-hosting/cheap?isc=hos1g203&slid=&pgrid=48518771507&ptaid=kwd-302440314468&mkwid=sO8a04e7i_pcrid_251166183727_pkw_%2Bcheap%20%2Bwebsite%20%2Bhosting_pmt_b_pdv_c_&gclid=Cj0KCQjw6rXeBRD3ARIsAD9ni9DKS78DU910-TmS-PWMJTDNk3CzeE7raWd8P5mbNZcrF7TF7Wng2EoaAgRUEALw_wcB

[ezhes]

If you need help hosting, I have a ridiculous amount of server gear (across multiple geographic locations..) as part of probably one of the most expensive hobbies. While I'm not able to give financially, I'd love to offer my server gear up to host the site. I see that it's already a docker container so I could just deploy it to my swarm and we should be good to go so long as you point the domain name to me.

[darthvader45]

I think that sounds amazing. Wonder if that could work.