Feature request: A way to set limitations to usage of specific resources (KV reads, writes, requests, etc.) in the dashboard, and being notified when these limits are approached.

oliver-oloughlin commented 10 months ago

What problem are you trying to solve?

I feel like it is inevitable that mistakes will be made, either by me or others, that will cause a project to spend a lot of unnecessary resources. There is currently no way of setting a cap on resource usage, like KV reads and writes, and also no way of being notified when there is excessive spending.

Describe the solution you'd like

I propose a dashboard feature that enables you to set limits on usage of specific resources for a specific project. I also think it would be very helpful to receive some sort of notification when these limits are being approached, and when they are actually hit. I think being notified by email makes the most sense.

Describe alternatives you've considered

Right now, usage has to be manually monitored.

Documentation, Adoption, Migration Strategy

No response

inverted-capital commented 7 months ago

Or a prepayment option ?

A spend limit would be good to implement first - I'm quite nervous about runaway usage - the power of deno means we can blow a lot of cash quickly 🔢

KyleJune commented 6 months ago

I think a feature like this is pretty important and would make it easier for people to be comfortable using Deno's metered services. Without limits, users are risking the potential for a large unexpected bill due to either a bug in their code or an external malicious actor finding a way to abuse their website.

I've seen Vercel run into issues related to not having hard limits. I believe they have soft limits and webhooks that will notify you and an API for pausing, but no simple hard limits that users can set. In the most recent case of this issue, many people expressed that they didn't know about the webhook and the API they could use to pause their deployments.

Here is a situation where someone accidentally racked up a 3k bill in 6 hours due to a bug in a simple astro app. https://twitter.com/shoeboxdnb/status/1643639119824801793?t=FgzNgaVXJQnNymXMD5hYAg&s=19

Here is a more recent case of someone getting a very large bill (23k) from vercel due to someone abusing their stripe webhook endpoint. https://twitter.com/michaelaubry/status/1757539928534315322?t=VGLyBFqwEiTAdNvXYG9ZJA&s=19

Here is a comment where they explain their way of handling spend limits that I mentioned earlier. Many people commented that they didn't realize the limits they set wasn't actually protecting them and that they need a webhook for pausing it to actually be protected. https://twitter.com/leeerob/status/1757960730865696892?t=mCD60pTl5ePP9iQCmP_zeQ&s=19

I think it would be nice to have a soft limit that just notifies you and a hard stop limit that stops your deployment. That way people can comfortable put their projects on deploy.

Beyond the piece of mind for developers/companies using Deno Deploy, Deno would also benefit from it as it would help prevent situations where customers unintentionally rack up large bills they are unable to pay.

runelk commented 6 months ago

I've seen this issue pop up again and again over the years in various serverless solutions, going back to posts like e.g. How we spent $30k in Frebase in less than 72 hours (from 2018). A lot of the time the solution seems to be either:

setup a gateway, WAF, or some other form of protection, which tends to require knowledge + time&effort to the point where it kind of invalidates the number one selling point of serverless (build and deploy without worrying about server stuff)
"get in touch with support if you run into trouble and we'll probably cancel the bill", which IMO is not enough to give your devs a good nights sleep

It's a tough one to get right as most solutions (rate limiting, hard limits, etc.) come with their own problems (queue posts about startups loosing revenue because of downtime), but I think it's worth spending some extra resources on since it doesn't seem like any provider has come up with an adequate general solution yet. The first ones to get this right would gain a lot of love from devs I think.

inverted-capital commented 6 months ago

An easy start would be to let us see the current billing amount programmatically ? Then we could periodically make an API call to check our current bill and decide something from there ?

KyleJune commented 6 months ago

Here is another example where someone with a free plan on netlify ended up with a bill for 104k in 4 days for a simple static site because there were no hard usage limits and no way to configure them. More arguments for having hard limits can be found in this hacker news thread about it. Netlify is shown as a user of Deno on the Deno.com homepage. If the service they provided was deno subhosting, I imagine users of deno subhosting would want a way to set hard limits too.

https://news.ycombinator.com/item?id=39520776

beingminimal commented 6 months ago

@ry any consideration?

inverted-capital commented 5 months ago

In my stripe account for my pro plan with deno deploy, I see this image:

This burst in read requests happened today - all I'm really after is to be able to read this information programmatically - clearly it is being communicated to stripe - am I missing something in how I can read this for myself and implement some checks to help me sleep at night ?

kewp commented 4 months ago

One solution I'm working on is a wrapper around KV that acts as a rate limiter - e.g. for every read operation you first check a value in KV like 'kv_read_limited' and then do nothing if it is set ... Then for every read you do count how many bytes you got, then save that in KV too ... then have a cron job that turns on 'kv_read_limited' if it exceeds a threshold e.g. 1 million units per day, and sends an email / slack if needed ... would slow down KV access a lot but at least you would be sure these blowups couldn't happen. Would be a nice library to have so everyone could use it. BUT it would be even nicer if this were just a parameter you could set in each project on deploy! hint hint!

kewp commented 4 months ago

I've made a start on a wrapper for this, would appreciate any help to make sure it works as expected https://github.com/kewp/kv-limiter

arnauorriols commented 1 month ago

Just informing we are days away from releasing a spend limit capability. It will essentially let you set a dollars limit on your bill, that once reached your apps will stop accepting new requests. Stay tuned!

kewp commented 1 month ago

Great news!

inverted-capital commented 1 month ago

Thank you kindly Deno Deploy people 🥳

One less random massive cloud bill to worry about 😅

denoland / deploy_feedback