Hitting 429 because of DailyStorageUsage with policy "storage", what is this?

[jordy-homerr]

We are using azure appconfiguration for our test and production environment, however as of lately we are unable to add or edit anything on our feature flags or configurations due to the following response we're getting when trying to mutate our configuration. The calls we're doing to get the current state of the flags and configuration do work properly.

Response we're getting

--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
{
  "type": "https://azconfig.io/errors/too-many-requests",
  "title": "Resource utilization has surpassed the assigned quota",
  "policy": "Storage",
  "status": 429
}
--------------------------------------------------------------------------------

Our metric in azure "DailyStorageUsage" is constantly hitting 100%, but we have no idea what this metric actually means and how to avoid this. We used to have feature flags that were mutated a lot and used up quite some storage, but these were removed over a week ago already, so this cannot be the problem. The storage limit for us on the standard plan should be 1gb and we're deff not storing this much configuration and feature flags.

Is this a bug? Or how can we solve this issue? It's blocking us. A support request was created for this as well, but the support has in no way been helpful, support id for reference #2410300050000802

[zhenlan]

Hi @jordy-homerr, looks like you hit the storage limit of the Standard tier. Please see the following FAQ for more information and potential options:

https://learn.microsoft.com/en-us/azure/azure-app-configuration/faq#my-application-receives-http-status-code-429-responses--why

Please note that the storage usage includes active key-values and feature flags as well as all their historical changes in the past 30 days. Deleting a key-value or feature flag will not release its storage immediately. If you made changes to a feature flag frequently, it could accumulate storage usage very quickly. The storage occupied by history revisions will be freed automatically after 30 days.

[jordy-homerr]

Hi @zhenlan thank you for your answer, finally an actual answer. Why is there a 30 day historical data period and can we override it/force delete everything? We deleted it, so we'd like to actually have them deleted so we can use our app configuration again without waiting 30 days.

[zhenlan]

Hi @jordy-homerr One of the benefits of the Standard tier is the retention of a 30-day revision history (compared to a 7-day retention period for the free tier). However, we recognize that there are scenarios where users may prefer a shorter revision retention period. We are currently working on a feature that will allow you to customize this setting. Stay tuned for updates!

[jordy-homerr]

Hi @zhenlan, thanks for your quick response. So that means there's currently no way of purging the revisions or emptying the "bin" (deleted flags and config) right?

[zhenlan]

@jordy-homerr that's correct. Even with the customizable retention setting, you can change the retention period for future revisions, but you cannot purge existing revisions. At this point, you may either wait for the old revisions to expire or consider upgrading to the Premium tier, which offers higher storage and additional capabilities.

[jordy-homerr]

Hi @zhenlan, I would highly suggest thinking of creating such feature, because it creates problems for production environments that are unable to be solved by the customer. The way to solve this issue now is to let the customer wait for 30 days, I don't think that should be desired in any way. Asking users to upgrade while they've already done interventions to NOT have to upgrade and then telling them wait 30 days doesn't sound like a great solution.

[zhenlan]

Hi @jordy-homerr Just to share some design considerations: As you may know, App Configuration supports soft delete and purge protection. An in-store purge function could potentially serve as a backdoor to bypass the soft delete and purge protection. A purge function can be independent on the retention customization feature. New features can be considered for the future. We will continue to listen to customers' feedback. Thank you for sharing your feedback.