Daily Datastore quota limit reached

[bsiegert]

BulkTracker was updated to use "Cloud Firestore in Datastore Mode". Ever since then (I think), the app has been mostly unavailable.

The reason is that the "daily quota is exceeded", example:

2021/12/02 20:28:07 rpc error: code = ResourceExhausted desc = Quota exceeded.

Checking the Quotas page in Pantheon (https://console.cloud.google.com) says that there are no daily quotas, so this is both unexpected and concerning. I wish I knew what sort of daily quota I am exceeding.

For now, I disabled the Task Queue in Cloud Tasks, which should stop ingestion of new build results.

I probably should do a bulk deletion of data that is older than, say, one year. But even so:

• WTF is the type of quota the app is exceeding and why?
• Where can I track that particular quota?
• Is this due to ingestion of new data?

And of course: How to mitigate and fix?

[bsiegert]

I have written a blog post about this outage: https://bentsukun.ch/posts/bulktracker-outage

In the end, the daily quota that had been exceeded was the "daily spending limit" for App Engine. As this limit is deprecated, it is not shown prominently in the UI, so I forgot to check it.

In the meantime, I have deleted a bunch of old data, which brought storage costs down.