search

smallbets / userbase

Create secure and private web apps using only static JavaScript, HTML, and CSS.
https://userbase.com
MIT License
2.29k stars 125 forks source link

Missing Data Stored #428

Open jneterer opened 1 year ago

jneterer commented 1 year ago

See the attached image below, it looks like data stored hasn't been updated since July 8th, at least for me. Maybe others? Happy to help troubleshoot in any way I can.

Screenshot 2023-07-24 at 8 27 03 PM
j-berman commented 1 year ago

ACK, reviewing

j-berman commented 1 year ago

Context

The meter that tallies up data stored per user/app/admin runs in a nightly cron job. Userbase now has a number of admins who have lots of apps/users/data stored. The nightly meter job was doing a large uncapped number of concurrent DynamoDB reads and writes at a time to tally up data stored for each.

Of note, we (the Userbase service) use the meter to see which admins have stored more data than their subscription plan allows. If an admin exceeds their respective plan's storage limit, their apps continue to function normally but we ask that they upgrade their storage plan.

The issue

In the nightly meter job, the uncapped concurrent DDB reads were hitting up against DDB provisioned limits causing the meter job to fail sporadically.

The solution

The meter process is now smoother and better controlled (source). The cron job now runs to completion. The "data stored" in the admin panel should now properly update in ~24 hour intervals.

Will close this in a week if it runs smoothly as expected every night.