API accounting proposal

Here's my plan for API request accounting:

Overview

Use Lua with Nginx to write to a redis hash
Display current month's count in the api view
Set alert threshold to raise webhook and/or send email
Show long-term trend on a graph
Data capture and storage

We'll use Nginx and Lua to increment a counter in Redis, synchronously with the request. It's very fast.

Data structure in Redis

There are various options for data structures. If we just want this month's usage, that's pretty simple:

Simple counter

key: api_accounting:[sitename].rooftopcms.io:[api_key]
value: [int]

You can INCR on the value to keep the counter up to date.

Sorted set for timeseries

We could use sorted sets to store timeseries data:

key: api_accounting:[sitename].rooftopcms.io:[api_key]
score: [timestamp]
value: {
   timestamp: [timestamp,
   count: 1
}

We don't need to increment the counter; a new member is added to the sorted set for each request. Querying ranges will be very inexpensive because the timestamp means we already have the data date-sorted. I don't know how to SUM the count, though.

Displaying the counters

In its simplest form, we could just show the current month's counters against the API key. It would be nice to poll the count periodically and raise an alert (webhook and/or email) if it went over a predefined threshold.

Using the nested set approach, if we can cheaply sum, we could graph requests over time.

rooftopcms / rooftop-cms