Calculate and expose the "health" of a feed

[jamespfennell]

Having the API tell whether a feed is healthy or not would be really beneficial. Consumers of Transiter could rely this information to their users so that their users can make informed decisions about the data. Also, if relevant, consumers could notify the source of data that's something wrong.

The "health" may be composed of multiple metrics. Example metrics that seem reasonable:

• Number of seconds since the last successful needed update. Good for detecting stale data.

• The average periodicity of updates in the last hour. Good for detected a general degraded feed.

[jamespfennell]

seconds_since_update num_updates_last_hour

[jamespfennell]

"last_successful_update": {},  // or null
"windows": [
    {
        "start_time": 123,
        "end_time": 123,
        "count": 45,
        "outcomes": [{
            "status": "SUCCESS",
            "result": "UPDATED",
            "count": 40,
        }],
        "update_periodicity": 15,  // or null if not two updates. Handle the case when two updates way in the past
    }
]

[jamespfennell]

Added statistics as in previous comment.

Other nice to haves:

• last update
• last successful update
• better information on the time between updates. We have the average now, but knowing the max would be awesome.

[jamespfennell]

This ticket was created before I knew about monitoring best practices (i.e., using something like Prometheus).

Transiter now has Prometheus metrics for API latency and feed update latency. I think we should add the following metrics:

• The update time, per feed and status. We can use that to see how stale feeds are.
• ?

Then we can close this issue.