Middleware page caching

[dburriss]

Summary

In order for the paging to be as performant as possible, we want to cache page results.

Initial design notes here.

Related concepts:

• Caching in .NET - general mechanism for caching in .NET
• Response caching - when the server handles the caching and response
• Output caching - caching controlled by the server code

Motivation and goals

Since the return of events is based on paging, and the list of events is immutable, we can cache any completed page indefinitely since the contents of the page will never change. The messaging mechanism relies on polling for changes, so it is important that requests for pages, especially the latest page (tail), put as little strain on the server and database as possible.

Since caching is so important, relying on response caching can be unreliable. Not only does output caching allow clients to send a no-cache header, it is also inappropriate for the tail page as only the server knows when this will change. With only a few clients requesting every second for the last incomplete page, this could put a significant load on the server and database.

So a solution needs to consider the following goals:

• [ ] Allow caching of complete pages, long term (configurable since seldom accessed pages need not be in cache). A TTL on pages may be appropriate here.
• [ ] Requests to pages, regardless of cached or not, need to be able to collect usage metrics (not yet implemented). I think this rules out response caching unless we can collect them before that middleware kicks in.
• [ ] Any cache mechanism needs to allow for notifications to be sent of new events being available. We might want to design this beforehand and take it into account. I will put something in a comment about this. The key here is if a back-channel notification is sent, say via EventGrid when the consumer receives that and looks for new events, they should be in the response.
• [ ] The backing for the cache should be configurable. A default of in-memory is probably fine but allowing configuration of additional stores should be possible and taken into the design.

In scope

• [ ] Default in-memory output cache of complete pages with configurable TTL
• [ ] Worker for updating cache of tail page
• [ ] Initial design notes on configuring different cache backing stores
• [ ] Initial design notes on how interception point for updating usage metrics
• [ ] Initial design notes on how this would interact with back-channel notifications

Out of scope

• Additional cache backing stores like Redis, MemCache
• Back-channel notifications
• Usage metrics

Unknowns

• Is IMemoryCache and IDistributedCache acceptable for implementation of this (and using existing implementations)
• Metrics and back-channel notifications are not yet designed
• Is response caching useful for anything as a first level (need usage metrics captured)

[dburriss]

@Grepsy you expressed interest in picking up this issue? Take a read and let me know if still interested.

[Grepsy]

Yes, I'm still interested ;-)

[Grepsy]

When considering

The messaging mechanism relies on polling for changes, so it is important that requests for pages, especially the latest page (tail), put as little strain on the server and database as possible.

and

Worker for updating cache of tail page

It seems going for a background worker that frequently updates the tail page might not be beneficial for reducing database strain. The worker updating the cache every x seconds will put a constant load on the database. An on-demand cache (only responding when a request is sent) with a very short expiration will have no load in absence of requests, yet scale up to the same max. load as a background worker under high request load.

If we have a requirement for high-performance near real-time event publishing I think it makes more sense to expose a websocket endpoint where event are streamed instantly instead of polling. This is the same mechanism I often seen used on exchanges publishing real-time price and trade updates. Historical data would of course still be done using the paged API.

[dburriss]

It seems going for a background worker that frequently updates the tail page might not be beneficial for reducing database strain.

You raise a good point. The more I think about it, and consider your points here, fleshing out some pub/sub mechanism for when an event is added will give us some options here.

Some quick ideas:

• On WRITE of the event we updated the cache (or just invalidate it)
• If cache is invalid a call falls through to the DB, then updates cache
• Above points work for something like Redis but what about in-memory?
• For in-memory another feature I had in mind could be used which is the back-channel notification
• The problem is I don't like the idea of relying on the back-channel notification since one of the appeals for this is being able to set this up with minimal infra

So the above makes me think we can make some assumptions:

• If using the in-memory, the load is low and in-memory can be used for complete pages and the tail is always a pass though. We could do some optimization here were 2 requests within a second of each other could get a cached value...
• If a higher load scenario, we can scale up the complexity of keeping the cache up-to-date. On event write, tell a background worker to update the cache, or invalidate it and pass though (would prefer former).
• Another planned feature is recording the usage metrics. Theoretically, this could be used to predict what is reasonable TTLs on the cache, or how often a background worker should run.

If we have a requirement for high-performance near real-time event publishing I think it makes more sense to expose a websocket endpoint where event are streamed instantly instead of polling.

High performance is relative but that isn't the explicit goal. There are plenty of tools that provide that, but are complex to setup and run effectively. Adding websocket may be a good idea but I would like to layer complexity on as needed. Starting with a REST based feed that provides a pull based mechanism needs to be at the core.