search

dlcs / protagonist

MIT License
7 stars 2 forks source link

Cache invalidation #252

Open donaldgray opened 2 years ago

donaldgray commented 2 years ago

Related to #251. Allow an item to be cleanly purged from caches.

There are multiple levels of caching, some of these are relevant to particular deployments. For example not all deployments will have CloudFront, some may have Akamai or no CDN at all.

Different caching to purge:

Cache invalidation will initially be implemented in #251 and can then be refactored into a separate queue / listener, - see https://github.com/dlcs/protagonist/issues/251#issuecomment-1645518176

When reingesting an asset, raise a cache-invalidation message at the end of engine processing. Need to think how this interacts with orchestrate-after-ingestion.

tomcrane commented 2 years ago

(see #233 for older version)

tomcrane commented 10 months ago

Task for @JackLewis-digirati - review what needs further work on this issue (might be very little). It may be the case that in-memory caching is mitigated enough.

JackLewis-digirati commented 10 months ago

From looking at the points on the above, the disk based and CDN cache are both complete and Redis hasn't been implemented yet.

This leaves the in-memory cache which hasn't been implemented.

However, this has been mitigated slightly by modifying the SQS setup to deliver messages after the point that in-memory caching would be in play (10 minutes). This does mean that in-memory caching isn't invalidated directly, but would disappear off the service entirely after 10 minutes

Currently, all handling of deletion within the .Net code is handled by the cleanup handler, so if in-memory caching is required to be removed. In order to do this, we could either update the API code to be able to listen to the SQS queue and remove data from the cache. This shouldn't be that difficult, but is likely overkill. Alternatively, the API could also force removal from the cache on a DELETE API request