Handle Deletions

[donaldgray]

Update API to cleanly handle items deleted from DLCS.

A deletion action can clean up some resources that the API knows about (ie DB rows) and then raise a notification that can be listened to.

The listener for this service can either be a single deletion service, or respective service can each listen for notifications. We'll need to remove:

• Cached assets
• Orchestrated source assets
• Generated derivatives (info.json, thumbnails)
• Generated NQ results that contain asset?

[donaldgray]

https://github.com/dlcs/protagonist/pull/438 contains DeleteHandler app. This is for Deliverator so will need exanded, namely to handle deleting info.json and T / F assets

[donaldgray]

The DeleteHandler project for Deliverator runs as an dockerized service for ease. We should consider running this as a lambda in AWS - either via container or as a script as it will be seldom used.

[donaldgray]

The storage-keys document outlines what keys can possibly be used for asset storage. These should drive the bulk of the delete logic, dependant on the type of asset. In addition to that we'll need to:

• Raise an SQS notification (or SNS->SQS) when asset is deleted. We could use IAssetNotificationSender to raise them. This is called but just logs the delete action.
• Delete orchestrated asset from NAS storage (ie lustre). This negates above statement about using a Lambda - it'll need to run on an instance with NAS mounted.
• Invalidate Varnish cache. We should be able to use varnishadm for this but we need to be careful to not expose this to public traffic, alternatively have a queue listener within the varnish image itself. Varnish could be caching a number of items depending on the item type. May need changes to varnish image.
• Invalidate CDN cache - this should be a configurable option.
• Ideally named query projections that contain this item. This will require some investigation as to how best to handle this as there are so many permutations of NQ results. It may require additional storage of which NQ projections contain which keys.

[donaldgray]

Deleting named queries need to raise a delete notification as it could result in the deletion if a number of generated NQ projections #540

[tomcrane]

Acceptance Criteria

MVP

• [x] Rename DeleteHandler to CleanupHandler
• [x] The CleanupHandler application is running in protagonist environment and listening to the SQS
• [x] Protagonist is raising SNS notifications for a DELETE operation on any asset
• [x] The SNS event raised specifies create, update, delete, cache-invalidate as an attribute (for sns message filtering) and includes both customer id and customer name (path element)
• [x] When CleanupHandler gets a delete message:
  • [x] Delete from S3. storage-keys document outlines what keys can possibly be used for asset storage.
  • [x] Delete orchestrated asset from NAS storage (ie lustre)
  • [x] Invalidate CDN cache (not configurable for MVP)

Invalidation of stored NQ projections is not MVP.

Invalidation of Varnish cache is covered by https://github.com/dlcs/dlcs-varnish/issues/3

(Later those last three cache invalidation operations can be refactored into https://github.com/dlcs/protagonist/issues/252 and CleanupHandler will call a new cacheInvalidation queue)

Note Removed Varnish cache from checklist as it's covered by linked ticket

[donaldgray]

The list of possible paths that can be invalidated will depend on the asset. In below examples I'm using assetId of 1/2/foo.

Rules based on delivery-channel (an asset may have more than 1 of these):

• If delivery-channel includes "file"
  • /file/1/2/foo - stream file
• If delivery-channel includes "iiif-img"
  • /iiif-img/1/2/foo/* - image-services for canonical IIIF Image Api version
  • /iiif-img/v2/1/2/foo/* - image-services for IIIF Image Api v2
  • /iiif-img/v3/1/2/foo/* - image-services for IIIF Image Api v3
  • /thumbs/1/2/foo/* - thumbs for canonical IIIF Image Api version
  • /thumbs/v2/1/2/foo/* - image-services for thumbnails, IIIF Image Api v2
  • /thumbs/v3/1/2/foo/* - image-services for thumbnails, IIIF Image Api v3
• If delivery-channel includes "iiif-av"
  • /iiif-av/1/2/foo/* - stream AV assets

Rules based on assetFamily (these will change to d-c in future)

• If AssetFamily is 'I':
  • /iiif-manifest/1/2/foo
  • /iiif-manifest/v2/1/2/foo
  • /iiif-manifest/v3/1/2/foo

A single invalidation includes 7 wildcards so we can only have 2 in-flight with CloudFront at any given time.