Convert existing ‘IOP’,‘ThumbnailPolicy’ and ‘DeliveryChannel’ values from Images table into ImageDeliveryChannels table

[tomcrane]

Donald:

At this stage we have duplicate records in IOP/ThumbailPolicy and ImageDeliveryChannels. This should make engine implementation easier.

[tomcrane]

Could this be a DB migration? Or a one-off script?

[tomcrane]

This migration can be run often and repeatedly as it doesn't modify the source data (a later cleanup can remove those fields).

Acceptance Criteria

• An application / script that can be run against a Protagonist DB to convert old data to new
• The script won't break if it reprocesses the same row, it will produce the same end state whether it has been run 1..n times
• The script leaves the original data intact
• All images for the iiif-img channel get the default iiif-img channel policy; this will be mapped to fast-higher internally.

[tomcrane]

This script / migration will need to create all the distinct policies for thumbs it encounters, so that it can create an entry in the linking table. E.g., for Wellcome, there are (at a guess) 3 different policies in use. After running, there will be three entries under here, rather than one:

{
    "@context": "https://dlcs.github.io/vocab/context/future.json",
    "@id": "https://api.dlcs.digirati.io/customers/2/deliveryChannelPolicies/thumbnail",
    "@type": "Collection",
    "totalItems": 1,
    "member": [
        {
            "@id": "https://api.dlcs.digirati.io/customers/2/deliveryChannelPolicies/thumbnail/standard",
            "@type": "vocab:DeliveryChannelPolicy",
            "id": "standard",
            "displayName": "Standard thumbnail sizes",
            "channel": "thumbnail",
            "policyData": "[ \"!1024,1024\", \"!400,400\", \"!200,200\", \"!100,100\" ]",
            "policyModified": "2023-09-19T15:36:58.6023600Z"
        }
    ]
}

i.e., the migration might have created 50m linking rows but on the way it only needed to make 3 policy rows. Although the default policy was probably created before processing the 50m.

[tomcrane]

Thinking about this task after writing https://github.com/dlcs/protagonist/blob/delivery-channel-sequence/docs/delivery-channel-switchover/index.md

That talks about emulating the wcDeliveryChannels, ImageOptimisationPolicy and ThumbnailPolicy fields by converting them at the API boundary into the new model internally in the DLCS, so that existing callers (mainly Wellcome) can switch over seamlessly. That includes calling the API with no values for these (which is the norm) - the dc relationships still get created from defaults.

That approach could be used to implement this issue; we do the conversion by calling the same conversion, per asset.

⚠️ This work starts to collide with the migration of shared services and other DLCS instances.

However, use of IOP/thumbnailPolicy is rare/nonexistent and this will be a deprecated feature.

[tomcrane]

Closing https://github.com/dlcs/protagonist/issues/630 as it is a subset of this issue. Part of this issue (620) is to convert implied/default thumbnail delivery channels into explicit links to thumbnail deliverychannelpolicy(ies), using the correct syntax, and not doing what #630 proposed.

[tomcrane]

Need to make sure that the necessary DCP entries exist before creating all the linking policies.

• At Wellcome we do have some non-standard thumbnail policies (for Miro - "small" and "tiny") - so they will need to be in the DCP table to link to.
• There are also some 1000,400,200,100 polices (rather than 1024,400,200,100) from a long time ago.
• UAL and IDA don't have the 1024 policy, they have larger (1500?)
• The script should not be dependent on specific policyData, it should migrate what it finds, so it can be used in all our migrations.

• At Wellcome, the presence of iiif-img in the current data implies iiif-img AND thumbs, with (for now) the 1024,... policy.

Also remember that only Wellcome is running protagonist and has the v1 deliveryChannels impl.