Trigger restoration and transfer from Eventbridge

[paul-butcher]

In order to get closer to a fire-and-forget approach, write a script that:

1. cracks the input list into batches of 20
2. Creates Eventbridge one-off schedules that invokes a restoration of four of these batches in the evening each day
3. Creates four evenly spaced Eventbridge one-off schedules to invoke a transfer of the shoots that were restored the previous day

See: https://wellcome.slack.com/archives/C02ANCYL90E/p1723028197261449

[paul-butcher]

Actually, what about a Step Function? Something like this:

The Restore function could run "tonight" and emit the list of shoot numbers with the date "tomorrow", broken up into three groups spread across the day.

[paul-butcher]

Actually. All of this is too complicated. We can have two lambdas, one that runs daily at 1930, another running daily at 0830

• A client script puts the shoots on the first queue in batches of 60 per message.
• Restore lambda pulls one batch off (1930),
  • restores them
  • puts them on the transfer queue
• transfer lambda runs at 0830 and transfers everything on the transfer queue.

If this turns out to overwhelm Archivematica, then we can interpose a step where Restore batches them into 20s and a separate lambda pulls a single batch of 20 three times a day to put them all on the transfer queue. But we should cross that bridge when we come to it, because (🤞) AM will be a bit more reliable next time.

[paul-butcher]

I've just had an experience of 60 at a time killing Archivematica, by filling up the failed folder, but that could have been due to some records failing yesterday and clogging up the place.