SNS topic instead of DDS API Call

[aray-wellcome]

I'm not sure if you all have access to this ticket but this is the one I'm referencing https://github.com/wellcomecollection/platform/issues/5494

[rsehr]

Do you have an existing topic we can use to test it? Also, do I need to set specific attributes to the messages or are all messages consumed?

[aray-wellcome]

@alexwlchan Can you help with this? Or point us to someone who can?

[alexwlchan]

You’ll need to create new topics in Terraform, and add permissions for Goobi to publish them. Ideally one topic for each of staging/prod. I can provide pointers if necessary.

[mgeerdsen]

@alexwlchan So the idea is for the topic to be set up in our environment (one for staging, one for prod) and you will subscribe to it?

[alexwlchan]

So the idea is for the topic to be set up in our environment (one for staging, one for prod) and you will subscribe to it?

Yup, although it'll be DLCS/iiif-builder that subscribes to it, not me.

[mgeerdsen]

@alexwlchan Could you let me know the account ID that will be used for the subscription to these topics?

[mgeerdsen]

We can now publish SNS notifications from staging Goobi workflow to the topic digitised-bag-notifications-workflow-staging as defined in PR #446

This still requieres cross account subscription and we need to check that the content of the message is correct.

[alexwlchan]

Could you let me know the account ID that will be used for the subscription to these topics?

It’ll be the Digirati account, which has ID 653428163053

[mgeerdsen]

Thanks The Digirati account should now be able to subscribe to "arn:aws:sns:eu-west-1:299497370133:digitised-bag-notifications-workflow-staging". We should now be able to test that the subscription works and that the messages contain the necessary information.

[tomcrane]

Thanks @mgeerdsen I'll update our Terraform and test all that tomorrow.

We should be able to hook up the DDS iiif-stage-new environment so that stage-goobi notifications to that topic and up with created IIIF Manifests.

[tomcrane]

Will the production version of this topic be called digitised-bag-notifications-workflow-prod? Just making matching terraform for it for later use.

[mgeerdsen]

Indeed, the topic for production will be called digitised-bag-notifications-workflow-prod

[mgeerdsen]

To move this to production we will have to tag the stage workflow image for prod, merge #446 (already used for stage), terraform deploy it and use prepared sql statements to insert the new notification step after the current API call step.

[mgeerdsen]

SNS topic and permissions etc. have been added via terraform. Prod Goobi has been updated to include the necessary plugin. We will update the workflows to include both the API call and SNS notification tomorrow.

[mgeerdsen]

and we actually did that, so this issue can be closed as well