CDLUC3 / dmphub

Simple metadata repository for networked DMPs
MIT License
3 stars 1 forks source link

Create an SQS Queue #114

Closed briri closed 2 years ago

briri commented 2 years ago

We should setup an SQS Queue to help us manage the new system's workflows. The queue will be used to help isolate the system from issues with external dependencies.

Lambdas have a maximum timeout of 15 minutes. While it's unlikely anything would take that long to process, we should ensure that anything outside our control does not interfere with the basic functionality of the system.

A queue will also give us the opportunity to 'pause' communications with external systems and reprocess failures.

An example of the basic message structure will be: {"topic":"pending-publication","dmp":"DMP#doi:10.12345/A1B2C3","paused":"false"}

The Lambdas will be setup to poll the Queue and process messages that match their designated topic and are not paused.

Lambdas triggered by API                      SQS Queue Topic                         Lambdas triggered by SQS
_______________________________________________________________________________________________________________

dmp-hub-lambda-post-dmp 
        and
dmp-hub-lambda-put-dmp
            |
            |
            |-----------------------------> pending-publication ----------------> dmp-hub-lambda-ezid-publisher
            |                                                                                       |
            |                         published (or failed-publication) <---------------------------|
            |
            |
            |-----------------------------> pending-download -----------------> dmp-hub-lambda-document-downloader
                                                                                                    |
                                        downloaded (or failed-download) <---------------------------|

dmp-hub-lambda-put-dmp-fundings 
        and
dmp-hub-lambda-put-dmp-related-identifiers
            |
            |
            |-----------------------------> pending-publication ----------------> dmp-hub-lambda-ezid-publisher
            |                                                                                       |
            |                         published (or failed-publication) <---------------------------|
            |
            |
            |---------------------------> pending-notification --------------> dmp-hub-lambda-provenance-notifier
                                                                                                    |
                                      notified (or failed-notification) <---------------------------|

dmp-hub-lambda-delete-dmp

            |
            |
            |-----------------------------> pending-unpublication --------------> dmp-hub-lambda-ezid-publisher
            |                                                                                       |
            |                       unpublished (or failed-unpublication) <-------------------------|