Implement retry logic for intermittent errors in communication with Communication Services.

Description

We already know of multiple potential faults that can occur when sending an email

connection to server is down
429 (?)

We need to define a strategy for how to handle the various errors. Do we set a status code to retry after a given time frame? Do we stop attempting as the error indicates it will never work? etc.

useful doc: https://learn.microsoft.com/en-us/azure/communication-services/concepts/email/sender-reputation-managed-suppression-list

In scope

Identify all known error cases and define an action plan for each case.
Create user stories to handle cases if required or enrich already existing issues

Constraints

Analysis

We have agreed on a data flow in the error situations. An additional topic for system errors or "system updates" will be created. Related microservices can use this topic to alert to instabilities or downtime in their system causing us to for instance pause the request for sending emails.

Tasks

[x] Implement changes in Notifications email
[x] Implement repository logic for setting flag to hinder retrieval of email notifications
[x] Implement repository logic for halting retrieval of email notifications if flag is set
[x] Implement changes in Notifications consumer to process system messages
[x] Add new email status Failed_SupressedRecipient
[x] Implement logic for picking up shipments that are sending in a longer period of time
[x] QA
[x] Test

Test cases

[x] Regression test happy path, email is sent and status logged successfully
[x] Deploy temp code for email Notification publishing a timeout service update to kafka and validate that app flag in db is set and that sending of email is paused for the duration of time

Altinn / altinn-notifications