Trust microservice - confirm reminder service

[marcinkoziej]

• [ ] Add state to the microservice - we can use LevelDB (doc) for nodejs (or sqlite?)
• [ ] listen on custom supporter confirm queue (eq [cus.4.confirm.supporter]). On message, store at key action-${actionId} the whole action message. Before it, check if it already exists or if done-${actionId} exists, and if so (repeated message) don't do anything. Also store retry-${actionId} the timestamp of next retry ( action.insertedAt + 3 days for example) as well as number of the retry (we have 1st and second retry, possibly more to configure)
• [ ] Listen on additional delivery queue too (the original delivery queue is used for sync , we can make a copy eg delivery_repeater) When delivery happens, mark the record as confirmed (remove the key action-${actionId}, remove schedule key retry-${actionId} and create a key done=${actionId} = true so we know this aciton was processed.
• [ ] Create a cron job, which iterates over the level db and checks which actions need repeating - check all the keys for retry-1234 keys and check if the date is in the past - means we should repeat the action. To do this, re-insert it into the confirmation queue, and under retry-${actionId} store the date and number of next retry. If we reached maximum retries, store "never" or other special keyword. Innserting the confirm message again into the rabbitmq queue for sending emails [exchange: "", routing key: wrk.${orgId}.email.supporter]. Please add a action.fields.reminder=true

[tttp]

Hi,

might be good to have a quick chat, my concern on the design: is there a way to avoid having to store PII in a separate datastore outside of proca-server?

ie, what about:

we have two proca api:

• queue all actions that {supporter_status = unconfirmed (sorry, can't remember the proper status AND inserted_at < now - x days?)
• change status to "ghosted" of all actions that (supporter_status=unconfirmed?

lastly, I'd prefer sqlite than levelDB, but it's probably because I'm not familiar with the second (eg is this easy to install? known pitfalls?)

[tttp]

how would the bounce handling fit in the picture here? eg, is there a way to avoid sending reminders to emails that hard bounced/blocked/flagged spam?

[marcinkoziej]

In cases we want to use the repeater for Campax case (open-action: register, close-action: signature) we would have to re-queue register actions (to send the email again) but proca already dropped email from the supporter (privacy feature).

So perhaps here using encryption should be used - then the service would store only encrypted payload and repeat it.

Another problem with how to design this api to requeue only once? If i do status=confirming AND inserte_at < now - x days, it will be TRUE for all moments after exact 2 days....

[tttp]

ok, may be we need to say we can't send reminders if encryption is on/that they need to remove encryption if they want us to send reminder?

I mean, to send an email you need to have the email of the recipient in clear, there

(v2: encrypt at another time than signature, eg if inserted_at < x days ie. it's plaintext when hot and encrypted at rest)

design the api: good point: we need to give a date range, not a max date

if status = confirming and insert date between [start, end].

[marcinkoziej]

Using a service to reque also gives some more control over where the action ends up, for example, when we have Sync + Thank You on delivery exchange, you can make a mistake and re-queue messages to sync them, but also re-send all the thank you emails (this was precisely the fuckup on T4F german partner). With a separete script, you can be more precise and just insert action to Thank you queue. The confirmation queue actually does not have any other action bound to it except sending emails (no other use case)..

[marcinkoziej]

design the api: good point: we need to give a date range, not a max date

if you run this api twice, within the range, you will send emails twice. how do you make sure you run it only once a day? (reliably)

Mitigation for now

The repeater worker should remove PII from contact.* before storing to leveldb - just leave contactRef, firstName and email. Other PII is not used by the confirmation sender, so we can requeue without it.

Problem solved without any big architecture change.

[marcinkoziej]

Or I can add api:

requeueAction(ids: [123], queue: EMAIL) { count }

[marcinkoziej]

@1v4n4 please start working on this, you can start building this app (leveldb, and mechanism to listen on both queues, etc) until we finalize whether action should be stored+inserted, or a api call made to requeueAction()