Send text notifications utilizing Pinpoint

[QaysarA]

Value Statement

As a VANotify consumer I want to send text messages So that text message notifications can be sent to the Veteran

Acceptance Criteria

GIVEN VANotify receives a request to send a text notification
WHEN Pinpoint is setup THEN a text message is attempted to be sent to the respective mobile phone

GIVEN VANotify receives a text request THEN VANotify passes the request to Pinpoint, and they receive it Pinpoint will try to deliver the text

GIVEN service is newly created When the service has not specified a sender from number THEN the text will be sent from the global default sender from number

GIVEN VANotify receives a text request When the service has been assigned a sender ID / from number THEN the text will be sent from that assigned sender ID / from number

GIVEN VANotify receives a text request When the request contains a specific / non-default sender ID / from number that has overridden the assigned service sender ID / from number THEN the text will be sent from that specific sender ID / from number

GIVEN VANotify receives a request to send a text notification
WHEN VANotify makes a request to Pinpoint THEN environment-specific project ID is used

GIVEN VANotify receives a request to send a text notification
WHEN VANotify makes a request to Pinpoint THEN the message type is set to transactional

Checklist

• [ ] Infra:
  • [x] Create a Pinpoint project for each of our environments - Dev, Staging, and Prod using Terraform
  • [x] Create SMS channel for the Pinpoint project(s): an SMS channel is a Pinpoint concept that is referential to marketing and a channel means a notification type like text, email
  • [ ] Request a Long Code for each of the environments UPDATE: waiting from Ike to see if long codes can be project (aka env) specific
  • [x] Modify permissions for execution role (api one) to allow pinpoint access
  • [x] If possible, define default senderId per environment _UPDATE: senderid is not used in US, so we can't set it.
• [x] logs with notification_id
• [x] metrics
• [ ] --feature flag-- UPDATE: no feature flag, just setting provider active status to false and low priority. Modify per environment when ready.
• [x] db migration
• [x] user flows test
• [x] config global sender default and other Pinpoint stuff
• [ ] investigate what happens when someone replies (right now, 2 way communication is out of scope, but does the message come back to us?)
• [ ] Can we update SMS spending limit here? Or does it have to be made via api call?
  • via Terraform, for Pinpoint and sms channel resources, there does not appear to be spending limit options. however there are messaging volume limits that can be set (it is a bit more granular on the sms channel resource end)
• [x] Per each environment for existing services, update default "sms_sender" to appropriate Pinpoint origination number, so they aren't random strings anymore. Prioritized services: Vetext, VANotify, VANotify Team
• [x] After deploy to prod:
  • [x] add prod app id to task definitions
  • [x] update default sms sender per existing services
  • [x] For Vetext service, set default sms sender to +18286786800

Assumptions

• Pinpoint is approved and available for utilization
• Pinpoint is defined as the primary provider (has the lowest priority)
• Sender ID / from number is the same thing - just different names that might be used for clarity you might also see the term "Service SMS Senders"
• We can test without long codes and AWS will utilize some shared number - but we also already have one long code from Pharmacy Use Case and AWS will prefer that
• We will have a default number for each environment so if we don't specify a number for the use case the default number can be used
• We will use Long Codes for now until we get approved for Short Codes
• Most messages sent will be transactional (as opposed to promotional)

Additional Info/Resources

• Refer to the CDS setup and config for Pinpoint link
• Refer to CDS implementation for Pinpoint PR
• We can't specify area codes we want in our Long Code requests via self-service

Resources

• Terraform aws_pinpoint_app link
• Terraform aws_pinpoint_sms_channel link
• AWS Pinpoint recommended developer guide link
• AWS Pinpoint Messages API reference link
• Pinpoint Python SDK link
  • methods of interest:
    • #get_app <-- responses contains application_id
    • #send_messages <-- requires application_id

Out of Scope

• Two-way communication
• Delivery status and research mode works
• Working on access to a Short Code

Open Questions

• Explore can we request long codes automatically via Terraform?
• As we are reading through the Pinpoint docs, we will keep in mind to explore whether we will need to create templates on the Pinpoint side. Maybe we do...maybe we don't?

[miabecker]

@QaysarA can we review what pinpoint is and how we are using it? I think it is related to the short and long codes right?

We don't have access to it yet, waiting on a Security waiver. Need admin security rights.

We have requested a short-code for Pinpoint - has to be registered with all of the telephony companies i.e. verizon, t-mobile etc. Can take up to 2 months to get the code. Only recently did we get documentation from Paras / VAs. AWS Support via VA Enterprise Cloud Support - ECSO (Enterprise Cloud Solutions Office).

[miabecker]

List of things to test in QA: Check provider list to make sure pin point is prioritized Send a text message to self one more thing I didn't here

[lingtran]

Kickoff questions:

• [x] Are we doing any additional or modifying existing rate-limits? -> No.
• [x] clarification for out-of-scope item "Delivery status and research mode works" - meaning we do not need to confirm delivery status? -> Correct. We will assume that the status will just be "sent" as usual.
• [x] enable feature flag? -> discussed with devs and decided not to do a feature flag. Instead, we can make API calls/do database migration.
• [x] confirm this is adding 'pinpoint' as new SMS provider

[ffafara-tw]

Are we doing any additional or modifying existing rate-limits?

No. However I'm curious where did this question come from - maybe we are missing something?

clarification for out-of-scope item "Delivery status and research mode works" - meaning we do not need to confirm delivery status?

Handling delivery statuses and research mode will be handled in a separate story - #273 So you might want to manually confirm in AWS console (if possible) that SMS was delivered. But passing delivery status to VANotify and subsequent processing is out of scope.

enable feature flag?

I've been changing my stance on that. Initially I wanted all the providers to be behind feature flags. This was primarily driven by the fact that we need to put settings for Twilio provider even when we do not use it, or the application won't start. However providers already have "active" flag that we can change using an API call (or via DB migrations). So my current stance is to not have feature flags as long as the provider does not interfere with the app (and startup) when not used / deactivated. I'll appreciate any feedback, thoughts, discussions on that topic.

confirm this is adding 'pinpoint' as new SMS provider

I confirm.

[lingtran]

@ffafara-tw

re: rate-limiting:

I may have misunderstood from our group discussion last week on Pinpoint, where we talked about this (straight from my notes)STORMING SESSION AROUND rate-limiting around SMS capabilities with primary focus on Pinpoint. This was towards the end of the chat. There are options to set limits within Pinpoint, but not required.

re: feature flag:

We are kind of all over the place with how we are going about it, the recipient identifier flag is an environment variable set in task defs, whereas the govdelivery client is in our config file...and also maybe we have not been great at managing the flags, for instance we still have our feature flag for GOVDELIVERY_EMAIL_CLIENT_ENABLED. Apart of the downside of managing flags, the upside is how it contextualizes features in our codebase from a product perspective. Leverage an api call or db migration might not express the same level of context. That being said, this is a simple make active true/false scenario, so either way is fine for this instance.

[lingtran]

Marisa and I combed through the documentations on both AWS and Python SDK side for Pinpoint. We did some testing in the AWS cli as well as the console. We were trying to send text via the CLI but ran into cli errors...to be continued on Monday. If successful, ducks are in a row to go! 🦆🦆🦆

[marisahoenig]

Possible question for meeting with pinpoint rep: do they know if it's possible to programmatically create long codes?

[ffafara-tw]

I don't think it is. I was looking for that in their documentation and AWS CLI docs and there wasn't anything there. We can ask them to make sure.

[miabecker]

Cool -lets ask to be sure. @Filip Fafara ffafara@thoughtworks.com I've asked Marisa, Ling and Emily to come prepared with some questions for tomorrow.

M

On Tue, Dec 15, 2020 at 5:24 PM Filip Fafara notifications@github.com wrote:

I don't think it is. I was looking for that in their documentation and AWS CLI docs and there wasn't anything there.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/department-of-veterans-affairs/notification-api/issues/251#issuecomment-745606118, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARTMDO7WHHXODCOKGMVHTZDSU7O2LANCNFSM4TPXKIEQ .

-- Maria Becker Product Manager Email maria.becker@thoughtworks.com Telephone 347-461-7592 Pronouns they/them or she/her [image: ThoughtWorks] http://www.thoughtworks.com/?utm_campaign=maria-becker-signature&utm_medium=email&utm_source=thoughtworks-email-signature-generator

[lingtran]

Things we hope to bring up during call:

• Longcodes - more explanation around this? Can we specific longcodes specific to projects? Seems to be account level setting. Can we programmatically request longcodes?
• How does 'quiet time hours' get enforced? Does this apply to api calls to Pinpoint?

FROM CALL:

• Ike to get back to us

New question:

• We assigned sender_id values according to environment specific pinpoint sms channel resources because we want to set the 'default' value. We tried a couple different value formats"+10123456789" and 10123456789. Upon deploying the change and visiting the environment-project specific sms channel, the default sender id remains unset. Is there an issue setting default sender id values via terraform?

[miabecker]

This is great! Do we need to ask anything about opt outs?

M

thought with care written with haste reading up to you

On Dec 16, 2020, at 10:44 AM, Ling Tran notifications@github.com wrote:

 Things we hope to bring up during call:

Longcodes - more explanation around this? Can we specific longcodes specific to projects? Seems to be account level setting. How does 'quiet time hours' get enforced? Does this apply to api calls to Pinpoint? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[jsmithVA]

Test plan doc (WIP) - https://docs.google.com/document/d/1XvyGqeji2u8lMCoCS-tHex3vyNTsZ-HCByFlJQOwNS8/edit

[lingtran]

Tested in dev Pinpoint integration, cross-referenced with logs. Datadog dev-pinpoint dashboard in the last four hours today (12/23): https://app.datadoghq.com/dashboard/rgg-5ee-rwf?from_ts=1608751307447&live=true&to_ts=1608765707447

It looks like celery workers stopped updating in the logs, so the last few messages I tried to send my phone never got, although the logs indicate success.

Emily and I tested without setting any default sms_sender information for service. The 'FROM_NUMBER' gets set when a service gets created without it. Initially when we tested, we did it without having Pinpoint as the primary provider nor was it active. In that scenario, the two long codes we have are alternately used.

Once activated Pinpoint and gave it a higher priority of 5 (while lowering SNS priority to 10), we were able to verify that Pinpoint was the provider used and text messages were sent. However, it appears that rather than use the specified long-code (pinpoint origination number) for dev, the messages were sent to the last number I received from the VA, which happened to be the number we assigned for staging. We will have to investigate why. We have a feeling it may be related to the 'FROM_NUMBER' usage...at this time the only things we have verified about the 'FROM_NUMBER' config var are:

• config['FROM_NUMBER’] == notification#reply_to_text == provider#send_sms(sender arg)
• config['FROM_NUMBER'] is set as default_sms_sender in a migration
• When creating a service, a ‘service sms sender’ is also created, with a default sms_sender value pulled from config[‘FROM_NUMBER’]

Attached are also Datadog graphs. There was an initial error due to access issues (and it was logged...issue has been fixed).

Remaining to do:

• We want to test (1) creating a service with a sms_sender set (2) see what happens when we reply to the message. (3) write tests with these different scenarios along with user flow tests
• We want to set the Pinpoint app id dynamically on the infra side and set it as a task definition env var. As of now to get to testing, we have set the app id as a config with a default value pointing to the dev pinpoint project.

[ffafara-tw]

It looks like celery workers stopped updating in the logs, so the last few messages I tried to send my phone never got, although the logs indicate success.

Which logs indicate success? It sounds like this might be the problem we have seen in the past where celery stops processing events. Can you check SQS queue?

Sent from my iPhone

On Dec 23, 2020, at 6:38 PM, Ling Tran notifications@github.com wrote:

It looks like celery workers stopped updating in the logs, so the last few messages I tried to send my phone never got, although the logs indicate success.

[lingtran]

end of day update:

• verified able to consistently send from assigned origination number in dev, WIP in staging (had to add app id as env var - to test tomorrow)
• in staging, made Pinpoint highest priority sms provider and is active; started with vanotify-team service in modifying its default sms sender number

[lingtran]

:) This log stream shows in staging we were able to send a SMS notification with Pinpoint given highest priority: ecs/notification-celery/9da2ee2070c3412289fdb83171857c1e

Datadog metrics for successes also captured here, note that staging.notifications.delivery.tasks.deliver_sms shows all the SMS activity, and the Pinpoint and SNS graphs show respective client activity: https://app.datadoghq.com/metric/explorer?from_ts=1609850439711&to_ts=1609864839711&live=true&page=0&is_auto=false&tile_size=m&exp_metric=staging.notifications.delivery.clients.pinpoint.success%2Cstaging.notifications.delivery.clients.pinpoint.error%2Cstaging.notifications.delivery.clients.pinpoint.request_time.count%2Cstaging.notifications.delivery.tasks.deliver_sms%2Cstaging.notifications.delivery.clients.sns.success&exp_agg=avg&exp_row_type=metric

[lingtran]

Update at the moment: dev and staging environments are set with default sms senders to respective long codes. Pinpoint has been elevated in priority over SNS as SMS client in dev and staging.

Prod project will be created upon infra deployment to production. After which, we can then grab the app id of prod project and drop it as env var on api side. Also, update default sms senders before activating Pinpoint

[lingtran]

Handy cli command to check delivered messages from specific pinpoint project/app (takes about 5 minutes for stats to update after a delivery): aws pinpoint get-application-date-range-kpi \ --application-id <app-id> \ --kpi-name txn-sms-delivered

[ffafara-tw]

Can you document that CLI command somewhere in vanotify-team repo? Maybe in documents/support folder?

On Tue, Jan 5, 2021 at 2:25 PM Ling Tran notifications@github.com wrote:

Handy cli command to check delivered messages from specific pinpoint project/app (takes about 5 minutes for stats to update after a delivery): aws pinpoint get-application-date-range-kpi \ --application-id \ --kpi-name txn-sms-delivered

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/department-of-veterans-affairs/notification-api/issues/251#issuecomment-754847180, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANNLRX6UWOJHV4CDIPSMEZDSYNRSPANCNFSM4TPXKIEQ .

-- Filip Fafara Consultant Email ffafara@thoughtworks.com Telephone +1 267 262 0689 <+1+267+262+0689> [image: ThoughtWorks] http://www.thoughtworks.com/?utm_campaign=filip-fafara-signature&utm_medium=email&utm_source=thoughtworks-email-signature-generator

[lingtran]

added user flow test for simple happy path, do not pass in personalisation in payload. do we want to test with personalisation? not sure what will be passed.

[ffafara-tw]

No need to use template with personalization.