V3 - Add Provider Logic for Email and SMS

[mjones-oddball]

User Story - Business Need

We want to ensure V3 has feature parity with V2. This ticket should ensure that the appropriate provider is chosen based on system configurations.

• [ ] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

User Story(ies)

As a notification platform I want the ability to send notifications from multiple providers So that I can support different digital notification channels and VA business needs

Additional Info and Resources

• Celery documentation
• https://github.com/department-of-veterans-affairs/vanotify-team/blob/master/Engineering/SPIKES/Multiple-Provider.md
• https://github.com/department-of-veterans-affairs/vanotify-team/blob/master/Engineering/SPIKES/provider_selection_flowchart.drawio.pdf

SES and Pinpoint are the default providers on all services. However, some services like VEText send through multiple providers and, as a result, we use template configs to specify provider.

The provider is determined in the following order:

1. template level
2. service level
3. priority provider logic (active, notification_type, priority #)

Engineering Checklist

Please note - it is critical we finish this work after the new year. Remaining V3 work should not be a code overhaul. Any refactoring requests must be approved by Product and Tech Lead.

• [ ] If you're unsure of the existing logic, please review existing documentation and v2 logic.
• [ ] Replace hard-coded providers here and here with logic to determine the correct provider for a given notification.
• [ ] Generalized methods for error catching and retry logging (so we can easily identify retry counts with specific notification ids and tasks)
• [ ] Add retries for Twilio here
  • [ ] Providers should throw retry or non-retryable exceptions rather than provider-specific errors
  • [ ] Define these error types and provide suggestions on how to produce them to QA
• [ ] Ensure new code uses service.research_mode from v2
• [ ] Unit tests

Acceptance Criteria

Givens

• We have a defined list of supported providers in the database, which contain a priority order
• SES is our only email provider we use currently
• Pinpoint and Twilio are the only sms providers we use currently, but Pinpoint is the provider we use by default for sms
• Services created through the portal set "email_provider_id" to SES and "sms_provider_id" to Pinpoint by default

Cases

Email - /v3/notifications/email

SES Confirm you receive an email in the following cases:

• [ ] Send email with provider defined in the template, "provider_id"
• [ ] Send email with provider defined in the service, "email_provider_id"
• [ ] Send email without a defined provider

SMS - /v3/notifications/sms

Use a valid sender for the tests below. If you try to send sms using a Twilio sender but provider is Pinpoint (or vice versa) this will fail. That logic is not altered as part of the v3 work.

Twilio Confirm you receive a text in the following cases:

• [ ] Send sms with provider defined in the template, "provider_id"
• [ ] Send sms with provider defined in the service, "sms_provider_id"

Pinpoint Confirm you receive a text in the following cases:

• [ ] Send sms with provider defined in the template, "provider_id"
• [ ] Send sms with provider defined in the service, "sms_provider_id"
• [ ] Send sms without a defined provider

QA Considerations

• [ ] Pinpoint basic errors were handled as part of #1360 but Twilio was not. So that will have to be validated with this ticket. Engineering should define what those errors are and how to trigger them. "The provider code can raise exceptions that need to be handled, possibly by retrying."
• [ ] Manually validate above cases in Perf once deployed there
• [ ] Automate this or create ticket for automation

Out of Scope

If something requires overhaul of V2 functionality to make it work for V3

[kalbfled]

@mjones-oddball @k-macmillan The "existing logic" checks 2 feature flags that I doubt are necessary anymore. Do we need to keep that? If not, I could probably replace the somewhat overly complicated "cient_to_use" function with a single database query.

[mjones-oddball]

@kalbfled what are the feature flags? I would need to know more about it before making a judgment call. CC @k-macmillan

[kalbfled]

1. PROVIDER_STRATEGIES_ENABLED
2. TEMPLATE_SERVICE_PROVIDERS_ENABLED

[mjones-oddball]

@k-macmillan interested to hear your thoughts on if we need to keep the feature flagged or not. I can't think of a reason off the top of my head.

[k-macmillan]

Will re-address with va-enp-api.