Exception Investigation: notifications_utils.recipients:InvalidPhoneError

[ccostino]

This is one of the errors we've seen captured in New Relic that we'd like to dig into and understand, if not also resolve.

Error message: The string supplied did not seem to be a phone number. Exception: notifications_utils.recipients:InvalidPhoneError

Traceback (most recent call last):
File "/home/vcap/deps/0/bin/celery", line 8, in <module>
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/__main__.py", line 15, in main
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bin/celery.py", line 236, in main
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py", line 1078, in main
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py", line 783, in invoke
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bin/base.py", line 135, in caller
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bin/worker.py", line 356, in worker
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/worker/worker.py", line 202, in start
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bootsteps.py", line 116, in start
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bootsteps.py", line 365, in start
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/concurrency/base.py", line 130, in start
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/concurrency/prefork.py", line 109, in on_start
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/concurrency/asynpool.py", line 464, in __init__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py", line 1045, in __init__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/concurrency/asynpool.py", line 482, in _create_worker_process
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py", line 1157, in _create_worker_process
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/process.py", line 120, in start
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/context.py", line 331, in _Popen
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/popen_fork.py", line 22, in __init__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/popen_fork.py", line 77, in _launch
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/process.py", line 323, in _bootstrap
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/process.py", line 110, in run
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py", line 291, in __call__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py", line 361, in workloop
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/app/trace.py", line 651, in fast_trace_task
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/app/trace.py", line 453, in trace_task
File "/home/vcap/app/app/__init__.py", line 441, in __call__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/newrelic/hooks/application_celery.py", line 123, in wrapper
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/app/trace.py", line 736, in __protected_call__
File "/home/vcap/app/app/celery/tasks.py", line 205, in save_sms
File "/home/vcap/app/app/notifications/process_notifications.py", line 110, in persist_notification
File "/home/vcap/app/notifications_utils/recipients.py", line 624, in validate_phone_number

Implementation Sketch and Acceptance Criteria

• [ ] Look through our log history at https://logs.fr.cloud.gov/ and see if any patterns emerge that might indicate what conditions cause this
• [ ] Try and replicate the error, if possible
• [ ] Follow up in this issue with any findings and possible remediation steps

Security Considerations

• Be mindful of not sharing sensitive information.
• We'd like to reduce the errors that occur in the application where possible to maintain application stability and performance, and reduce the chances that the application ends up in an error or invalid state.

[terrazoon]

Using the new log-debugging and csv file download capabilities, we were able to determine these errors were caused by one or two users unexpectedly including commas in the custom fields for their messages. This causes the auto-generated "one-off" csv files to be unparsable, and the app ends up confused about what the phone number is.

A separate issue was created to address a fix for the commas, so this ticket should be for the investigation only and can be closed

[ccostino]

Thank you, @terrazoon!