BUG: GA4 Pixel GET Route Throwing ValidationErrors

MackHalliday commented 3 months ago

Description

The GA4 pixel tracking endpoint is generating 4xx errors due to various query parameter handling issues. These errors are cluttering logs and affecting tracking accuracy. The primary issues include:

Query parameters being truncated, removed, or not processed correctly.
Apple email clients stripping all query parameters from GET requests.
Validation errors due to unexpected characters or formats in the query parameters (e.g., spaces or malformed URLs).
[x] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

Steps to Reproduce

View logs here
Use body params from 400 errors to test calling endpoint GET /ga4/open-email-tracking and duplicate the error

Impact/Urgency

Medium This appears to impact a small number of GET requests. Roughly 600 for the 378,000 emails sent

Expected Behavior

[x] Ensure the system catches and logs errors when query parameters are modified or stripped
[x] Investigate and fix the URL encoding/parsing issues causing schema validation errors
[x] Endpoint gracefully returns a 200 if there is a problem with validation
[x] Update the Slide deck with the key win and a demo slide

QA Considerations

Confirm system logs a graceful error when query parameters are modified or stripped

Additional Info & Resources

See slack thread regarding the issue

npmartin-oddball commented 3 months ago

Hey team! Please add your planning poker estimate with Zenhub @coreycarvalho @cris-oddball @EvanParish @k-macmillan @kalbfled @MackHalliday @mchlwellman

mchlwellman commented 2 months ago

Here are the error types and some sample payloads:

GA4 Validation Errors


GA4 ValidationError: 'VBA Education/*/* does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads: 
    - /ga4/open-email-tracking?campaign=em_enrollment-submission-notification&campaign_id=ff7913a5-39f5-4185-907d-754d6f87554c&name=email_open&source=vanotify&medium=email&content=VBA%20Education/4266121d-f301-434b-a97f-8ec1597143e9/c
    - /ga4/open-email-tracking?campaign=ua-sco_resolution-confirmation&campaign_id=7dd3b8ac-1c31-4e35-872e-e3492e443197&name=email_open&source=vanotify&medium=email&content=VBA%20Education/4266121d-f301-434b-a97f-8ec1597143e9/d5ac6e2e
    - /ga4/open-email-tracking?campaign=em_enrollment-submission-notification&campaign_id=ff7913a5-39f5-4185-907d-754d6f87554c&name=email_open&source=vanotify&medium=email&content=VBA%20Education/4266121d-f301-434b-a97f-8ec1597143e9/df3

GA4 ValidationError: '*/*/* does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=Debt:%20New%20Benefit%20Overpayment%20-%20VBA&campaign_id=7efc2b8b-e59a-4571-a2ff-0fd70253e973&name=email_open&source=vanotify&medium=email&content=dmc/b3357476-adf1-4350-be1b-3604034665f9/efcf7164-1191-4'
    - ?campaign=Appeal%20decision%20mailed%20(Non-contested%20claims)%20-%20email&campaign_id=6f514913-0c85-4bc9-8343-9f6272574d1c&name=email_open&source=vanotify&medium=email&content=BVA/c157bd6e-6e3b-4561
    - ?campaign=QuickSubmit%20Confirmation%20MultiUserTypes&campaign_id=c2ccd60a-37f3-47f7-9a2b-afecf23f80f7&name=email_open&source=vanotify&medium=email&content=QuickSubmit/3368b0ba-63db-4c34-b90c-1c4e0ff827f3

GA4 ValidationError: '*' is a required property
Payloads: # I've checked 4, they are all the same...sometimes 'medium' is the missing property, and sometimes it is 'campaign'
    - ?campaign=20-10206%20Freedom%20of%20Information%20Act%20(FOIA)%20or%20Privacy%20Act%20(PA)%20Request%20Confirmation&campaign_id=07d76dd0-2b8f-4866-8c97-45eb2638d07e&name=email_open&source=vanotify

GA4 ValidationError: '*' was expected
Payloads:
    - HEAD https://api.va.gov/ga4/open-email-tracking?campaign=Confirmation%20of%20VA.gov%20Profile%20Information%20Changing&campaign_id=8d34e7d2-00f9-4cdf-9c34-6c5c87fe9d5b&name=email_open&source=vanot 400 0.01484
    - ?campaign=Debt:%20New%20Benefit%20Overpayment%20-%20VBA&campaign_id=7efc2b8b-e59a-4571-a2ff-0fd70253e973&name=email_open&source=vanotify&medium=ema

GA4 ValidationError: '' does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=Account%20Activated%20MultiUserTypes&campaign_id=3b01b2b7-897d-4277-b0b8-0994a313ef88&name=email_open&source=vanotify&medium=email&content=

GA4 ValidationError: 'va.gov/5bda137e-689e-4532-b3d2-2c81c0324331/* does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=Form%20526ez%20email%20confirmation&campaign_id=0c1335fc-ca67-408a-b2ae-08b91ffaceba&name=email_open&source=vanotify&medium=email&content=va.gov/5bda137e-689e-4532-b3d2-2c81c0324331/25febfaa-209

GA4 ValidationError: 'Mail Automation Services/e5af2191-4903-4426-8177-f91' does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=WebPMC%20Application%20Submitted%20&campaign_id=896208be-2a70-49e2-bb6b-7e53a47254dc&name=email_open&source=vanotify&medium=email&content=Mail%20Automation%20Services/e5af2191-4903-4426-8177-f91

GA4 ValidationError: Additional properties are not allowed (' Pen Benefits' was unexpected)
Payloads:
    - ?campaign=Direct%20Deposit%20Update%20Confirmation%20-%20Comp%20&%20Pen%20Benefits&campaign_id=36d8ba76-9a08-4dfa-a044-7fe6ec267520&name=email_open

GA4 ValidationError: '*' does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=Form%20526ez%20email%20confirmation&campaign_id=0c1335fc-ca67-408a-b2ae-08b91ffaceba&name=email_open&source=vanotify&medium=email&content=v
    - ?campaign=686-c%20Application%20Confirmation&campaign_id=9770facf-1a47-44c4-a558-968e8b696aae&name=email_open&source=vanotify&medium=email&content=va
    - ?campaign=526%20EZ%20Application%20Reminder&campaign_id=b9c6513b-efa2-45b7-9691-5c70925dc450&name=email_open&source=vanotify&medium=email&content=va.gov/5bda137e-689e-4532-b3d2-2c81c0324331/39412ea7-a138-

GA4 ValidationError: 'VA Profile' does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
    - HEAD https://api.va.gov/ga4/open-email-tracking?campaign=Auto-auto%20notify%20BAI&campaign_id=4b3436a0-d0c7-4325-91e5-9c2e30deb7f8&name=email_open&source=vanotify&medium=email&content=VA%20Profile

GA4 ValidationError: Additional properties are not allowed ('cont' was unexpected)
Payloads:
    - ?campaign=Direct%20Deposit%20Update%20Confirmation&campaign_id=bd5b7827-15c3-4d18-b97b-97aa29b0d6db&name=email_open&source=vanotify&medium=email&cont

GA4 ValidationError: 'Mail%' does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=WebPMC%20Application%20Approved&campaign_id=99c335f2-f237-491b-8fef-9f5994fd67be&name=email_open&source=vanotify&medium=email&content=Mail%25

GA4 ValidationError: 'va.' does not match '^[^/]{1,255}/[0-9a-zA-Z-]{36}/[0-9a-zA-Z-]{36}$'
Payloads:
    - ?campaign=526%20EZ%20Application%20Reminder&campaign_id=b9c6513b-efa2-45b7-9691-5c70925dc450&name=email_open&source=vanotify&medium=email&content=va.

GA4 ValidationError: Additional properties are not allowed ('amp;campaign_id', 'amp;content', 'amp;medium', 'amp;name', 'amp;source' were unexpected)
Payloads:
    - ?campaign=QuickSubmit%20Confirmation%20MultiUserTypes&amp;campaign_id=c2ccd60a-37f3-47f7-9a2b-afecf23f80f7&amp;name=email_open&amp;source=vanotify&amp;medium=email&amp;content=QuickSubmit/3368b0ba-63db-4c34-b90c-1c4e0ff827f3/3b3bdf24-05c1-4ec2-b61d-ee8ee2725fd4
    # GET https://api.va.gov/ga4/open-email-tracking?campaign=QuickSubmit%20Confirmation%20MultiUserTypes&amp;campaign_id=c2ccd60a-37f3-47f7-9a2b-afecf23f80f7&amp;name=email_open&amp;source=vanotify&amp;medium=email&amp;content=QuickSubmit/3368b0ba-63db-4c34-b90c-1c4e0ff827f3/3b3bdf24-05c1-4ec2-b61d-ee8ee2725fd4 400 0.01925

mchlwellman commented 2 months ago

The plan for fixing this is to

Change the url for the pixel to have the notification id in the route and remove query strings altogether
Change the route handler to not do any schema validation, just pass the notification id on to the celery task and always return an image
Change the celery task to look up the notificaton and build the GA4 payload from there
Fix/Add unit tests

cris-oddball commented 2 months ago

QA testing is blocked because nothing is being received when sent through Perf. The notifications (smsand email) report as delivered, but I haven't received any of them (and that part is necessary to be able to validate this work).

cris-oddball commented 2 months ago

QA PASSED

My issue earlier was that I had Postman configured to use a test key (which never sends, but always reports delivered). I switched that back to a normal key and I am back in business.

[x] TC01 - sent an email, open the email, check that the logs show a post request to google analytics.

Hard to navigate the GA console, but here we see the notification_id from the above test is recorded

Screenshot 2024-09-25 at 4.36.56 PM.jpg Screenshot_2024-09-25_at_4_39_49 PM.jpg

Note that it is capturing data from the regression being run. I have not opened that email account in a while, so there are no email opens. I believe Google is pre-fetching the data, somehow, and brought that up to Ksenia. When testing using an icloud account with that email client closed, I did not observe the behavior.

Most likely, we will want to monitor what we see in the GA4 console for a while, since we are now building the POST request in the code (instead of relying on what email clients must do with munging the GET request path params) and Google will truncate any fields without rejecting them if they are over their value limit.

Will send this up to Prod in the morning and close out this ticket.

cris-oddball commented 2 months ago

Holding this commit for another one, so closing this ticket as done.

department-of-veterans-affairs / notification-api