Password Reset Emails - Githubissues

davismtl commented 5 years ago

I think we may have broken something in terms of passing ID or session to the email clicks.

Here you can see the volume of emails events relating to password resets at login: https://analytics.amplitude.com/mozilla-corp/chart/n0igskz

However, if I do a funnel of the email steps, there is a massive drop as of October 30th: https://analytics.amplitude.com/mozilla-corp/chart/00zoicz

This does not seem to be impacting all emails though.

davismtl commented 5 years ago

CC @irrationalagent

davismtl commented 5 years ago

Something else that's strange here is that if I ignore the click and just look at forgot_submit > reset_password email

there seems to be a big spike in conversions.

https://analytics.amplitude.com/mozilla-corp/chart/ejheqn5/edit/6a5vt6d

(note to self, we should align event naming around forgot vs reset)

davismtl commented 5 years ago

Note that if we look purely at the volume events, nothing seems to be off.

philbooth commented 5 years ago

...there is a massive drop as of October 30th...

Coincides with FxA train 123 fwiw. I'll dig into it on Monday.

philbooth commented 5 years ago

Obvious candidate being metrics context changes. We stopped sending it in train 123:

mozilla/fxa-content-server#6639

philbooth commented 5 years ago

Example event flow from local dev:

Content server:

{"op":"amplitudeEvent","event_type":"fxa_login - forgot_pwd","time":1543488869076,"device_id":"b993c4cf137c484eb7e3b164c60f59ac","session_id":1543488852908,"app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","ua_browser":"Firefox","ua_version":"65.0","utm_campaign":"fx-welcome","utm_content":"fx-activate","utm_medium":"email"}}
{"op":"amplitudeEvent","event_type":"fxa_login - forgot_submit","time":1543488875102,"device_id":"b993c4cf137c484eb7e3b164c60f59ac","session_id":1543488852908,"app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","ua_browser":"Firefox","ua_version":"65.0","utm_campaign":"fx-welcome","utm_content":"fx-activate","utm_medium":"email"}}
{"op":"amplitudeEvent","event_type":"fxa_email - click","time":1543488881947,"user_id":"92edc9c2522745d29c96d99daab0f759","device_id":"c159ad0f348f4885b027ae7613433d8a","session_id":1543488852908,"app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{"email_type":"reset_password","email_provider":"other"},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","ua_browser":"Firefox","ua_version":"65.0","utm_campaign":"fx-forgot-password","utm_content":"fx-reset-password","utm_medium":"email"}}
{"op":"amplitudeEvent","event_type":"fxa_pref - view","time":1543488889609,"user_id":"92edc9c2522745d29c96d99daab0f759","device_id":"c159ad0f348f4885b027ae7613433d8a","session_id":1543488852908,"app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","ua_browser":"Firefox","ua_version":"65.0","utm_campaign":"fx-forgot-password","utm_content":"fx-reset-password","utm_medium":"email"}}

Auth server:

{"op":"amplitudeEvent","event_type":"fxa_email - sent","time":1543488875186,"user_id":"92edc9c2522745d29c96d99daab0f759","session_id":1543488852908,"app_version":"126","language":"en","event_properties":{"email_type":"reset_password","email_provider":"other","email_sender":"ses","email_service":"fxa-auth-server","email_template":"recoveryEmail","email_version":1},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","sync_device_count":0,"sync_active_devices_day":0,"sync_active_devices_week":0,"sync_active_devices_month":0}}
{"op":"amplitudeEvent","event_type":"fxa_email - sent","time":1543488889112,"user_id":"92edc9c2522745d29c96d99daab0f759","session_id":1543488852908,"app_version":"126","language":"en","event_properties":{"email_type":"reset_password","email_provider":"other","email_sender":"ses","email_service":"fxa-auth-server","email_template":"passwordResetEmail","email_version":1},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","sync_device_count":0,"sync_active_devices_day":0,"sync_active_devices_week":0,"sync_active_devices_month":0}}
{"op":"amplitudeEvent","event_type":"fxa_login - forgot_complete","time":1543488889285,"user_id":"92edc9c2522745d29c96d99daab0f759","device_id":"b993c4cf137c484eb7e3b164c60f59ac","session_id":1543488852908,"app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","ua_browser":"Firefox","ua_version":"65.0","sync_device_count":0,"sync_active_devices_day":0,"sync_active_devices_week":0,"sync_active_devices_month":0}}
{"op":"amplitudeEvent","event_type":"fxa_login - complete","time":1543488889286,"user_id":"92edc9c2522745d29c96d99daab0f759","device_id":"b993c4cf137c484eb7e3b164c60f59ac","session_id":1543488852908,"app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{},"user_properties":{"flow_id":"ddb4d3455676fc74125b0e90f744b9563c9f855a1501f270028a01d6721e30c7","ua_browser":"Firefox","ua_version":"65.0","sync_device_count":0,"sync_active_devices_day":0,"sync_active_devices_week":0,"sync_active_devices_month":0}}
{"op":"amplitudeEvent","event_type":"fxa_activity - cert_signed","time":1543488889385,"user_id":"92edc9c2522745d29c96d99daab0f759","app_version":"126","language":"en","os_name":"Mac OS X","os_version":"10.13","event_properties":{},"user_properties":{"ua_browser":"Firefox","ua_version":"65.0","sync_device_count":0,"sync_active_devices_day":0,"sync_active_devices_week":0,"sync_active_devices_month":0}}

Splicing those into chronological order and pulling out the pertinent columns gives us this:

timestamp	server	event	user id	device id	session id
...69076	content	fxa_login - forgot_pwd	b993c4cf...	...52908
...75102	content	fxa_login - forgot_submit	b993c4cf...	...52908
...75186	auth	fxa_email - sent	92edc9c2...	...52908
...81947	content	fxa_email - click	92edc9c2...	c159ad0f...	...52908
...89112	auth	fxa_email - sent	92edc9c2...	...52908
...89285	auth	fxa_login - forgot_complete	92edc9c2...	b993c4cf...	...52908
...89286	auth	fxa_login - complete	92edc9c2...	b993c4cf...	...52908
...89385	auth	fxa_activity - cert_signed	92edc9c2...
...89609	content	fxa_pref - view	92edc9c2...	c159ad0f...	...52908

So presumably the route cause is that device id on the click event, even though user id and session id are consistent. I don't see anything in the resume token for propagating device id, so if we add that hopefully it should fix this.

philbooth commented 5 years ago

(or maybe it's the missing device id on the sent events, either way I'll try to fix both)

philbooth commented 5 years ago

An auth server patch to train-126 may fix this, ref: mozilla/fxa-auth-server#2763.

If not, I have a content server patch ready for train-127 that will definitely fix it, ref:

https://github.com/mozilla/fxa-content-server/compare/pb/activity-109?expand=1

philbooth commented 5 years ago

Train 126 went out yesterday evening, let's keep an eye on the charts from this issue today/tomorrow to see whether it's fixed. (/cc @davismtl @irrationalagent)

philbooth commented 5 years ago

Early signs are that https://github.com/mozilla/fxa-auth-server/pull/2763 may have made a tiny improvement in the click rate, but hasn't really made a dent in the massive drop that this issue was opened for. I'll PR the content server change for train 127.

mozilla / fxa-activity-metrics

Password Reset Emails #109