SPIKE: Identify interstitial/welcome login page options

[lauraGgit]

User story

As a member of the public, I would like to login in more quickly, so that I can get back to the the app i want to go to.

Notes

For this page: secure.login.gov/sign_up/start (when users are coming from an agency application service provider)

• What is the value to the user in this story?*

• see LG-938

• questions to answer: new users vs number logins per month.

• What are things we should consider when making this story *

Acceptance Criteria

• [x] Product team has user behavior data to make a determination
• [ ] the recommendation is secondary

Tasks to complete the story

• [x] Develop some data on helping make this decision

Definition of Done

• [ ] Pull requests meet technical definition of done
• [ ] Usability tested

[lauraGgit]

• Bounce rate for initial page /sign_up/start
• duration on the 2 pages /sign_up/start and /
• number of users clicking to create an account vs number of authentications
• variance among service providers - for the metrics above - focus on the top 3

[MacHu-GWU]

This is still under editing, plz wait until you can't see this message

Here's what I found:

If you click Sign In from any service provider, for example: www.usajog.gov, you will land https://secure.login.gov/sign_up/start?request_id=24a655b9-8e12-4ff3-ba50-4dbe3c280198

If you click Manage Account from https://www.login.gov/, you will land https://secure.login.gov/

Define:

• initial page from SP: secure.login.gov/sign_up/start
• initial page from Login: secure.login.gov/

Observation Data:

Observation Date: 2019-02-21

Initial page metrics: SP vs Direct Login

Metrics	https://secure.login.gov/sign_up/start	https://secure.login.gov/
Bounce Rate (last week)	9.39 %	46.42 %
Exit Rate (last week)	9.46 %	12.69%
Duration (last week)	15 sec	5 min 46 sec
Bounce Rate (last month)	9.88 %	45.93 %
Exit Rate (last month)	10.14 %	12.6 %
Duration (last month)	16 sec	6 min
New User (last week)	163K	3.9K
Old User (last week)	1600K	52K
New User (last month)	647K	20K
Old User (last month	5800K	202K

Definition for https://secure.login.gov/sign_up/start:

• New User = user comes to https://secure.login.gov/sign_up/start and then go to https://secure.login.gov/sign_up/enter_email
• Old User = user comes to https://secure.login.gov/sign_up/start and then go to https://secure.login.gov/

Definition for https://secure.login.gov/:

• New User = user comes to https://www.login.gov then go to https://secure.login.gov/, then go to https://secure.login.gov/sign_up/enter_email
• Old User = user lands to https://www.login.gov then go to https://secure.login.gov/, then trying to enter email / password (no matter succeed or not)

Data Source, Google Analytics Link:

• last week: https://analytics.google.com/analytics/web/#/report/content-engagement-flow/a48605964w119657604p125189404/_u.date00=20190210&_u.date01=20190216&_r.hlEdgeStartNode=1-1-235771201&_r.hlEdgeEndNode=2-2-638817874&_r.threshold=50&_r.engageMode=screens&_r.screen=~2F/
• last month: https://analytics.google.com/analytics/web/#/report/content-engagement-flow/a48605964w119657604p125189404/_u.date00=20190101&_u.date01=20190131&_r.hlEdgeStartNode=1-1-235771201&_r.hlEdgeEndNode=2-2-638817874&_r.threshold=50&_r.engageMode=screens&_r.screen=~2F/

secure.login.gov/sign_up/star metrics break down by SP

Time Range	Metrics	usajobs.gov	ttp.cbp.dhs.gov	sam.gov
last week	Bounce Rate	11.66	1.72	2.89
	Exit Rate	12.24	2.27	4.23
	Duration	18	9	14
last month	Bounce Rate	11.94	2.26	3.04
	Exit Rate	12.55	2.85	4.49
	Duration	18	10	14

Note: since url from same service provider are not exactly the same, it is little hard to aggregate the data by service provider in Google Analytics. So I just manually did the top 3 important customer.

[MacHu-GWU]

@lauraGgit The above comment is the relative data I found from Google Analytics.

I DON'T think to compare metrics between /sign_up/start? vs / validate the statement that: to /sign_up/start? page, replacing the sign in button with email / password input box, will increase or decrease our conversion rate. because users landing /sign_up/start? and / are different group of customers.

If we can create an experiment that redirects usajob users randomly to https://secure.login.gov/sign_up/start?request_id=usajob and https://secure.login.gov/?request_id=usajob, then we can draw some solid conclusion.

[lauraGgit]

We may have to revisit this next week when we have the request_id correctly set up. And we might not need the experiment for now.

Just to clarify - the new users are the users clicking on the https://secure.login.gov/sign_up/enter_email vs. going to the / url?

[lauraGgit]

Time Range	Metrics	usajobs.gov	ttp.cbp.dhs.gov	sam.gov
last week	New users	80k	55k	8.6K
	Old users	553K	254K	49k
last month	New users	303k	224k	28.6k
	Old users	1,508k	620k	152k

[MacHu-GWU]

@lauraGgit This is how I get New Users and Old users count from a SP, my number has like 30% difference from yours. Would you mind to tell me the detail of your method?

Google Analytics -> Customization -> Custom Report:

Configure Filter:

• page = /sign_up/enter_email (for New User), / (for Old User)
• previous page path = /sign_up/start
• source = usajobs.gov

[lauraGgit]

Since this sprint is now over, let's conclude this spike and I can discuss the methods at our meeting on Wednesday

[MacHu-GWU]

We may have to revisit this next week when we have the request_id correctly set up. And we might not need the experiment for now.

Just to clarify - the new users are the users clicking on the https://secure.login.gov/sign_up/enter_email vs. going to the / url?

Yes:

new user = /sign_up/enter_email old user = /