Visualize new account creations associated with the Organic email using SSOe and MPI data

sdickson-gt commented 9 months ago

Issue Description

As a stakeholder in Login.gov adoption, I need to see qualitative data regarding how many users created Login.gov and ID.me accounts after receiving the Organic email so that I can determine the effectiveness of the email campaign and make informed decisions.

Each dataset provides different insights into user behavior.

DataDog has authentication data for mobile and SSOe but only for VA.gov. And we can only readily access metrics for emails point forward from when configuring DataDog for a specific user group.
IAM SSOe Splunk can provide historical authentications for VA.gov and other applications, but does not include authentications with VA.gov mobile.
IAM MPI credential export will have the source of truth for when users first introduce a new credential to VA. This is only available as a monthly extract from MPI.

Tasks

[x] Configure a SSOe Splunk with a query for first authn date by user and csp for the Organic segment
[x] Configure a SSOe Splunk with a query for total distinct user authns by csp for the Organic segment
[x] Configure Domo with a datafeed to pull Splunk data
[x] Create a Domo dataset with the set of users in the Organic segment and joined with the MPI Credential dataset
[x] Create Domo visualizations

Acceptance Criteria

[x] Domo Dashboard updated with a visualization of total distinct user authentications by CSP since the test date and indicating SSOe as source.
[x] Domo Dashboard updated with a visualization of first user authentication by CSP since the test date and indicating SSOe as source.
[x] Domo Dashboard updated with a visualization of credential added date for Login.gov and ID.me and indicating MPI as source. This should include all historical data.

sdickson-gt commented 8 months ago

After reviewing data from the different sources, MPI appears to be the most accurate and most complete for reporting newly added credentials.

The numbers for the "new user" data from DataDog are lower vs SSOe and MPI - still TBD, but this may be due to DD not creating "new user" log entry if users are already known to VA.gov with another credential.

The numbers for the SSOe data look accurate, but SSOe data is missing new accounts created via SiS, which are significant (somewhere around 20% give or take???).

https://va-gov.domo.com/page/923236635?userId=1480079782

lmorris3 commented 8 months ago

Closing as done.

department-of-veterans-affairs / va.gov-team