🌟 Analytics - Application Completion and Drop off rates

[RM-1978]

✨ Feature

New metric: We would like to know how many users complete the application process and where they drop off.

🕵️ Details

Definitions

1. Drop off rates = Users abandon the process entirely.
2. Completion rate = User successfully complete the process compared to the number of users who start.
3. Save and continue rate = User saves their session at a specific step and complete the process at a later time.

Required:

• [ ] User success rate for application submission
• [ ] User drop off rate after profile review
• [ ] Save and continue rate after profile review
• [ ] Drop off rate after adding first experience or reviewing experience
• [ ] Save and continue rate after adding first experience or reviewing experience
• [ ] Drop off rate after reviewing résumé/Career timeline
• [ ] Save and continue rate after reviewing résumé/Career timeline
• [ ] Drop off rate at review before submission

Nice to have:

• [ ] Breakdown of success & failure by step/task
• [ ] Drop off rate after Education requirement completion
• [ ] Drop off rate after Skill requirement completion

Knowing which users left and came back might allow us to breakdown the stats in interesting ways

📷 Screen shot

As discussed with Brinda, completion rates of each of the application steps pictured above would be logged then viewed using Azure tools to identify areas for usability and accessibility conformance testing.

🙋‍♀️ Proposed Implementation

This can be achieved using either application insights or Power BI

Bonus if you could filter out admin usage. In any case, the difference between people who go to people who come back is a useful metric.

Potentially adding a time stamp and identifier to the log (or similar) so we know who is leaving and how long it takes them would be useful if easily possible. (could also be broken into a separate issue if that's best)

✅ Acceptance Criteria

A set of assumptions which, when tested, verify that the feature was properly implemented.

• [ ] We can track completion/success rates for the application process
• [ ] We can track drop off rates for each step of the application process

[petertgiles]

@RM-1978 This sounds like a good task for PowerBI. In the pool_candidates table is a column submitted_steps that can tell you where each application is at. Additionally, this column is tracked in the activity_log for more detailed timestamps.

If this task won't need dev intervention, can I close this issue?

[RM-1978]

Thanks @petertgiles, I think that this solution is reasonable, but can we keep it open until I have had a chance to chat with @gobyrne and @brindasasi about it? I would like to work on building user research around these metrics for Design and eventually User Acceptance Testing with Brinda.

[gobyrne]

I moved this to the design backlog until it's actionable by dev. I've also sent a request to get the breakdowns of all draft applications based on what step they are on.

This is nowhere near as detailed as I'd like though. I wanna see how long people are active (and inactive) on each step of the application process, combined across all their sessions for a given application that they submit. We could look into how to do that.