Dashboard for User & Project Metrics

[KastanDay]

1. North start metric: documents per project (or even tokens per project).
2. North start metric: conversations per project.
3. Total number of projects (over time)
4. ⭐️ Number of unique users per project.
5. ⭐️ Number of projects per user.
6. Also add the cumlative.
7. ⭐️ Per project, number of days since last interaction?
8. Avg session duration, or number of chat messages per session?
9. Pageview per user.
10. Per user, number of projects interacted with.Stretch goal:* What was the last user interaction (or screen recording) before they dropoff for >30 days?

[KastanDay]

New metric: Admins vs non-admins.

• Per conversation, was it initiated by a User or Admin?
• Per course, what percentage of conversations were initiated by a User or Admin?

Average number of "turns" per conversation. Example 4 turn conversation:

1. User: "What is this HW?"
2. Bot: "It's this... xyz."
3. User: "No I meant this"
4. Bot: "Oh ok it's this".

[96feilin96]

New Metrics:

1. Active admin vs active users

2. Bounce rate = (Single-page visits / Total visits) × 100%

3. Cumulative graphs for everything

4. Accuracy of response, user satisfaction --- add satisfaction

5. Admin side of uiuc chat – ask for access

6. Usage patterns= what are the peak hours or peak times

7. Customer segmentation 2- admins and users

8. Market the product ? survey of professors/ student who already use it

[KastanDay]

⭐️ == WORK FOR YOU ALL 👉 == WORK FOR KASTAN

⭐️ Retention per URL? Per project, number of days since last interaction?

✅ Per user email, number of days since last interaction? -- good enough with retention chart. Retention per user. ✅ Active user - track every time a user logs in to a specific course/project. e.g. Find users who viewed course within 7 days. ✅ bounce rate - session duration after going directly to a page... ✅ Number of unique users per project. - unique user who has pageview on pagename...

👉 /chat feedback from user - thumbs up / report problem 👉 Project created (Properites: Project_id, person_id) 👉 Admins updated (Properties: number of admins, list of admin person_ids, and project_id) 👉 Users updated (Properties: Number of users, list of user IDs, project_id, is_public_project)

[KastanDay]

Finishing work

A/B test the new landing page:

• Bounce rate // Conversion (new users sign up // returning users click on ANYTHING).
• 7 day user retention (return rate)

Clean Dashboard of metrics on Posthog

• Lots of metrics in "Product analytics" https://us.posthog.com/project/46448/insights

[KastanDay]

@KastanDay to do: New metric I want to track: e2e latency of time to first token. From sending a "Chat" to "first token response".