planetary-social / nos

nos.social social media for all of us, using nostr
https://nos.social
Mozilla Public License 2.0
117 stars 14 forks source link

[Analytics] Add analytics for search on Discover #1304

Closed setch-l closed 3 weeks ago

setch-l commented 1 month ago

Overview We want to know how many people are using search today and ideally how many people are successfully completing their search on Nos.

AC:

mplorentz commented 1 month ago

Here is some code we stubbed out while talking about this in the iOS ticket workshop: search_analytics.patch.

Tracking the number of results after pressing enter wasn't that useful because the number changes a lot, but firing an event when the user clicks through to a profile was straightforward.

martindsq commented 1 month ago

@mplorentz @setch-l Since the requirement to "track click-throughs on profiles from the list of results" is mentioned only in the second bullet point, it's unclear whether tracking the enter key for the first bullet point is necessary. I've submitted a PR (https://github.com/planetary-social/nos/pull/1379) to track the enter key for submitting that event to meet the AC, but we’re seeking clarification on whether this is required.

mplorentz commented 1 month ago

@martindsq good question. My suggestion would be to fire a "Discover Search Started" event when the first character is typed in the search field, rather than when the Return key is pressed. This still allows us to track how many users started a search vs how many found results they were interested in tapping on, and doesn't exclude those where the Return key is never tapped.

I guess I don't mind tracking when the enter key is pressed exactly but I think if we do it should be named "Return Key Pressed in Discover Search" or something to make it more clear that many searches probably happen without that event ever firing. The other alternative I can think of would be to fire an event after each key is tapped in the search screen but that might have performance implications and PostHog actually charges us per-event so let's not do that.

setch-l commented 1 month ago

@martindsq @mplorentz - re: pressing enter: My assumption with this tracking is that most people will press enter when searching and only switch behavior over time. I'm ok with Matt's suggestion to track when a character is typed instead.

The second bullet of tracking profiles clicked is to help us understand what % of searches are successful. If a user conducts a search, but doesn't click through - the implication is that the search didn't yield the results they were seeking.

We can use this difference to evaluate the performance of our search.

martindsq commented 1 month ago

@setch-l I created an insight that shows the ratio of searches / profile opened from searches: https://us.posthog.com/project/56924/insights/VEVFXeaR. It doesn't filter internal users yet so, remember to turn the switch on after doing UAT. The ratio is not meant to be treated as an exact count as the user could be opening more than one profile from the same search, or doing new searches without clearing the previous one, or edge cases like that that could be unrealistic to catch. However, I think the ratio will be very useful for analyzing trends after introducing new features or modifying the current ones. In other words, it's useful for the big picture but not meant to be overly scrutinized.