[Collaboration] Insight Discussions

[alexkim205]

This issue provides context about what we (@clarkus @paolodamico @alexkim205) decided to tackle first in the Collaboration Milestone.

Notes from call are in Google Docs.

Insight Discussions

Question - How do you not lose context when you're sharing PostHog insights/dashboards internally and externally?

Answer - Insight/dashboard-level discussion threads.

This feature would add a discussion thread to each saved insight/dashboard that would allow internal team members to comment and cross-tag each other to engage in conversation. Historical context will not be lost and allows for insight contexts to change with time.

Today's answer - Insight-level annotations today don't encourage discussion and aren't very discoverable. Toyed with the idea of making annotations more collaborative, but realized annotations is solving a very different problem.

Breakdown

• Fully baked in notification system
  • Email and in-app notifications as MVP
• "Get team members" API and selector
• Real time message channeling
  • Client and server socket logic
  • Setup ASGI server (Daphne)
  • Setup django-channels
• Add comments into saved insight model
• Message panel and GUI
• Global app notification system
  • Might have to think about how this clashes with our current toast notification system.
• Create comment CRUD

Action Items

• [ ] @alexkim205 Basic wireframes by Wednesday standup (11/24)
• [ ] @alexkim205 Explore engineering constraints by Wednesday standup (11/24)

[paolodamico]

Thanks for putting this together @alexkim205! I think it's worth creating two issues in the main repo that detail the two main avenues that we came up with (discussions & sharing), and document there product specs (feature description, scope & wireframes).

Also, I would suggest if you could have the wireframes by Monday/Tuesday so we can have a quick round of feedback.

[clarkus]

I think this is on the right track, but I see opportunities to separate areas of work slightly differently. Fundamentally this breaks down into these feature areas. I've tried to elaborate on short and long term considerations for each. Thoughts?

• Notifications
  • Fundamentally, some system that pushes alerts for important events to one or more channels (in-product alert, email, sms, slack message, etc).
  • Secondarily, a preferences system that allows a user to configure the channels, frequency, etc to make notifications work well for their needs. For example, I only want to see a digest email on Monday mornings instead of seeing 1 email for every event.
  • Longer term, if the scale of notification becomes sufficiently large, a centralized place to manage visibility, preferences, etc. Some kind of inbox or other feature dedicated to managing communication within PostHog.
  • Toasts are for contextual, low priority messaging directly related to actions the user has taken. Notifications, or alerts, are a higher priority notification that offers one or more related actions. These are generated by the product, other users, etc, but not by the current user.
• Sharing
  • Very much related to permalinks, but we'll need to know how sharing works with project permissions. For example, if I share an insight from a private project with a non-member of that project, which permissions take priority?
  • Fundamentally, we need to push a permalink to a specific channel (email, in-app, etc). Secondarily, a user might want to add context to this share action -a simple message that clarifies why the item is being shared.
  • Sharing doesn't have to be between two users. What if I as a user want to opt into a recurring notification for a specific dashboard or insight. For example, I want to subscribe to a weekly email for my KPI dashboard. I also wan to subscribe my manager so he can track my KPI performance.
• Comments / Discussions
  • This can get gnarly, so framing the problem can really help here.
  • How many levels of discussion are supported? Can I respond to a message?
  • What visibility options are available to comments? Can I send a private comment to one other user? Can I make a note just for myself?
  • What objects can be tagged or mentioned? For example, @-mentioning another user or directly mentioning a related insight by its name, id, etc. Can I mention a project?
  • How does this work with annotations?
  • What formatting options are available? Is this simple text, markdown, rich editing, etc?
  • How long-lived are comments? Can they be archived? Deleted?
  • Longer term, comments can often move into a lightweight task management experience. For example, a user adds a comment asking why a value appears inaccurate and mentions their teammate. They can also mark that comment as assigned to the mentioned teammate such that the priority of the comment changes. This also adds some structure that allows for

[alexkim205]

This is awesome! I love how you broke it down from MVP -> short term -> long term.

Notifications

• I agree with this order of needs but I propose switching the order of secondary (preferences system) and longer term (centralized notifications place) since having an inbox may indirectly solve the problem that a preferences system is trying to address.

Sharing

• For example, if I share an insight from a private project with a non-member of that project, which permissions take priority?

• I think permalink permissions should be as flexible as possible by default, with the option to restrict it if needed. My gut tells me this would also encourage more collaboration. This tightly couples permissions rule to the context of each link/insight/dashboard instead of constricting functionality to project-level permissions.

• Google Docs does this pretty well where the default permissions for a newly created link is that anyone with the link can access the doc (read-only if you're not a user, and r/w if you're logged in as a user in the org); however, the user has the option to change the rules if they see fit (restricting to org users, with specific people, or public to anyone with the link).

• This would also mean we need to define clearer what a (1) private insight looks to the public and (2) private insight looks to internal users.

• Sharing doesn't have to be between two users. What if I as a user want to opt into a recurring notification for a specific dashboard or insight. For example, I want to subscribe to a weekly email for my KPI dashboard. I also wan to subscribe my manager so he can track my KPI performance.

• Self-subscribing + subscribing others is a really awesome feature that I think belongs as a stretch goal for now because there're scaling concerns that we need to consider before implementation. Suppose all 100 employees of a company subscribe to a main product dashboard which contains 5 insights. That's 500 subscriptions in total that Django has to push out notifications to, multiplied by N number of channels to notify, every time any one of those insights is modified.

• I think a sustainable MVP for now would be only allow subscribing other people and cap the number of subscriptions per insight/dashboard to 10 or so.

Collaboration

Agreed, this is the one that can blow up into a bunch of features so I think it's important to prioritize/deprioritize a few things you brought up here.

In scope (MVP for now)

• One level of discussion.
• You can respond to a message, in the same way you can "Quote reply" in Github.
• Tag other internal users
• Markdown
• Annotations - for now completely separated from annotations
  • @paolodamico do you think there's a considerable amount of overlap btwn discussions and annotations?

Out of scope (for later)

• Send private comments. Not sure how much value sending private comments to other users in our app be, but who knows? 🤷
• Tag all the things (insights, groups, projects, dashboards, events, etc.)
• Archive/delete/pin comments
• Tagging with task priority

[mariusandra]

• Real time message channeling
  • Client and server socket logic
  • Setup ASGI server (Daphne)
  • Setup django-channels

Having brought up the real time communication issue myself in a standup, I definitely think we should do this. Modern webapps are "instant", and polling every page every second for changes on everything is very inefficient. Just think of Github. The experience is so much nicer when you can see replies and edits instantly.

However, I suggest not underestimating the scope here. Setting up ASGI is already a big project, considering we are a self hosted platform with various types of deployments everywhere. It'll require a lot of testing and coordination with the platform team as well. Ideally they would even own it.

I'd also weigh the pros and cons of this approach (seems solid for me, tbh) against a more "microservices" approach, where the real time connections are handled in the plugin-server, or a new messaging-server, and routed accordingly by nginx. As additional context: we'd love to be able to route HTTP requests to the plugin server and back. This would unlock plugin webhooks (onRequest) and frontend plugins (plugins could serve bundled JS). Perhaps there's a common solution here?

That said, if we can do this in django, we probably should.

[neilkakkar]

This is great, Alex!

A couple of points I wanted to explore further: (edit: each point its own comment because its getting too long)

1. I want to challenge the assumption that

   Users deserve a rich and modern chat experience that updates/notifies users in real time.

Take a thought experiment here: You have an Insight you've created, and now you want to share context about it. You do it either in Github (not really real-time, can tag people, notifications are emails), or Slack (real time, immediate, no reloads to see what people are saying, notifications come in Slack itself).

What are the kinds of discussions you'd want to have here?

For example, the kind of things I'd expect to see on the above insight are: (1) Decision made about tracking one top level metric. (2) Any changes to actions that make up this metric. (3) Long-form comments about people mentioning why they don't think this is the right metric to use / etc. etc. (4) Links to other insights providing drill-downs / exploration on another dimension of the same data. Like, discoveries broken down by versions - which tells us version X is doing much better. (5) Questions like: Why did this metric shoot down after correlation was released? 😝 (or even better, someone doing the analysis and just writing the answer. The question maybe belongs elsewhere - not 100% sure here) .

What I wouldn't want to / expect to see are comments like: (1) Stream of consciousness threads like: "oh, I think correlation borked something". (2) "@mariusandra did you break something on the app?". (3) ???

This is because the latter don't do a good job of preserving context. Rather, they make it unparseable & unweidly.

I'd argue that the discussion on an insight is more like a GitHub issue than a Slack thread. This is because:

1. What you say is persisted over lifetime of an insight, so it's more like a thread of context vs real-time discussions on a specific point.
2. PostHog is more like GitHub than Slack: It's not always open, not running in the background, you open PostHog when you want something done (like finding an answer to your questions). (This makes in-app notifications kind-of worthless).

Underlying the differences here is a core model: How we build this feature implies how users use it (and in turn, whether it turns out to be useful, or something that users abandon). To figure this out, we need to answer this core question: Are Insight Discussions like GitHub issues, or like a Slack thread? (I think the former, you probably think the latter, but if we can clear this out, say ex: how do users think about it - then we can choose a better path forward. Perhaps @paolodamico / whoever has been doing user discovery here has more insight?)

[neilkakkar]

2. In-App real-time notifications

Here's a couple of questions that I don't know the answer to, but would expect us to answer before we decide to build this. (The GitHub analogy from above helps)

No matter if we go the real-time route or pull route, think these questions are valid.

1. Am I more likely to see a notification in-app, or if it comes as an email to me?
   • My hypothesis: People open email more often than PostHog, hence the latter.
2. Am I more likely to act on a notification in-app, or if it comes as an email to me?
   • My hypothesis: People come on PostHog with an objective in mind: Answer question X, check how experiment Y is doing. real time Notifications then are more of a distraction than useful.
   • At the same time, for both these questions, having a (non-real-time) notifications section might make sense, since it preserves context in the same place (I am already on PostHog, so I'd like to see PostHog related notifications to act on, once I'm done with why I opened PostHog)
3. In-App Notifications are a solution. The problem they solve, best I understand, is (borrowing from the RFC):
   • I'm a new user, and I want to get a sense of what my teammates are doing on PostHog. Notifications provide a good way to feel the "heartbeat" of the team.
   • I'm interested in "Number of Discoveries", and I want a way of keeping up with the discussions happening on this.
   • (lmk if any other problems I'm missing)
   • The question, then is: Do we have other ways of solving both these problems?
   • My hypothesis: emails are good enough for solving the above^. So, I'd propose pushing back on implementing this until we have a better sense of how this feature is doing. Assuming I'm not missing a use-case central to notifs that can't be solved otherwise.

[posthog-contributions-bot[bot]]

This issue has 2097 words at 6 comments. Issues this long are hard to read or contribute to, and tend to take very long to reach a conclusion. Instead, why not:

1. Write some code and submit a pull request! Code wins arguments
2. Have a sync meeting to reach a conclusion
3. Create a Request for Comments and submit a PR with it to the meta repo or product internal repo

Is this issue intended to be sprawling? Consider adding label epic or sprint to indicate this.

[neilkakkar]

(sorry, PostHog bot sir)

[rcmarron]

Slight word of warning: In the past, I've been burnt/seen people be burnt adding discussion features to products that don't end up getting used. Take this with a grain of salt because those were different products, but 2 main learnings from those times were:

1) People generally don't want another communication channel for their organization outside of email/slack. So, if we're going to add discussions to posthog, we should have a really good answer to: why sharing the link in email/slack doesn't work for the user?. Other products with "successful" discussion features like Figma/Github/Google docs bring deep connections between their product and the conversation. You're collaborating about a specific design component/line of code/sentence, and you need the context from the broader project to effectively collaborate. I'd challenge us to think more about why the user really needs to have the discussion in Posthog instead of in email/slack. Have any customers made specific requests around wanting discussions in Posthog?

2) Doing discussions right is a lot of work. Much of this has been covered above, but I want to make sure we don't underestimate how much work it takes to make a "good" discussions feature. I haven't looked into what open source components are available these days, but in the past, I've seen sprints disappear trying to get text editors with @mention autocompletes, wysiwyg markdown editors to be "right". Also features around "notification preferences" etc. get messy.

The path I've seen happen is: 1) decide we should add discussion to a product 2) build a basic discussion 3) see pretty low adoption 4) spent too much time adding features like "@mention" or notifications to get people to use it 5) realize our reasoning for "why someone should discuss this here" wasn't good enough to overcome the stickiness of email/slack.

And I want to make sure we don't go down this same path.

[alexkim205]

Really appreciate everyone's input here and super glad we're discussing these things before implementing. I think it's getting clearer that rebuilding Slack/Github threads in PH isn't the most impactful thing we can do at this point, nor would it make sense for our users. It seems more likely that our users will defer conversations to chat apps that already work, and context might not be shared in a predictable/consistent way in-app.

Taking a step back from the (notifications + discussions) solutions above, it might be helpful to reframe the problem we're trying to solve from "How do we get people talking in PH?" to "How do we get more people to use PH regularly?" It looks like solving this problem of visibility has more compounding benefits than a notifications system / chat approach would, since we'd be able to capture a larger audience of users. More user activity = more demand for collaboration = understanding how our users want to collaborate.

Based off of @neilkakkar and @rcmarron's feedback, I think we can table the discussions/notifications work for now in favor of other ways to bring users into the app. Here are a few suggestions that've been thrown around already (shamelessly echoing other peoples' ideas here). Note that there's quite some overlap with #7245.

• Invite colleague button
• People currently viewing insight indicator
• Robust annotations on insights and dashboards that let you tell stories
• PSA Slack/Teams/etc. bot for various different channels that users can interact with
• Weekly digest email bot
• Add yours here...

Also tagging @kpthatsme since you probably have some better ideas about user acquisition/backlinking.

[paolodamico]

Alright this issue is huge so dispensing with pleasantries and only replying to key points. Still think this may be worth doing.

• Agreed with @neilkakkar this is more like GH comm. You write stuff to preserve context, not for real-time communication.
• Even without considering the above, I don’t think doing a whole ASGI real-time thing is worth it in the MVP (marginal improvement for the user for a lot of work).
• To @rcmarron’s point. 100% aligned, I think the point of discussion is preserving knowledge at the source (like we do with GH issues), not have a conversation (and hence what I think it’s worth exploring).

We’ll have a discussion on Monday in our standup anyways, so we can just talk there and later update this issue for visibility.

[alexkim205]

Closing this one for brevity and continuing convo in main design doc: #7282