search

PostHog / posthog

🦔 PostHog provides open-source product analytics, session recording, feature flagging and A/B testing that you can self-host.
https://posthog.com
Other
20.56k stars 1.22k forks source link

Experiment diagnostics: advise what to do #22101

Open jurajmajerik opened 4 months ago

jurajmajerik commented 4 months ago

Feature request

Is your feature request related to a problem?

We now have this checklist, which is what they see

image

But I think they don’t know how to fix it, and reasonable, because we don’t have any details on how to fix. I think we should add links to each of these, pointing to an FAQ detail page that explains what to do in this case?

Describe the solution you'd like

In this case, the customer was using Segment without the full set of features. More here: https://posthog.com/docs/libraries/segment#enabling-all-features-via-segment-with-your-website

Besides Segment, it would be helpful to compile a list of any other cases when this can happen, add a new section to the FAQ, and link to it from the checklist.

Describe alternatives you've considered

Additional context

Debug info

- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
jspauld commented 3 months ago

Here's what I see:

image

and I have no idea why. Plenty of events have come in that include the flag. Here's what I see if I switch to the feature flags view:

image
jurajmajerik commented 3 months ago

Hey Paul, it's possible this was a caching issue. I've inspected your experiment and saw that now you have just one item from the list outstanding ("Events with control variant received"). To make sure you're seeing the most up-to-date result, go to the triple dot menu on the top and select "Refresh experiment result".