RFC: Automated churn investigation

[joethreepwood]

I'll keep this short and sweet. By chatting with @daibhin I realised that nobody is currently owning any sort of consistent investigation when a user churns. Currently it's spread out between many individuals, who share feedback ad hoc, and there isn't a consistent method for gathering it that I'm aware of.

This is valuable feedback to have and could be done relatively discretely and painlessly.

There are two potential ways we can solve this with some automation, so I'll expand on what they are and we can decide how/if to implement.

CC relevant parties @raquelmsmith @simfish85 @camerondeleone

Option 1: Email automation

We could relatively easily use Customer.io to send an automated email to users who perform the event billing subscription cancelled, asking them to share feedback on why they cancelled. We'd probably direct them to a short form, or use landing page reporting to get pre-made responses.

To my knowledge we don't have per-product cancellation events, so if we wanted to know why someone cancelled a particular product then this may fall through the gaps. That's a disadvantage. The advantage is that we could follow up with users after the event more easily.

Option 2: User survey

We could use the PostHog user surveys system to target users in greater detail, and even run experiments over the process in a better way (ask for interview, etc).

The advantage is that we could use this to hit users with a feedback request at the point of cancellation. Disadvantage is that it's more intrusive to the process and we cannot target them with surveys after the cancellation.

Feedback wanted

Any info on best practices or strongly held opinions is appreciated. My feeling is that Option 2 would be a better process, and that I can (with probably a bit of assistance around the billing system from @raquelmsmith) set this up relatively quickly.

[simfish85]

I like it - can we do both? (To get maximum coverage)

[joethreepwood]

That would feel excessive to me personally

[daibhin]

For Option 1 it could be useful to use this suggestion from @jamesefhawkins in Slack a while back https://posthog.slack.com/archives/CT2BU33N1/p1687815678991669

There is probably also a third option...

Option 3: During cancellation flow

Certainly not the most flexible and requires the most be built but having an optional textarea / multiple choice list within the product immediately after someone cancels could be enough.

It maybe feels like Option 2 with the benefit that it more nicely fits in the flow rather than being perceived as a distraction when the user is trying to cancel, and hence be less likely to fill it out.

[camerondeleone]

I love this idea. Perhaps the move is to do something more automated from customers that have spent, say, less than $100, and then offer a call for customers over $100 in exchange for the refund. I'm happy to do these calls...

[joethreepwood]

Oh, I like that approach. We can then also test the different response rates between methods.

It sounds like the right way to do this then would be:

• Utilize user surveys to target users with a feedback prompt at the moment then churn
• Users who spend less than $100 get a text prompt asking them why they're cancelling
• Users who spend more than $100 get an interview prompt (directed to Cam) offering them a $100 refund in exchange for a call

The challenge for this would be that we don't seem able to currently target surveys based on events. 🤔

[joethreepwood]

Quick chat with Neil regarding ☝️ suggests event-based targeting won't be done for a long while yet, for infra issues.

@daibhin How big a lift would it be to build this directly in to the product as per Option 3? 🤞

[joethreepwood]

@neilkakkar You mentioned we could actually do this with surveys by setting a property. Can you let me know what's involved in that so I know whether it's a me-thing or a ask-for-help-thing? 🙏

[neilkakkar]

Whenever a user churns from a product, send an event like posthog.setPersonProperties({churned_from_flags: True}) - this sets the churned_from_flags property on the current person to True.

On surveys, create a new survey that targets churned_from_flags OR churned_from_recordings, etc. etc. - and a specific URL if you'd like. Whenever this person visits the url then, they'll be shown the survey.

It's also fine if you want to update this property on the backend instead, it just means the survey wouldn't be shown immediately, but only after the property is ingested.

[neilkakkar]

.. might be a good idea actually to create different surveys based on the churned product?

[daibhin]

Took a brief look but I haven't worked in the billing area at all so I could very well be misinformed. It looks like we capture an event when someone cancels their subscription that I'm sure could be hooked into & user properties added to.

Unfortunately it's on the backend which would mean that it's not going to be instantaneous as Neil explained. I also think that even if we set the properties that there is no way to target the survey on properties that have recently been set (correct me if I am wrong @neilkakkar) meaning the survey could be sent to all historical cancellations too.

Also having looked at the cancellation flow, it all happens outside of the product in Stripe. From my recollection redirecting to conditional pages (e.g. go here if a subscription is cancelled) isn't super easy in Stripe Billing so it might not be possible to redirect them to anything we custom build. There's also no guarantee that users return to PostHog at all after they cancel.

If Customer.io support event based email triggers that might actually be the best first version of this

[raquelmsmith]

I like the survey idea, and this is trivial in the frontend! https://github.com/PostHog/posthog/pull/16744

Will need to make the surveys for each product once we start receiving events.

Collecting these responses is one thing - having a process to deal with them is another. Each product team should own its own churn, even if we help them get set up with some tooling to help. I'm not sure what that process should be, but I feel like it should involve a webhook that posts to somewhere relevant (eg in Slack). Do survey responses trigger webhooks? 👀

[joethreepwood]

Thanks! Posting in Slack, in the Feedback channel, is probably the best starting point.

[joethreepwood]

Now that we've had an event come in (apparently someone unsubscribed from Group Analytics) I've set up a first survey for this.

@neilkakkar would you mind taking a momentary glance to ensure I've not done anything stupid, then launching it if you're happy?

[raquelmsmith]

This feels kind of... stilted?

[raquelmsmith]

I triggered unsubscribe events for the other 3 products so you should be able to make all the surveys now @joethreepwood

[joethreepwood]

Yeah, I was really trying to balance the aesthetic side because my default language would be longer. And when it's longer it looks weird.

[raquelmsmith]

How about "Why did you unsubscribe?"

cc @liyiy for design feedback here.

[joethreepwood]

Close enough!

[liyiy]

That looks good, you can also edit these surveys even while they're in progress if you want to re-word anything in the question or description text (just try to avoid renaming the survey name completely)

[joethreepwood]

this is now running and feeding in to the feedback channel

[joethreepwood]

Just adding a note here to log that, because of targeting issues with surveys, we were showing this to users on all domains. We really only want to show this within the app, so I've changed the targeting -- but surveys can't currently target multiple domains. As a result, I've created separate surveys for EU and US users.

I've also added a thank you message and 5 day wait period, as these features are now available.