Fixing the PostHog x Customer.io Integration

joethreepwood commented 1 year ago

Context

We have an app which acts as an integration between PostHog and Customer.io, sending events from the former to the latter. It was made by Emmanuele, who was planning to make some needed fixes to the integration before he left -- but unfortunately he did not have time to start this work.

I've discussed briefly with @charlottemcginn and am raising an issue so we can define the work and so the Growth team can decide how/if to prioritize the fixes.

Customer.io Integration

We use this integration to power our Customer.io campaigns (most importantly, our G2 review requests and onboarding flows), but the integration is also available to customers.

There are some known issues with the integration not functioning correctly, as well as some requested functionality that would enable the Marketing team to leverage the app more effectively.

Known issues

We get regular errors appearing in Customer.io where attributes fail to change. This usually happens when the integration attempts to update a user profile with an email address, but that email address is already associated with a different user ID.

Requested functionality

(Minor) Make the app a two-way integration. In an absolute ideal world, we would pass metrics from Customer.io back to PostHog so we could run more comprehensive analysis. This isn't something Marketing requires, but would be a nice to have.
(Major) Make the app recognise groups/organisations, as well as projects. Currently, the app doesn't recognize groups, teams or organizations in any format - nor does it have any view of projects as separate from accounts. I believe this sometimes creates errors when we try to validate if a user has ingested events (due to them creating multiple projects). It also prevents us from creating flows which recognize if a user is paying or not (because that occurs at an org level).

Next steps

Would love to get input from Growth team on how/if we can address these issues, or how else to move forward.

raquelmsmith commented 1 year ago

This usually happens when the integration attempts to update a user profile with an email address, but that email address is already associated with a different user ID.

What would the expected behavior be in this case?

lharries commented 1 year ago

For reference, with the CDP work we'll making a new way of sending all the events to customer.io, but this won't tackle syncing persons across or updating the attributes

joethreepwood commented 1 year ago

What would the expected behavior be in this case?

Some info via the Customer.io docs on this here.

Also, to give this some scale, the last error notification we had was last week. It claimed to have detected this error 4,713 times in the last 24 hours.

I'm not 100% sure of the best or normal way to handle this, as I'm not sure how the integration functions at this point, but I'd suggest the best solution would be not trying to create a new Person in Customer.io if the email is already assigned to a person?

joethreepwood commented 1 year ago

Some additional recommendations which the Customer.io team have advised after we experienced an error with a snippet:

Jumping right into your question here, looking at a few examples in your Activity Logs for some of the recent updates to the attribute "$referring_domain", I can see that the Posthog API Integration is commonly sending the value "$direct" as an attribute update, as you can see in the few activities I screen-captured below:
Screenshot-2023-04-18-at-10-38-03-AM.png

This is the reason your Liquid logic has occasionally been directing people to "https://%24direct/login", because in the condition used in your ["login_url" Snippet](https://fly.customer.io/env/127208/snippets/7), the evaluation is to see if the attribute holds a value. This is why "$direct" is being used in rendering the Login URL if the person receiving the message had that value for the attribute at the time of sending.

I would highly recommend getting in touch with your Developers, and please ask them to review the data being sent to Customer.io via the Integration. Another curiosity I spotted while looking into those changes, for example, on "$referring_domain" updates for this [Example Profile](https://fly.customer.io/env/127208/people/e8e107018925ba8f02/activity?name=%24referring_domain&type=attribute_change). The latest attribute update has a long list of User Agent sources under the same update. This means that multiple calls were processed in the same batch by our backend systems, which we commonly see with API Requests being sent as a collection of calls or with very small intervals in between.

joethreepwood commented 1 year ago

This is an additional issue - when someone closes their account, it seems we maybe don't remove them from the relevant cohorts in Customer.io? Meant this inactive user got an email (which was deliberately sent to all affected users regardless of subscription status) even though they don't have an account and should no longer be in the cohort.

joethreepwood commented 1 year ago

Following up on this, this is definitely a major issue brand-wise. Even when we delete a user and org in PostHog, it seems we don't delete them in Customer.io. This is driving a growing number of users to reach out to us asking to be unsubscribed because they no longer use the service.

I can, on an ad-hoc and manual basis, add some additional filters to upcoming broadcasts so that we only send emails to users who have logged in over the last, say, 90 days? But that's not going to be foolproof and I have no way I can think of to do this for automatic campaigns.

Pretty significant issue.

corywatilo commented 1 year ago

Any reason we can't have a Customer.io unsubscribe button in there?

joethreepwood commented 1 year ago

Any reason we can't have a Customer.io unsubscribe button in there?

We can, and we do for 99% of emails. The exception are the rare service emails regarding outages, which we send to unsubscribed users too as we need to notify them about these.

jamesefhawkins commented 1 year ago

could we have just two lists we add people to ie:

service emails
non essential emails

then offer subscribe to both, all via customer.io?

joethreepwood commented 1 year ago

Discussed this with James. I'm going to do the above as a temporary workaround, but the issue remains that we should still really remove users from Customer.io when they are removed from PostHog. At the moment, Customer.io's list can literally only ever get bigger.

raquelmsmith commented 1 year ago

I think I will have some bandwidth to pick this up next sprint.

joethreepwood commented 1 year ago

@raquelmsmith Shall I close this out, or is this something you think we may have time for in coming sprints?

raquelmsmith commented 1 year ago

I don't have time for it right now. Since it's an app maybe @PostHog/team-pipeline would be responsible for it?

tiina303 commented 1 year ago

Maybe we can hack on it together some time later @raquelmsmith, but not on my immediate roadmap/sprint plan

jpvalery commented 12 months ago

@joethreepwood 👋

I think this might be related to https://github.com/PostHog/customerio-plugin/issues/26 so linking it here.

As discussed, here are the documentation links I mentioned:

Merging users

Objects (for Organizations/groupId)

From our discussion, it seems that updating https://github.com/PostHog/customerio-plugin/blob/main/index.ts would suffice to get these fixes out.

joethreepwood commented 6 months ago

User has also submitted this issue: https://github.com/PostHog/posthog.com/pull/8039

PostHog / meta