Proposal to change signup flow to get better data on users

[andyvan-ph]

Problem

During signup, we ask people:

• What is your role?
• Where did you hear about us?

Both are optional, but "What is your role?" has a drop down menu of options, while "Where did you hear about us?" is a freeform text input.

As a result:

• "What is your role?" has a response rate of 66%
• "Where did you hear about us?" has a response rate of... 6.6%

This means we get very poor quality data on where people hear about us

Solution

We should keep both questions optional, but add a dropdown option to "Where did you hear about us?".

Based on the responses we do get (see this insight) I'd suggest something like:

• Search engine (Google, DDG, etc.)
• Friend or co-worker recommended
• Read a blog post or newsletter
• Online forum (Reddit, Hacker News, Slack etc.)
• Twitter
• GitHub
• Other

Hypothesis

Adding these dropdown options will:

1. Massively increase response rates from ~6% to ~50% or more
2. Will have no negative impact on conversion because it's still an optional field
3. Give us a better indicator on what marketing activity is working / where to focus energy

Additional suggestion

• Add 'Marketing' as a role option: I have a strong suspicion most people who select 'Other' on the role question are probably marketing or content people. It would be useful if we could confirm this by adding this as an option.

[jamesefhawkins]

seems reasonable. will mean we lose out on more subtle insight, but doing this for a bit would be quite interesting to see if it makes trends more obvious as they're noisy right now

maybe worth adding youtube as it's own thing given Ian's sstuff on there?

[andyvan-ph]

RE: YouTube, we could but:

1. I think we're going to dial back YouTube in Q3
2. It's one of a few we could add but it would make the list kind of long.

Alternatively, we could do 'Other (please specify)' and have a text input, but feels like a lot of faff for the user.

[raquelmsmith]

It kinda feels like we should be able to get this info from the traffic sources/referrers. Are we not able to?

[andyvan-ph]

Yes and no. There's a bit of an issue around the merging of source data I'm going to raise with the product analytics team, but any referral is data "last touch" in nature. This would help us capture more ephemeral influences and answer questions like:

• Trends around word of mouth
• Where 'Direct' referrals come from (~30 to 35% of website traffic is direct)

[charlescook-ph]

It looks like we changed this to a drop down menu when I was out. I think this was a mistake, and we should change it back to a free text field:

• It limits the variety of responses we collect - people are not clicking 'Other' and typing in something, and the responses to the question show this if you look at the last couple of weeks. Before we used to get a long tail of random answers, different websites, newsletters etc. people had found us on, which were useful nuggets of info. I was happy to manually sift through these and present them each week as it was interesting to read, even if it was a bit manual.
• I think a lot of people are just clicking on the top option by default, polluting the data. Historically 'friend recommendation' and variants on that were 20-30% of responses we received, but if you look at last week, that's now over 50%. I'm not convinced we've suddenly had a massive upsurge in people recommending us.

(These reasons are why having a free text field is good practice

If we're trying to solve for getting more data, a better solution is probably to have a mandatory free text field, but I appreciate we are trying to balance conversion to signup with collecting data (/not annoying engineers).

While this particular change has improved the completion rate to 20%, I think the data we are now receiving is poor quality, so I'd at least suggest we revert to the previous format for now.

[jamesefhawkins]

how much do we need the "role" field?

I'd argue the "where are people coming from" question is way more important right now, and thus perhaps if we add just a free text area optional field and don't ask the role, then we may find a higher % complete it

[charlescook-ph]

how much do we need the "role" field?

I think @camerondeleone @simfish85 and @joethreepwood rely on it quite a bit for scoring leads, onboarding etc. (but correct me if I'm wrong!)

[andyvan-ph]

While this particular change has improved the completion rate to 20%, I think the data we are now receiving is poor quality, so I'd at least suggest we revert to the previous format for now.

Could we improve / change the order of the options? I notice we don't have Google in there at all, atm.

Anyhoo, you feel strongly about this and I don't think anyone else does, so makes sense to change it back. I agree the quality of data isn't that much better, even if the response rate is higher.

[joethreepwood]

I think @camerondeleone @simfish85 and @joethreepwood rely on it quite a bit for scoring leads, onboarding etc. (but correct me if I'm wrong!)

This is correct. The new onboarding email proposal, for example, is the start of a longer-term plan I have to tailor email comms to a users' role. https://github.com/PostHog/meta/issues/123

role_at_organization is a very useful property to have.

The argument to change to a free text field seems strong.

[raquelmsmith]

This has actually been running as an experiment (forgot about it!), which says we get significantly more responses to the field when it's a drop-down. But if you feel like it's too low-quality I can change back to just the free text field.

[charlescook-ph]

Sounds like folks are ok with it (and I do appreciate I came in a little hot on this one...), so yes please if we could go back to free text field that would be great ❤️

[simfish85]

I am very against changing the role field to be free text as it will break our ICP person scoring in HubSpot. If need be we could add an 'Other' option in there?

[charlescook-ph]

I am very against changing the role field to be free text as it will break our ICP person scoring in HubSpot. If need be we could add an 'Other' option in there?

I don't think we're planning to change this at all, just change the 'where did you hear about us' field back to free text.

[simfish85]

I don't think we're planning to change this at all, just change the 'where did you hear about us' field back to free text.

That's cool - Joe's reply above read like it could be talking about the role field so wanted to make sure.