Draft: Customer Survey Questions

[joethreepwood]

I'd like to run a user survey to get some more insights into a few issues. Plan currently would be to create this survey as a basic form and push it out to users through the Slack community and the usual channels for getting product feedback interviews.

There are some general questions in here, but two key topics I also wanted to investigate were: Do users use ETL/integration providers, such as FiveTran? What features do they find attractive when considering PostHog?

Questions below would be prefaced with a 'We will not share this data, this is anonymous' blurb.

Sharing here for any feedback or questions that others may want to add/rephrase. Tagging @marcushyett-ph, @leggetter and @paolodamico.

---

What team are you in?

• Product
• Engineering
• Data / Analytics
• Marketing
• Design
• Other (Please specify)

How are you hosting PostHog?

• We’re self-hosting.
• We’re using PostHog Cloud.

Which features most interested you about PostHog? (Choose up to 3)

• Funnels & Paths — I wanted to know where users were dropping off
• Persons & Cohorts — I wanted to understand more about my individual users
• Sessions & Recordings — I wanted to see how users interact with my product
• Trends & Retention — I wanted to analyse product data over time
• Feature Flags — I wanted to experiment or roll-out features differently
• Dashboards — I wanted to track and share information internally
• Plug-ins — I wanted to integrate PostHog with existing tools
• Other (Please specify)

“I needed technical help to deploy PostHog successfully.”

• Very true
• Somewhat true
• Somewhat false
• Very false

“I found it easy to achieve what I wanted with PostHog.”

• Very true
• Somewhat true
• Somewhat false
• Very false

“I found it easy to share findings from PostHog when needed.”

• Very true
• Somewhat true
• Somewhat false
• Very false

Does your company use a data warehouse, such as Google BigQuery or Amazon Redshift?

• Yes
• No
• Don’t know

Do you use data integration tools, such as FiveTran, Stitch or Rudderstack, to connect your data warehouse to other tools?

• Yes
• No
• Don’t know

Have you used PostHog’s plugins to integrate with your data warehouse?

• Yes
• No
• Don’t know

How would you feel if you could no longer use PostHog?

• Very disappointed
• Somewhat disappointed
• Not disappointed

Is there any other feedback you’d like to share?

• Free text field

Would you be interested in sharing further feedback in a call?

• Yes
• No

[leggetter]

The questions look good to me.

Was very surprised that ETL isn't mentioned at all in the customer interviews doc

Something about using the API and how? For example, HP wanted to front-load a lot of data.

I'm also interested in finding out:

• What technology stack our customers use; where they deploy, what programming languages they use, what platforms their apps run on (web, iOS, Android...) - so we can prioritise developer experience type work.
• What communities they're involved in - so we can work out which external communities to engage with

But understand we don't want too many questions or to shoehorn in question that feel out of place.

[joethreepwood]

@leggetter Great, thanks.

Yeah, I think we may have to drop the communities question. I'd love to know the answer too, but asking basically 'Hey, where do you hang out?' is a bit awkward in terms of getting answers and length, etc.

What's the best way to ask the stack question, in your opinion? We could do an open text field, but for getting clean data I prefer to give options where we can.

[leggetter]

See the Stack Overflow technology survey / Technology section for a good idea of the sorts of things we're looking for and options that are available.

How about:

• What platforms do you build for? (allow multiple)

  • Web
  • iOS
  • Android
  • Other (free text)

• What programming languages/runtimes do you use: (allow multiple)

  • Python
  • Java/JVM
  • Kotlin/JVM
  • C#/.NET
  • Ruby
  • JavaScript/Node.js/Deno
  • PHP
  • Go
  • Swift
  • Rust
  • Objective-C
  • Flutter
  • React Native
  • Other (free text)

• What client-side frameworks do you use? (allow multiple)

  • React.js
  • Angular
  • Vue
  • Svetle
  • Android Jetpack
  • Other (free text)

• What web frameworks do you use? (allow multiple)

  • Next.js
  • Express
  • Spring
  • Ruby on Rails
  • Django
  • Flask
  • Laravel
  • ASP.NET
  • Other (free text)

• Which of the following cloud services do you use/do you deploy to?

  • Google Cloud Platform
  • Amazon Web Service
  • Azure
  • IBM Cloud
  • None
  • Other (free text)

I'm wondering if we should also have question around data focused technologies? We've got some yes/no questions, above. But could we do a deeper dive to get specifics?

Should we sometimes have a "prefer not to say"? Some information may be considered private.

[joethreepwood]

Hm - this is a fair few questions and could impact completion rates. We'd also need to add a "Don't know" option to each one to accomodate non-technical people (or make these conditional questions for those who select 'Developer', but then we lose sight on how technical non-developers are...)

We can accomodate these questions if you think they'd be very helpful, but just want to check in on that with you first so we know this info would have a use.

[leggetter]

How about a:

• Do you code?
  • Yes
  • No

Only if the answer is Yes we show the additional questions?

[joethreepwood]

Yeah, that's what I meant by the conditional question. If the answer to the first question ('What team are you in?') = Engineering / Data then we ask these questions. If not, we skip them. We'll avoid getting an understanding of some points (e.g. what if you're a CTO/CEO and click 'Other' for the first question), but that may be the best solution.

Question still is though, is the information useful enough that we should ask these extra five questions or is it just curiousity?

[leggetter]

Question still is though, is the information useful enough that we should ask these extra five questions or is it just curiosity?

It would definitely be useful information to refer to when making decisions. For example, creating tutorials addressing how to integrate PostHog with specific technologies. The technologies that are in use also helps with community identification.

If we do a survey now I'm guessing we won't want to do another for a few months otherwise customers may be fed up with getting surveys?

You've made me realise I need to take a step back. What are the main goals of the survey? What are we going to do with the data?

[joethreepwood]

Main goals for me:

• Assess what our userbase looks like in terms of needs/technical ability
• Assess how much of a priority onboarding/activation/collaboration should be
• Assess if there's a good opportunity around ETL integrations

This will guide some of what we do after funnels, among other things

I'm happy to bring in your questions too, absolutely, just want to make sure there's value in them.

[leggetter]

If we feel the tech questions will reduce the completion rate I'm happy to wait. But if we're comfortable with them I'd appreciate us getting the insight as the data will be useful and utilized 👍

Another topic: I know we've said the data will be anonymized. But knowing whether a company is at their growth phase would also be very valuable as it would put more weight to their answers.

[paolodamico]

Just one perspective and feel free to disregard. Context that might be helpful is that I've done some surveys in the past with just ~3-5 questions and even through Slack results were very, very wanting (never more than 10). Can't get the latest numbers of our NPS survey (see https://github.com/PostHog/product-internal/issues/9) but even with a short 10-second survey in-product, response rates have been quite low too.

One of the main benefits of doing a survey is being able to have quantitative numbers, but I don't think we'll get anywhere near in number of responses for it to be representative. Even with actual product usage we still rely a bit more on intuition and qualitative feedback. My suggestion would be to use a combination of qualitative feedback and other mechanisms to get proxy data. That is, rely on customer calls and maybe reaching out individually on Slack (particularly more to users on the growth phase, OSS, compliance-heavy industries, ...) to get a sense of the story, and then trying to find quant data to validate. For instance, we can know which technologies they use based on where they ingest data from, or we can know if they've used plugins to send data to PostHog just from our own data, or which features they use the most.

An alternative approach you could consider is removing the PostHog-flavoured questions and making this more of an industry benchmark survey, asking it to people that aren't necessarily our users through some survey service. Haven't done this in the past, but might be an option.

Let me know what you think of the above and if quite happy to provide specific feedback on the questions if you do decide to move forward. Also, not sure if this might be better off in an internal repo?

[marcushyett-ph]

The questions look good to me.

I second @paolodamico's concerns around getting responses, would love to understand if you have any thoughts on how we might boost response rates? @joethreepwood @leggetter

[joethreepwood]

Thanks @paolodamico -- That's really good context, as I'd have hoped we could get a decent number of responses. If we feel that's unlikely even with a short form then maybe we'd be better off changing the approach.

There are three key questions I really want to get a grip on in this survey.

1. How technical are you?
2. Who do you share PH insights with?
3. Do you use any ETL software?

I feel like the first question is something we can intuit from other sources, so we can drop that. The remaining two will help inform some of the details around plug-ins and onboarding/collaboration I've raised recently.

I've looked at third-party surveys before and generally not been a fan. They tend to be expensive and not very focused. One approach that may get us higher response rates could be to reduce the survey to these two questions and then raise them directly with everyone who's done a user feedback call in the past. I could also raise the latter question on/after @yakkomajuri developer demos. This way we're targeting it personally to people who have shown strong engagement.

We're unlikely to get 100s of results this way, but if we can get a few 10s then we have something?

[paolodamico]

Definitely! Think that can work to get us some useful insights, I'll start asking right away as I have a bunch of usability tests / customer calls scheduled this week. You could also reach out individually to users on Slack (or on private channels) and think we'll get a ton more answers. We can document the results in an issue or spreadsheet.

Great context too on third-party surveys, thanks!

[leggetter]

would love to understand if you have any thoughts on how we might boost response rates?

I like @joethreepwood's idea of making it a follow-up to @yakkomajuri's sessions. Following up on other direct engagements - which we do seem to have a lot of - with requests to complete the survey may also help. In both these cases, I'd vote for us to have a survey/link that we share rather than capturing manually.

We could also incentivise the survey with a gift card, donation on their behalf or Amazon voucher. I've seen the latter improve response rates.

How technical are you?

I'd vote for keeping a question related to the person's hands-on tech abilities. But instead, going with "Do you code?". I feel that gives us the insight we're currently looking for (but I could be wrong?) rather than a breakdown of job titles where we have to infer if someone is "technical". What does being technical mean, anyway?.

[paolodamico]

Makes sense to have a survey link @leggetter for post-call follow-ups, but I also think it could be quite interesting to hear their thoughts during the call as it'll increase success rate and open the door for more details, so perhaps we can do both.

I guess in terms of a reward we could do something like a weekly draw to make it cost effective but also attractive, because I don't think giving a gift card of anything less than ~$20 will be attractive enough.

In terms of do you code question, this is roughly the way we've asked it in the past, so aligned there. Though two things to consider: a) we already ask this in almost every interview (as it informs our approach for each call) and b) I'm not sure how useful of a data point it'll be today, as the answer is (anecdotally) yes the vast majority of the time (with our current user base). Do you have a bit more context in what you're trying to accomplish with this question?

[posthog-bot]

This issue hasn't seen activity in two years! If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[posthog-bot]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.