Add analytics to help inform decision making

alex-ketch commented 5 years ago

In order to make informed product and marketing decisions we'd like to have more data to go on. A few key factors for us are:

to respect people's privacy
avoid hosting it ourselves and incurring server management responsibilities
if possible, support other open-source initiatives

With this in mind, here are a few solutions that could satisfy our criteria.

Fathom

✅ Open Source
✅ Privacy oriented
✅ Pleasant UI
❌ Limited insights (see demo). Lacking things like visitor device type (mobile, tablet, desktop), browser type, location, etc. I don't think it would make for a very useful option on Stencila Hub

Countly

✅ Open Source
✅ Broad feature set
✅ Can handle crash reports too if needed
✅ Pleasant UI
✅ Retroactive goal/funnel analytics so long as events have been captured
❓ Not sure about pricing, would need to reach out to them to get a concrete number

Matomo

✅ Open Source
✅ Broad feature set
❌ Goal tracking does not work on historical data
❌ Relatively conservative quotas (team size, data retention) on the base plan

I believe both Countly and Matomo support A/B testing, and are pretty equal in terms of feature parity.

Heap

✅✅✅ Captures almost all events automatically, and gives flexibility to refine analytics/marketing strategy over time
✅ Generous free tier with very loosely enforced limits
✅ Pleasant UI
❌ Not open source
❌ Not as feature rich as Countly or Matomo

Google Analytics

✅ Many marketers are familiar with Google Analytics
✅ "Free"
❌ Beyond broad metadata capture, each event has to be manually defined and triggered (things like button clicks, form completion, etc.)
❌ Not as feature rich as Countly or Matomo

I'd like to get some opinions on the above mentioned options, proposals for alternative solutions, as well as any other consideration for how we approach capturing data (things we want omit, messaging around asking for consent, etc).

beneboy commented 5 years ago

Captures almost all events automatically, and gives flexibility to refine analytics/marketing strategy over time

I know this was what attracted me to Heap when we discussed it in NZ. Thinking about it some more – what is the overhead of this? In terms of browser lag and sending more data back to the server on everything that happens.

I'm probably worried about nothing and @alex-ketch you mentioned you used it before and I guess this was never an issue?

alex-ketch commented 5 years ago

@beneboy To be honest, I haven't measured the impact, but that's because I never noticed anything in terms of overhead, slowdown, or SEO penalization. Here's a link from their docs regarding specs and impact.

alex-ketch commented 5 years ago

Came across another good candidate, Simple Analytics, that's similar to Fathom. Not open source, but privacy focused and provides a bit more features (demo).

alex-ketch commented 4 years ago

I think a new contender might be the winner, https://github.com/PostHog/posthog Has auto-event capturing similar to Heap, and is open source.

stencila / website