16 Ways to Measure Network Effects -- Andressen Horowitz

[owocki]

Update as of 2019-02-04:

Slides: https://docs.google.com/presentation/d/1z-E4ua1mhNv7vGwZtHB92llfEf4JklyJHSRR8qJc3VA/edit#slide=id.g4e17621652_0_3

Googlesheet visuals: https://docs.google.com/spreadsheets/d/1MW4q2JoY3n8KAoqJopIRIlMWRT8hFzu1GrW87wtGCio/edit#gid=373823490

---

I had a meeting with Jesse Walden ( https://www.linkedin.com/in/jessewalden ) of a16z crypto a couple weeks ago. He asked me to read https://a16z.com/2018/12/13/16-metrics-network-effects/ .

They invest in companies that have

1. a cohort of users that love them.
2. network effects

We definitely fit in (1), but we need to do some data sleuthing on the network effects stuff.

From this blog post he sent me, we need to know the following. We should find a way to triage this data report and create something thats representative of the project's numbers, so we can go back to him and get advice from a16z crypto on areas to focus in Q1.

• [x] #1 Organic vs. paid users What percentage of your new users are organic?

• [ ] #2 Sources of traffic As the network grows, how much traffic/transactions on the network are generated internally, arising from the network itself vs. from external sources. (I dont think this applies for us until there are other standard bounties networks that generate traffic for us).

• [x] #3 Time series of paid CAC How much do you need to spend to acquire supply?

• [ ] #4 Prevalence of multi-tenanting How many of your users also use other similar services? How many users are active on similar services?

• [ ] #5 Switching or multi-homing costs How easy is it for users to join a new (and even a non-existent) network? How much value can users get as a new user from joining a different network?

• [x] #6 User retention cohorts Is your user retention improving for newer cohorts?

• [x] #7 Core action retention cohorts Is retention, as defined by users taking a core action for the product, improving for newer cohorts?

• [x] #8 Dollar retention & paid user retention cohorts Are newer cohorts retaining better on a dollar basis, for every given time period, than older cohorts?

• [x] #9 Retention by location/geography Are participants in the oldest markets — for businesses with local network effects — better retained, than those in newer markets?

• [x] #10 Power user curves (aka L7 & L30 charts) Are users shifting to the right side of the power user curve? In other words, are they becoming more engaged over time?

• [x] #11 Match rate (aka utilization rate, success rate, etc.) How successfully can the two sides of the marketplace find each other?

• [ ] #12 Market depth Is there enough supply and does it fit users’ needs?

• [x] #13 Time to find a match (or inventory turnover, or days to turn) How long does it take for supply and demand to match?

• [ ] #14 Concentration or fragmentation of supply and demand How concentrated is the marketplace on the supply and demand sides?

• [ ] #15 Pricing power How much are you able to charge for your product? What would your customers be willing to pay to stay on the network?

• [x] #16 Unit economics How is the business doing, basically?

• [ ] fill out qualitative questions best we can, or "kick the can down the road"

• [ ] retention by coding skillset

• [x] active funder power users - do funder consistently come back month over month?

• [x] funder dashboard fix

• [x] dean ramadan email list

• [x] email top of the funnel

• [x] take a look at active email subscribers data and see if need any massaging

• [x] take a look at what happens very long term week 50, where do we stop attritioning

• [x] get a list of what we consider lurkers from the retention side

[PixelantDesign]

Great list! Curious to see where we stand on these. Some seem like they might be qualitative (4 & 5).

[owocki]

For the qualitative ones, I think we can just provide a paragraph or two of how we think about them as a starting off point.

[frankchen07]

just a heads up this is in progress, not sitting around

[frankchen07]

#1 Organic vs. paid users What percentage of your new users are organic?

Google Analytics reports ~20% of new users and ~18% of visits being organic from Jan 2018 to Jan 2019, but because GA has a unique definition of "users", this metric has questionable accuracy due to potential double counting (if a user comes back on a different browser, etc, they would be counted twice).

#3 Time series of paid CAC How much do you need to spend to acquire supply?

https://docs.google.com/spreadsheets/d/175TW6Exc5Qw-wcljMXW51HoLJIDUlT6OWlm6Q76oKVE/edit#gid=861886020

The $6-$8 CAC previously reported is not completely true, and we shouldn't be anchoring off of it. Of the paid acquisition channels we were going for, facebook seemed to be the most effective. Our data from 2018 July to 2018 November shows that over time the cost per lead was ~$23, ~$9.50, ~$8.50, to $14 - there doesn't seem to be data in September. Facebook ads have been stopped as of 2018 December.

Twitter and Stackoverflow yielded even worse results and higher cost per lead, so those pursuits were discontinued as well.

The cost per lead numbers are worsened by the fact that these dollar calculations are only to get a sign-up - we weren't tracking to see if the sign-ups converted and did a bounty.

Dean has provided Facebook login creds to see if we can connect Metabase data to get actual bounty completions off the CPC. My hunch is that if just generating signups/leads is already high, it'll be even higher once we factor in who did a bounty, and also by country (India was much cheaper), so I would say priority on setting that integration up is lower.

**#6 User retention cohorts Is your user retention improving for newer cohorts? and

7 Core action retention cohorts

Is retention, as defined by users taking a core action for the product, improving for newer cohorts?**

Retention chart is defined as cohorts who join, and then perform a contributor or funder related action 0, 1, 2, 3, etc months later.

Funders

1. Absolute acquisition of funders has held steady throughout 2018, but absolute numbers are low (15-20 funders per month), with the holidays (2018-12) yielding a large decrease.

2. Newer cohort retention is generally the same from in 2018, with the odd month of July having a higher than normal retention rate as compared to months before (older cohorts). The deviation of retention percentages can change quickly due to the small absolute number of funders onboarded per month.

3. Retention seems to make a recovery within months (usually around 3-4 weeks). This may suggest that funder cadence is every 3-4 weeks.

Takeaways:

1. We need additional onboarding of funders.

2. Consistency for funders is key. We want to see retention dropping less as the months go out, and as we hit the asymptote, we want that base asymptote to be higher. How can we tighten the loop on funding issues? (Oh, we have that in our first sprint! :) )

Contributors

1. Absolute acquisition of contributors has increased in 2018. Early 2018 we averaged in the 5-600 users per month, and the last three months we've experienced > 1k, and even almost up to 2k of new users per monthly cohort.

2. Retention of newer cohorts seems to drop off similarly to older cohorts (50% within the first month after joining), and eventually asymptotes around 1%.

3. Digging in further, it seems the drop off occurs within the month that contributors join. The retention curve states within the first 1-2 weeks it seems they do less contributor actions as they go on, but it may be because contribution lasts > 1-2 weeks. Extending the retention curve to see if we see a another peak is in store. (see #8 & #9)

Takeaways:

1. We have good absolute growth of contributors in the past three months, as compared to quarters before.

2. What does the high dropoff in month 0 / week 1-2 for contributors equate to in terms of the engagement loop?

3. Segmentation by behavior or by other separation to figure what causes that 50% drop off in the first month is critical to understand.

#8 Dollar retention & paid user retention cohorts Are newer cohorts retaining better on a dollar basis, for every given time period, than older cohorts?

A Metabase query is up for this, but because we don't have revenue, I'd like to double check these calculations to ensure I'm understanding them correctly. They may not be necessarily relevant in our case.

#8a Segment by behavioral cohorts For users who did XYZ, are newer cohorts retaining better than older cohorts? For users who did XYZ, how do they compare against users who did not do XYZ?

• Next step from #6 & #7.

#9 Retention by location/geography Are participants in the oldest markets — for businesses with local network effects — better retained, than those in newer markets?

• Hm, unsure what "oldest" markets mean here. From #6 and #7, I've segmented by country as well.

#10 Power user curves (Contributor, Funder) Are users shifting to the right side of the power user curve? In other words, are they becoming more engaged over time?

• Over time, newer cohorts of contributors have shifted slightly to the right of the power users curve (meaning more contributors ~5-8% of them are active > 7+ days of the week), but the classic "power" smile has not formed (no power users yet).

• Over time, newer cohorts of funders have shifted towards being active 14-18 days of the week, but the absolute number of funders who are actually power using beyond single digit days is fairly low (1-3 funders). Although the power user curve doesn't quite form a smile, these 1-2 funders across most months in 2018 can be considered "power funders."

#11 Match rate (aka utilization rate, success rate, etc.) How successfully can the two sides of the marketplace find each other?

#12 Market depth Is there enough supply and does it fit users' needs?

#13 Time to find a match (or inventory turnover, or days to turn) How long does it take for supply and demand to match?

#14 Concentration or fragmentation of supply and demand How concentrated is the marketplace on the supply and demand sides?

#15 Pricing power How much are you able to charge for your product? What would your customers be willing to pay to stay on the network?

#16 Unit economics How is the business doing, basically?

Dashboard as of 2018-01-10:

• Funder LTV: assuming a 10% margin (cut), $94.66
• Contributor LTV: assuming a 10% margin (cut), $1.27

I'd add caution here that in actuality, our LTV is effectively $0 (for bounties) because we are in the process of validating / testing the proposition of a % cut. So the calculations above are theoretical.

[vs77bb]

1 Organic vs. paid users

What percentage of your new users are organic?

Google Analytics reports ~20% of new users and ~18% of visits being organic from Jan 2018 to Jan 2019, but because GA has a unique definition of "users", this metric has questionable accuracy due to potential double counting (if a user comes back on a different browser, etc, they would be counted twice).

Given we're not doing any marketing spend right now, 100% of our new users are organic, right? The problem is this isn't huge growth in absolute terms thus far, though it's not bad. Not sure how GA is coming up with the 20% figure.

[vs77bb]

Funder LTV: assuming a 10% margin (cut), $94.66 Contributor LTV: assuming a 10% margin (cut), $1.27

Most of our CAC has been aimed at the contributor, traditionally, given we want to find folks who will do a great job. These numbers start to paint a picture in the other direction -- where we should be aiming to find funders who are willing to pay for bounties.

This side, in OSS, may be hard to find. Yet it's where the LTV lies.

[owocki]

Google Analytics reports ~20% of new users and ~18% of visits being organic from Jan 2018

hmm this metric seems low. i think you mean that 20% of these users were from organic search. whereas organic acquisition is all users we didnt pay for!

Organic Acquisition channels == Anything but paid search. So actually I think the number is closer to 78% per my reading of the acquisition stats in GA.

The $6-$8 CAC previously reported is not completely true, and we shouldn't be anchoring off of it.

I agree that the CAC for all of 2018 was much higher than $8.

I guess the question is -- what CAC do we report? We tested a lot of different targets/creatives over time, and I think $6 CAC is the best we got. It's probably safe to assume we'd launch new paid campaigns on customer segments, and with marketing that targeted our most efficient campaigns.

we weren't tracking to see if the sign-ups converted and did a bounty.

I think we should be able to do this moving forward.. We track the utm variables in the DB now.

We need additional onboarding of funders.

Maybe its worth scheduling a brainstorming here?

Retention of newer cohorts seems to drop off similarly to older cohorts (50% within the first month after joining), and eventually asymptotes around 1%.

Super interesting.. So for every 100 new users, we eventually have one 1 user who stays active consistently forever.. (er... not forever, but for a while)

What does the high dropoff in month 0 / week 1-2 for contributors equate to in terms of the engagement loop?

Worth a brainstorm here too.

Over time, newer cohorts of funders have shifted towards being active 14-18 days of the week

Do u mean 14-18 days of the month ?

[frankchen07]

78% per my reading of the acquisition stats in GA

This is really confusing to me. I see what you're saying about 'organic search'. 78% could be a closer, but the way Google Analytics tracks the term 'users' is technically not unique. I'll see if I can confirm the rough 78% number by patching together Dean's aggregated data, and user join dates from Gitcoin. edit update: Doing some back of the napkin math using Dean's results on Facebook acquisitions and our newly acquired unique (no thanks GA) users, comes out to around 80.5%.

I guess the question is what CAC do we report?

I would recommend reporting CACs by channels/cohort, maybe only Facebook and by quarter? Because they're variable and we've cut off marketing, I don't think reporting one averaged metric makes sense.

We track the utm variables in the DB now.

Yep, I actually have some queries in Metabase up using that data. Specifically, I don't think we can (or have yet) back-filled that data for the past (2017 and most of 2018).

scheduling a brainstorming here?

Done.

Super interesting.. So for every 100 new users, we eventually have one 1 user who stays active consistently forever.. (er... not forever, but for a while)

Yeah, roughly. This is the asymptote of the retention curve. For every cohort, the goal is to not have as many people drop off at every stage, and hopefully have a higher baseline.

cohorts of funders have shifted towards being active 14-18 days of the week

Yes, you're right, week -> month. Critical typo 🤦‍♂️

[owocki]

This is really confusing to me. I see what you're saying about 'organic search'.

Happy to jam. (though it sounds like you maybe came to a conclusion from the rest of the paragraph above?)

I would recommend reporting CACs by channels/cohort, maybe only Facebook and by quarter? Because they're variable and we've cut off marketing, I don't think reporting one averaged metric makes sense.

this is a good idea.. if i give you access to the gdoc for doing projections, can you put some of the cohorts into it? (tracking here https://github.com/gitcoinco/data-ops/issues/37 )

[owocki]

@frankchen07 on our call today, lets figure out how to make the results of your data analysis consumable by a16z crypto.