Impact evaluation KPIs for Leadership + Advocacy

[OpenMatt]

Goal

Provide more detail on impact evaluation for Leadership + Advocacy.

• How are we going to think about and measure external impact on our key hills going forward? (literacy, inclusion, privacy)
• Show progress / next steps / thinking behind how we're setting targets for Advocacy and Leadership

  Roles

• PM: @OpenMatt
• Responsible: @adamlofting
• Accountable: @davidascher
• Consulted: Executive Team @msurman @chrislarry33 @aplohman

  Timelines

Key milestone for this work is the Feb 4 board meeting

Detail

• The board needs detail about our theory so they can critique it or buy in to the theory. Offer it at a holistic level.
• Note: this doesn’t necessarily mean baseline numbers. But they will want to get a sense of levels of ambition.
• Emphasis on impact evaluation. We need the other half of Adam’s story. Have a theory on the impact evaluation.
• The three red boxes (privacy, literacy, inclusion) are the key things to have a story on. What is the theory behind the statements?

  Advocacy

• Framing context on Advocacy side to other benchmark organizations. (e.g., EFF, etc.) \ e.g., We have lots of people on our mailing list, but what's the actual impact?

  Leadership

• Show how those leaders connect to "the movement." The grassroots.

[adamlofting]

I'm back! Looking forward to digging in on this.

[OpenMatt]

Specific deliverables needed by Feb board meeting

• [ ] x
• [ ] y
• [ ] z

Deliverable for this week:

@adamlofting will update this ticket and have a one-page brief by Jan 15

[OpenMatt]

@adamlofting feels to me like the best next step is to make sure that we turn our various conversations about this thread into a crystal clear deliverables list that the Executive Team can sign off on. Can you commit to producing a one-page brief that includes that by Friday? Sound reasonable? I assume it can bake in some of the specific work you're doing w. Advocacy around Encryption as a test-bed campaign.

[OpenMatt]

Notes and context

• @bhuvanshri is researching existing "web health" indices. (But this work may not be far enough along by board meeting.)
• Kaitlin / MLN is contracting some of the people that informed the thinking behind Network Strength
• Knight is also doing something here (@adamlofting will follow up)

[adamlofting]

Thanks for the notes @OpenMatt

:+1: Agreed re deliverables document by Friday.

Kaitlin is talking to Network Impact about network health evaluation for MSL, and OpenNews are also talking to Network Impact via Knight in their current evaluation project. Seperately to that, Knight have previously worked with Network Impact (not Mozilla).

[OpenMatt]

• @adamlofting I consolidated your two docs. Let's use this as the single document of record going forward: http://mzl.la/feb_kpi
• great work, as usual

[OpenMatt]

@adamlofting

want to circle up on next steps w. you on this. Main things are:

• 1) We need to start putting in baseline numbers where we have them (sub KPIs mostly)
• 2) And say when we are going to have baselines for ones we don't have
• 3) And start to have some opinion on targets
• by way of example: the MoCo Marketing program review (shared w. @adamlofting ) does a great job of this in a single slide.
• We could have a similar slide each for advocacy and leadership, even if numbers are rough (5000 leaders?)

[OpenMatt]

@adamlofting does that make sense?

[adamlofting]

Thanks Matt. That does make sense, but I expect I'll have follow up questions on this tomorrow. Will keep you posted.

[msurman]

Hey, @adamlofting. Good work on this so far. We've got solid bones for a board document and a deck.

A suggestion on process based on conversations w/ David a few minutes ago:

1. You own the doc
2. I own the slides
3. We try to get both board ready by end of the week

Processwise,

A. Try to get as far as you can w/ Matt's questions above by tomorrow at 7am pacific A1. I know there is a ton that will still be undone / have questions, that's fine B. I work on a slide outline on the plane tomorrow, which flies at 730 B1. I'll share this back with you at the end of the day C. We should probably talk through where we're at Wednesday, with David included

Works?

[msurman]

@adamlofting @davidascher

On content, I like all the general framing up front. Want the board to get into thinking about why and how we're going to do all this work.

In addition to the stuff above, we also need to do a next round of edits that simplify and align with the story the board's already seen. The two important slides from last time are these:

Imagine that this is what the board has seen and will want to riff off. We had an engaged 15 min convo on the impact goals columns in these slides.

They will have also read the topline goals and KPIs stuff in the business plan (which has been updated). I'm walking them through that again this week.

So, in going through an simplifying, good to use the language and structure from these two places a little more literally to organize your narrative.

[adamlofting]

This all makes sense, and process works for me. I'm getting on with this now. :+1:

[OpenMatt]

Notes from @msurman :+1:

1. What KPI do we want to use for agenda-setting?

One option is a survey of key influencers knowing the issues we're working on (w/ MoCo). The other is number of citations that reference Mozilla and our key hills.

2. What do we want the impact goal for MLN to be?

The board pushed back on 'we start to see new ideas and breakthroughs on our big issues', which isn't quite what's in the business plan but is close. They suggested that the effectiveness of the network be based on the role leaders play in shaping the agenda and rallying citizens. We need a proposal and a decision on this ASAP. And then we need to start talking about how we'll evaluate it. Chris / David: have you had a chance to discuss this? Do you have a proposal?

3. How will be baseline MLN 'alignment'?

I really like the the doc that clearly states MLN doesn't exist as a formally constituted thing. So, we're going through a process of inviting people into something new. Given this, I wonder if there is a need / opportunity to test alignment early on with a survey re: who is working on what hills? And how strongly they identify w/ Mozilla? This could be framed as a random survey of people who have worked with us or could more explicitly say we're creating the network and we want their input. Thoughts? If people like this idea, we should put it in the doc.

There are a couple of other small items to decide:

• Do we want to include revenue in our KPI dashboards? I would suggest yes.
• Do we show the Mozilla-topline list growth number in our dashboard? I vote yes.
• Should the network strength KPI include something about activity / engagement / contribution?
• Do we need both the alignment and focus parts of the network KPI?
• Do we need both the cohesion and connectivity parts of the network KPI?

It would be good to get people to weigh in on these things over the next day, especially #2. Happy to discuss verbally if that's needed. And, more importantly, happy to just go with the group decisions in most cases. I just want to get these things clear and decided.

[OpenMatt]

@adamlofting great. please update this ticket when you complete your next pass? I'm going to start working today on turning your document into slides.

[adamlofting]

Quick update:

All of the comments / edits from the word document were transferred to the Google Doc and have either been resolved or are still visible as open questions.

I've also made an update to the Network Strength KPI today (partly from the questions above and also from some initial playing around with how the numbers will connect.)

Do we need both the alignment and focus parts of the network KPI?

No, lets have alignment include Mozilla and the hills in one category

Do we need both the cohesion and connectivity parts of the network KPI?

Cohesion is important, but also gets into complex issues of network theory and generally makes the KPI much messier. I've moved this out to a seperate item called 'shape of the network' as something to monitor and work on, but we don't need to build it into the KPI calculations.

This simplifies the KPI greatly, we're now just adding up the value of the connections in the network (where the connections are weighted by their strength, influence and alignment).

There are a few open questions from this thread that I'll transfer into the doc as comments so they don't get lost.

[xmatthewx]

I'm curious about @OpenMatt's comment above:

The board pushed back on 'we start to see new ideas and breakthroughs on our big issues', ... They suggested that the effectiveness of the network be based on the role leaders play in shaping the agenda and rallying citizens.

Shouldn't an effective network also yield leaders who the shape the internet through innovative, people-first use of tech? For example, a former Open Science fellow in the healthcare domain might create a patient-centered, privacy-minded tool. This could change the landscape for millions without a measurable shift in the agenda or rally of the masses. In fact, doesn't a shift in agenda and user opinion need to manifest in tech to achieve our mission?

[hannahkane]

This ticket is a milestone that has passed. Should we roll into current or future milestone?

[adamlofting]

I think we can close this now.

An update was shipped for the board meeting, and any follow up work can be captured in new tickets.