Develop metrics for resourcing

[cassiemc]

Was talking to luke today and it'd be interesting to know how tools in other orgs are supported. e.g. this hypothetical: if JS fiddle has 100k users that means it gets supported by 4 people. This feels like incomplete thinking (depends on strategy too, among other things) but I need a more metrics-y head to dig into it with me, b/c the gist of using data to help drive where we allocate resources seems compelling.

[OpenMatt]

@davidascher : do you have thoughts here?

[davidascher]

Good topic.

In general, we should have "baseline" resources to resource the projects based on current users, and invest additional energy based on the strategy -- especially when it comes to building "new" stuff, that has to be based not on usage we have, but on usage we intend to have (and can justify with a strategic plan, marketing approach, etc).

It's also true that in general, we won't have hard numbers, which is why having a strategic plan that is cogent, well articulated and motivated, and agreed upon, is critical to making rational choices on where we invest.

W.r.t. thimble for example, we probably should take some time to understand where we see it going, why and how it fits in strategy, etc.

[adamlofting]

adding myself to this thread :smiley:

[hannahkane]

Agreed re: needing to better understand how Thimble fits into the overarching strategy. This document is in progress, but is pretty spare when it comes to how it fits into the big picture: http://bit.ly/1GhZkHN

Outside of Thimble, I'm not sure that usage metrics are the best way to determine resourcing. The MLN site, for example, is meant to serve a relatively targeted audience, but they are important to the ML plan and may require more support than their raw numbers would indicate.

[adamlofting]

I think what we're looking for is impact metrics rather than usage metrics, in relation to a proposed project or piece of work (or sometimes just a theory of potential impact even without hard numbers).

We're not good at this yet as an org, but this is part of the Mozilla Learning strategy, and we had a good session about it at Whistler. It will always be tricky to compare the investment to impact ratio for a mass audience product like Webmaker to a deeper engagement project like Fellowships, for example. But a rationale or prediction of impact can still help with resourcing within a given program.

For example, when comparing several different ideas for work we could do on Webmaker, # of users engaged or # of new users who remain active for 30 days might be suitable proxies for impact metrics (though these are usage metrics). And could help us decide whether to work on an iOS version of the app, or more advanced onboarding, or the occulus rift VR experience next (just examples of course :smile:)

[cassiemc]

I totally get how we decide what ideas to move forward with using numbers to make projections and would love to see more of that happening in the product plans. (Dan McKinley's presentation at Whistler was a fantastic primer.) I suspect much of that will fall into place as soon as we get some baseline numbers post-webmaker launch and Maker Parter.

Thinking about the ways I allocate resources however, I understand that some of it seems arbitrary. What I am really looking for is a bit of a process (if even loose) for making better people-allocation decisions and for sharing that process with others. Here's how I currently weigh up the decision and the pertinent facts I would ideally look for.

• How important is this piece of work to the organization, what other pieces of work depend on this, and why are those dependencies important? (impact metrics, strategy, dependencies)
• What are the imminent pressing goals and deadlines and what else is going on concurrently (timelines)
• Would / could we hire a contractor (budget)
• Can we maintain consistency (someone who has familiarity with the project) and who has the skills and interests that best match the need (people stuff)
• What is the proposed team and who'd fit that team best (inter-team planning)
• What are upcoming holidays, PTOs and in general, the design team's capacity for delivering (velocity)

I'd like to distill this question – "Why does Project X need more people than Project Y" – into a more rational, transparent and quantitative picture because that top question in bold has been the hardest to answer when justifying assignments.

Other tangents on my mind:

• How do we justify team growth and plan for it (vs. reacting)
• How do we stop calling it 'resourcing' and make it more people-centered

There are probably other ways I could improve this thinking. I welcome feedback!

cc @valianttry who's expressed interest in understanding more about resourcing process, and @flukeout who I should've tagged in the original comment.

[adamlofting]

I think that's an excellent framework / set of questions to work through. One point to add might be: Have we made commitments to 3rd parties (i.e. funders)? as that's usually a trump card in these decisions.

Just to comment on the cc for @valianttry, I think one area where there is currently ambiguity about resourcing (both engineering and design) is caused by our re-org where we no longer have an engagement team. That team were often the catch-all third bucket of work (alongside the webmaker and learning networks 'buckets'). And the design and engineering allocations still use these '3 buckets'.

There is now a specific fundraising team (whose remit wouldn't include the MozFest site for example), and other work like the CRM project which (although more engineering than design at the moment) is cross-team and is ambiguous about where those resources should come from in relation to the currently allocated buckets. I.e. when CRM work relates to learning networks, should that work come from the existing learning networks allocated team?

I'd anticipate more cross-team 'platform' type projects in the future as we try to rationalize things, so it would be good to think about how that can work. The first example on my mind is building a Fellows application form / database / process that can be re-used across teams. This could add another point to the set of questions: How many times can (or will) this piece of work be re-used by other teams or to solve similar problems?.

cc @simonwex because I think these questions work equally well for engineering.

[hannahkane]

@cassiemc - that's a fantastic list. +1 to @adamlofting's addition.

I have a suggestion for an additional question: is this an ongoing project or a finite project? Or perhaps, "What are the expected maintenance needs beyond initial implementation?"

I'm not sure that "why does Project X need more people than Project Y?" is the most useful question, though. I think of it in terms of, "What does Project X need? What does Project Y need?" If it's comparative, we may end up with neither project being properly staffed.

It seems to me we need buckets for each of the ongoing projects, and another bucket (or several) for the finite projects.

[hannahkane]

One other idea—the MLN team has been doing some interesting work to align our programs towards a common goal using SWOT analyses (https://en.wikipedia.org/wiki/SWOT_analysis).

I think it would be interesting to use the SWOT tool to align the various dev/design projects with the larger Mozilla Learning plan.

[valianttry]

@adamlofting you spent time building a relatively detailed "resource needs" spreadsheet for fundraising developer needs -- is that a template we could use for other areas so that we can scope more precisely? Right now I'm facing challenges because @shaghdoosti is onboard and her planning will shape up soon -- and will call for building / designing things... but there's no clear path to make a request. She's not really "Fundraising" and there's no such thing as "Engagement" any longer. Currently it's unclear who would PM those things that Sara needs to create, and therefore how we can plan the engineering time and heartbeats to complete that work.

^ That's meant as an example of "orphaned" resourcing -- not just complaining :)

cc @davidascher as a follow-on to our conversation last week.

[adamlofting]

@valianttry Yep, it's definitely worth doing, though that doc I had for fundraising was pretty hacky.

A better example to build from would be the Webmaker Roadmap: https://docs.google.com/spreadsheets/d/1ZvdNJLHuW0fIBt_-WcR7Ns6OhZgaMy6MI9kfjlXF-9o/edit#gid=0

It's the same principle, but better organized and mapped to releases over time (for fundraising I just totalled up all the things we currently want to do this year).

The basic model is start from a high-level plan, break that down into small enough tasks, then work with the engineers and/or designers to estimate those tasks as best as we can (having someone with technical PM skills in this process helps if it's building new software).

[hannahkane]

What are columns F through N in the Webmaker Roadmap? Are those estimates by the people in the column name, or are they estimates FOR the people in the column name? If it's the latter, I don't think this doc can work as a model, because it assumes it has resources, rather than determines resources needed.

I'll also point out that engineers and designers are needed to even estimate the work. That has been a blocker for me in coming up with a "staffing needed" list.

[xmatthewx]

The people in those columns gave an estimate of the work for each line. It's useful in theory, not always in practice.

[davidascher]

Good conversation folks. Highest level, just to clarify -- this is what program line reviews, strategic plans, and work plan deriving from those are for. The latter is a place where we have lagged in our formalism, and as a result some stuff hasn't been made as explicit as it should be. I think to get there we need to both understand the platformish aspects of a line of work (both technical and brand for example), the methods (long campaigns, iterative sprints, exploratory vs. optimizing, internally-driven, community-driven, or partnership-driven), and the skillsets needed. This is going to take time to develop as an institutional skill, so let's dig in but also be patient with each other.

With respect to the question that @valianttry raises for example, I think the work needed to support @shaghdoosti https://github.com/shaghdoosti's plan should be "downstream" from the Advocacy & Policy strategic plan, and will feed into the Advocay* working group plan which will inform 2016+. It's a complex topic because it involves both mofo and moco, and in that particular case it will likely have interesting geographic aspects, both of which reinforce the notion that we need to plan through a 6mo-1y timeframe. Specifically, if a big part of the plan is to help the indian community with their regional activism needs, we probably need people in compatible timezones to do a bunch of the daily work, and a method for coordinating that doesn't require people to be up in the middle of the night.

[cassiemc]

This issue goes hand in hand with the new process planning in #652. It's great discussion, but nothing actionable as far as I can tell, so going to close this in favor of refiling smaller issues. Please reopen if anyone disagrees!