Define a product team and practice

[choldgraf]

Context

Currently, our engineering team is focused around Site Reliability Engineering. They deploy open source tools into production for the services we run, but they do not focus on building new tools and end-user enhancements. This is because our primary strategic priorities are to build a robust, scalable, reliable, efficient service, using pre-existing open source tools.

However, there are many cases where new development would be impactful for our communities, and some cases where it has been explicitly requested or where we've agreed to it in contracts. We are currently doing this work in an ad-hoc fashion with bits and pieces of the SRE team's time, but this is not sustainable or scalable.

- [ ] Decide on the major questions 2i2c should answer to decide whether to engage in product development
- [ ] Make a decision on a short-term and long-term plan for product development
- [ ] ...define next steps based on the above

Proposal

We should define what a "Product development team" looks like at 2i2c, how it relates to the Site Reliability Engineering team, and how we'll bring new development activities into our practices. This might mean defining different management / reporting structures, new kinds of team roles with different skillsets, and new sustainability models for this team.

Our goal should be to have the team structures, processes, and policies to sustainably and efficiently develop and improve technology that our SRE team can deploy into production.

A few things to think about:

• Define goals, responsibilities, and backlog
• Define staffing and sustainability model
• Define management structure
• Define relationship and personnel transfer with the SRE team
• Hire or move our first engineers into this team

Examples of our or open source projects that need more product management / development

• The Binder project: has a lot of opportunity for funding and development, but currently has no dedicated team or funding, even though many of us are maintainers there. e.g. https://github.com/2i2c-org/meta/issues/531
• The Executable Books / MyST projects: Chris is one of the leads of these projects and they need more funding, we could participate in development cycles there somehow. e.g. https://github.com/2i2c-org/meta/issues/365
• The Pangeo Forge project: Needs product leadership and future development, may be a way to make our cloud services a better value by helping communities get data into the cloud. e.g.: https://github.com/2i2c-org/leads/issues/129
• Publishing workflows like the ones described in https://github.com/2i2c-org/meta/issues/542 would need more product development on the integration / glue side from us.

[damianavila]

• Define relationship and personnel transfer with the SRE team
• Hire or move our first engineers into this team

These are particularly important points where a "rotation" between SRE and the Product teams might expose our engs to different stuff so they are not burnt nor bored about always doing the same stuff and also get an early exposure to Product things so they can efficiently deploy them when they are using the SRE hat.

[choldgraf]

@damianavila I totally agree - here's a nice chapter on team overload in the Google SRE guide.

I think there are two parts to this:

1. We need to have the capacity to constantly re-invest in our own SRE processes to reduce operational overload. Google's SRE docs have a nice page about this.
2. We need to have team processes that intentionally move people into and out of a "high stress" aspect of the job. I think a great first step is what you describe: have a development team that can operate more at its own pace, an SRE team that is a bit more "reactive" to outages etc, and a way to move folks back and forth between them.

[damianavila]

From the last comment in https://github.com/2i2c-org/meta/issues/365#issuecomment-1488552566:

The biggest remaining question for 2i2c is: what role do we want to have in Executable Books, given that I am one of its PIs (and the perceived lead of the project).