kymr
commented
5 years ago Words
- The year was 2012
- operating a critical service at Netflix was laborious
- walking through wet sand
- Canarying was devolving into verifying endurance
- nothing broke after one week of canarying
- bouncing a rubber ball between teams
- harder yet to stop from bouncing between one another
- All of these were signs that changes were needed
- Fast forward to 2018
- We’ve invested significantly in
- Along the way
- we’ve experimented with many approaches
- inspires others to debate
- devs coordinating with the ops team on
- handing off code
- needed ongoing training
- The gap between code complete and deployed
- releases happening on the order of weeks
- Feedback went from ops
- where devs could push code themselves when needed
- off-hours production issues
- they would often defer to the ops release specialist
- they were motivated to do the day to day work
- but found it hard to prioritize automation
- others didn’t need to rely on them.
- In search of
- we took a step back
- What were we trying to accomplish
- why weren’t we being successful?
- involves a full set of responsibilities
- At an extreme
- These specialized roles create efficiencies within each segment
- while potentially creating inefficiencies across the entire life cycle
- Specialists develop expertise in a focused area
- They get more effective
- Grouping differing specialists together into
- inhibits the effectiveness of feedback loops
- inhibits the effectiveness of feedback loops
- by breaking down silos
- “Operate what you build” puts the devops principles in action
- Distributing this responsibility
- rather than externalizing it
- creates direct feedback loops and aligns incentives
- are empowered to remediate the pain
- adds significantly to
- helps to balance this out
- with the mission of
- Those centralized teams act as
- force multipliers by turning their specialized knowledge into
- Empowered with these tools in hand
- As additional tooling needs arise,
- centralized teams assess whether the needs are common
- collaborations ensue
- to warrant centralized investment
- Balancing local versus central investment
- finding novel solutions
- are worth the risk of multiple groups creating parallel solutions
- that will need to converge down the road.
- By starting well-aligned on the needs
- how common they are likely to be
- we can better match the investment
- we arrived at a model
- Full cycle developers are expected to be
- For many new-to-Netflix developers
- this means ramping up on areas
- other forms of ongoing training
- to impart this knowledge and build up these skills
- Knowledge is necessary but not sufficient
- apply engineering discipline
- They evaluate problems from a developer perspective
- what self-service tool will enable my partners to answer their questions
- without needing me to be involved
- by favoring systems-focused
- requires a mindset shift
- as the primary way that they create value
- viewing operations as a distraction
- so that they can get back to their “real job”
- At times they create software
- at other times they write test cases for that
- and still other times they automate operational aspects
- For this model to succeed
- teams must be committed to the value
- it brings and be cognizant of the costs
- Teams need to be staffed appropriately with enough headroom
- respond to partner support requests
- Time needs to be devoted to training
- Tools need to be leveraged and invested in
- Partnerships need to be fostered with centralized teams
- self-service partner support tools
- alongside business projects
- Without these, teams risk overload and burnout.
- adaptations are necessary
- won’t have the staffing to invest in
- nor will they need the complexity that Netflix’s scale requires
- it may be compelling to try them as a first pass.
- Evaluate what you need and be mindful of bringing in the least complexity necessary
- see devops topologies for an extensive list
- but has its downsides
- priority is given to a larger area of ownership
- Breadth requires both interest and aptitude in a diverse range of technologies
- focusing on becoming world class experts in a narrow field
- the need to be broad
- sometimes unfulfilling
- without requiring ongoing breadth
- we support them in finding those roles
- who value the breadth that owning the full cycle requires
- increases each developer’s cognitive load
- a team will balance more priorities
- having an on-call rotation
- flow-state type work
- teams devolve into everyone jumping in on high-interrupt work
- We don’t mandate adoption of those paved roads
- encourage adoption by ensuring
- is a far better experience than not using them
- is near impossible to achieve
- Realizing the returns on investment
- The path from 2012 to today
- whose earlier experiences motivated
- Deployments are routine and frequent
- are seeing similar benefits
- we’re cognizant that we got here by applying and learning from alternate approaches
- Want to be a part of exploring
Title
Summary
Reference