[Framework] Platform Engineer & Application Developer Organizational Models

[tatcoo]

On the Proton team, we’re seeing patterns emerge in how our customers define responsibilities for infrastructure provisioning within their organization. We wanted to share how we’re thinking about it and hear about whether this pattern aligns with the way your organization delegates infrastructure provisioning tasks. We also have some theories about the biggest challenges organizations are running into as they optimize their model to become more agile - let us know whether these theories resonate!

What are the models we see?

Organizations need to decide who is responsible for writing, maintaining, and troubleshooting IaC templates for infra, then who will consume those templates to deploy their applications. The way they accomplish this can vary based on what skills they want to incentivize app developers to build - whether they want or need their app developers to spend time on infrastructure design, or to focus more on application-related technologies. Organizations also make different decisions in how much they want to build out a platform engineering team, which influences whether they can build custom centrally managed tools (DIY), use existing tools like AWS Proton, or decentralize their platform engineering expertise across their organization.

We’re seeing the following models:

• Centralized Provisioning Management: In this model, the responsibility for architecting, deploying, and managing infrastructure falls primarily on a centralized platform engineering team. Organizations assign enforcement of controls into a specific role who has the task as their focus, including manual reviews and process management. A variant of this model is to have the central team create fixed templates that development teams can use, but development teams cannot directly make changes.
• Platform-enabled Golden Path: In this model, platform engineers clearly lay out “preferred” standards with sane defaults and example architecture that development teams can use as-is. Sophisticated teams can implement their own customization on top of this. The platform engineering team is in charge of template updates, and maintenance responsibilities are typically shared.
• Embedded DevOps: Platform engineers (also called DevOps engineers in this model) are distributed across the organization, sitting closely with development teams and helping them define, provision and maintain their infrastructure. A central platform and/or architecture team defines standards for which resources can be used and what configurations are acceptable, but DevOps engineers decide how to use them to best suit the needs of their development team. Individual teams are responsible for maintenance, while template updates are worked on in collaboration between central platform teams and development teams.
• Decentralized DevOps: In this case, development teams fully own and define their infrastructure. They might look for samples from a shared repository to get started, but they are fully in charge of design decisions. The central team creates a review process that ensures that infrastructure deployed does not violate company standards. If the review process is pre-deployment, there are fewer risks but potential big hits to agility. If the review process is post-deployment, it gives organizations great agility (with some mild risks).

What model is ‘ideal’?

This varies by organization. Central IT teams have the highest consistency and control, but may find bottlenecks in reviews and approvals reduce their agility. Platform-enabled Golden Path and Embedded DevOps have a balance of control and ability, but implement tools that help them coordinate consistent processes across their organizations. Decentralized DevOps teams have agility and independent innovation from teams, but end up with more diverse infrastructure and may have a harder time implementing controls.

So... does this make sense? How is your organization deciding which model to use, and what pain points (if any) are you running into?

[brentryan]

I think this aligns with Cvent's adoption of IaC. We probably have flavors of everything here with a heavy push towards decentralized devops using CDK over the past 3 years.

Governance of what can be done safely and securely is the hardest part because we want to allow teams to be autonomous. We utilize cfn-lint, AWS Config and reusable shared CDK constructs to help facilitate all of this. And I find that this works well if you know what you're doing but still brings a lot of complexity and many don't understand what they're doing. The biggest challenges still being the "update" cycle of a previously deployed application. And if you need to handle cross-account or multi-region applications it becomes even more challenging.

I think most tools over focus on the simple case. For example, how do you create a single region application in a single account. There's nothing ever about developer experience, update cycles, breaking changes, database upgrades, CI/CD... This is always left up to each customer to figure out.