search

open-gitops / documents

📑 Lasting documents from the OpenGitOps project which are versioned and released together (including the GitOps Principles and Glossary)
https://opengitops.dev/
Other
426 stars 49 forks source link

Adoption Journey Document #19

Open mikecali opened 3 years ago

mikecali commented 3 years ago

Now that GitOps is getting a lot of traction, we got a lot of questions from different parties how to start their own GitOps journey. I think this warrants a dedicated document of what are the things to consider before even thinking of adopting GitOps to avoid disaster or bad taste.

Something I have discussed with my colleagues are GitOps pre-requisites and we ended up with this list.

  1. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)
  2. Automation Maturity - Infrastructure as Code at the minimum
  3. Executive sponsor
  4. Champions / CoP - a team that can ‘go first’
  5. Standardisation (can’t have 20 x Git sources of truth)
  6. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
  7. Matrix view of crawl / walk / run (ie start w infra, then apps, then network)
  8. K8s has to be on the roadmap - how do we get ready for that
  9. Start Small!

Happy to expound this and contribute if needed.

lloydchang commented 3 years ago

A suggestion is to invert your list so it begins with "Start Small!" towards bigger "Communication Strategy":

  1. Start Small!
  2. K8s has to be on the roadmap - how do we get ready for that
  3. Matrix view of crawl / walk / run (ie start w infra, then apps, then network)
  4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
  5. Standardisation (can’t have 20 x Git sources of truth)
  6. Champions / CoP - a team that can ‘go first’
  7. Executive sponsor
  8. Automation Maturity - Infrastructure as Code at the minimum
  9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)

WDYT? @mikecali @sunil390 @Opsm0nkey @scottrigby

mikecali commented 3 years ago

I agree. Looks good to me.

lloydchang commented 3 years ago

@mikecali @sunil390 @OpsM0nkey @scottrigby I recommend adding "Independent deployability" and "Rapid application deployment" to the adoption journey document as numbers 2 and 3.

Hence the list becomes:

  1. Start Small!
  2. Independent deployability
  3. Rapid application deployment
  4. K8s has to be on the roadmap - how do we get ready for that
  5. Matrix view of crawl / walk / run (ie start w infra, then apps, then network)
  6. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
  7. Standardisation (can’t have 20 x Git sources of truth)
  8. Champions / CoP - a team that can ‘go first’
  9. Executive sponsor
  10. Automation Maturity - Infrastructure as Code at the minimum
  11. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)

As references:

(2014) from https://martinfowler.com/articles/microservices.html

independently deployable by fully automated deployment machinery

(2014) from https://martinfowler.com/bliki/MicroservicePrerequisites.html

Rapid application deployment

(2014) from https://blog.cleancoder.com/uncle-bob/2014/09/19/MicroServicesAndJars.html

You don’t have to coordinate with some huge deployment effort

(2019) from https://blogs.oracle.com/javamagazine/post/design-change-tolerant-software-with-cloud-native-patterns

Finally, something I’ve already hinted at is creating smaller units of deployment. Rather than a deployment encompassing a huge portion of your ecommerce system—for example, the catalog, search engine, image service, recommendation engine, shopping cart, and payment-processing module all in one—deployments should have a far smaller scope. You can easily imagine that a new release of the image service poses far less risk to the business than something that involves payment processing. Proper componentization of your applications—or as many would call it today, a microservices-based architecture—is directly linked to the operability of digital solutions.

(2019) from https://www.oracle.com/oce/dc/assets/CONT9DD07156DE4043BF8BA54229540143FC/native/107876-ebook-cloud-native-patterns-oracle.pdf

Your goal is to release frequently, and monolithic software doesn’t allow this; too many interdependent pieces require time-consuming and complex coordination. In recent years, it’s been soundly proven that software made up of smaller, more loosely coupled and independently deployable and releasable components (often called microservices) enables a more agile release model.

(2019) from https://www.oreilly.com/library/view/monolith-to-microservices/9781492047834/ch01.html

Independent deployability is the idea that we can make a change to a microservice and deploy it into a production environment without having to utilize any other services. More importantly, it’s not just that we can do this; it’s that this is actually how you manage deployments in your system. It’s a discipline you practice for the bulk of your releases. This is a simple idea that is nonetheless complex in execution.

Tip: If there is only one thing you take out of this book, it should be this: ensure you embrace the concept of independent deployability of your microservices. Get into the habit of releasing changes to a single microservice into production without having to deploy anything else. From this, many good things will follow.

To guarantee independent deployability, we need to ensure our services are loosely coupled—in other words, we need to be able to change one service without having to change anything else. This means we need explicit, well-defined, and stable contracts between services. Some implementation choices make this difficult—the sharing of databases, for example, is especially problematic. The desire for loosely coupled services with stable interfaces guides our thinking about how we find service boundaries in the first place.

(2020) from https://www.infoq.com/news/2020/05/monolith-decomposition-newman/

focus on the outcome, not the technology, and to always remember the goal is independent deployability

(2020) from https://www.theregister.com/2020/03/04/microservices_last_resort/

I should be able to deploy that shipping service into a production environment without having to change the rest of the system. This helps to ship software more quickly and it helps our teams work in a more autonomous fashion.

(2020) from https://gotopia.tech/bookclub/episodes/moving-to-microservices-with-sam-newman-and-martin-fowler

So, you need a reason to push for microservices. What would be your top three reasons that you think are valid reasons that people should be thinking of going down that path?

Yes. Number one is independent replayability and specifically, I would say taken to the extreme would be zero downtime independent deployability. So, the ability to make a change to a piece of functionality, deploy that functionality, and do so in a way that the rest of the system doesn't need to change, so that limits the scope of each release, but also implement that deployment in such a way that I get zero downtime deployment. So, that's really useful if you think about, you know, a SaaS based business where you can't afford downtime, and that's really, really key. So, I think that's kind of the big clear winner.

(2021) from https://www.oreilly.com/library/view/design-patterns-for/9781492090700/ch01.html

For the purposes of this book, we take a bottom-up approach to defining cloud native. We look at all the characteristics of cloud native applications, across the board, by going through every stage in the life cycle of a cloud native application—including design, development, packaging, deployment, and governance. Based on those characteristics, we’ve come up with the following definition:

Cloud native is building software applications as a collection of independent, loosely coupled, business-capability-oriented services (microservices) that can run on dynamic environments (public, private, hybrid, multicloud) in an automated, scalable, resilient, manageable, and observable way.

When your time kindly permits, perhaps you can read and/or listen to the above snippets then dive deeper into the links. Thank you!

mikecali commented 3 years ago

@lloydchang The idea of the list I shared is to put a recommendation around what an organization needs to consider or lined up before getting into GitOps adoption.

My opinion is that the 2 points you added might be encapsulated under this: Matrix view of crawl/walk / run (ie start with infra, then apps, then network)?

lloydchang commented 3 years ago

Thank you @mikecali

"(ie start w infra, then apps, then network)"

Two out of the three examples, "start w infra" "and "then network", aren't in the scope of GitOps' matrix view of crawl / walk / run. They are in the scope of K8s.

By incorporating your opinion, I'm recommending that the new list looks like:

  1. Start Small!
  2. K8s has to be on the roadmap - how do we get ready for that (i.e. start w infra, then network)
  3. Matrix view of crawl / walk / run (i.e. Independent deployability, Rapid application deployment)
  4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
  5. Standardisation (can’t have 20 x Git sources of truth)
  6. Champions / CoP - a team that can ‘go first’
  7. Executive sponsor
  8. Automation Maturity - Infrastructure as Code at the minimum
  9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)

WDYT?

lloydchang commented 3 years ago

@mikecali As for:

  1. Standardisation (can’t have 20 x Git sources of truth)

I recommend the Standardisation example be changed to:

  1. Standardisation (i.e. Choose a standard from the options in Argo CD and Flux CD to structure your repositories)

For example:

Argo CD https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/#separating-config-vs-source-code-repositories

Option A. Separating Config Vs. Source Code Repositories

Flux CD https://fluxcd.io/docs/guides/repository-structure/

Ways of structuring your repositories Option B. Monorepo Option C. Repo per environment Option D. Repo per team, Option E. Repo per app

WDYT?

FWIW, the following are trivia:

• Git is flexible, and the 20 x Git repositories can be combined via git subtree, or git submodule, or Repo, or depot_tools

• One can have 20 x Git sources of truth because Git is distributed and decentralized by design, and a Git commit is immutable with a unique SHA.

Two objects colliding accidentally is exceedingly unlikely. If you had five million programmers each generating one commit per second, your chances of generating a single accidental collision before the Sun turns into a red giant and engulfs the Earth is about 50%. https://github.blog/2017-03-20-sha-1-collision-detection-on-github-com/

OpsM0nkey commented 3 years ago

Could I also recommend that we add something about considering the right skill set at the start of the journey? GitOps is the primary enabler of a software driven operating model, so the ability to think like a software engineer (or at least understand SDLC principles) is vital - i.e branching, pull requests etc... I think it's almost assumed knowledge, but there are tons on traditional infrastructure people out there there that can't comprehend the power of the pull request just yet...

lloydchang commented 3 years ago

Good point @OpsM0nkey Related, not all software engineers use pull requests (PRs). PR isn't inherit in Git at all. PR is a feature in code review tools such as GitHub, GitLab, Bitbucket, Gitea.

lloydchang commented 3 years ago

Depending on the Git implementation in use for GitOps, there may need to be a walkthrough. For example:

Basic Gerrit Walkthrough — For GitHub Users Note: This document aims to provide a concise description of the core principles of code review in Gerrit for people that were previously using Pull Requests on Github or similar concepts. Nothing in this document is meant to state that one or the other might be better, but only aims to help new users understand Gerrit more readily. We use Github as the point of comparison since it seems to be the most popular service. https://gerrit-review.googlesource.com/Documentation/intro-gerrit-walkthrough-github.html

OpsM0nkey commented 3 years ago

Good point @OpsM0nkey Related, not all software engineers use pull requests (PRs). PR isn't inherit in Git at all. PR is a feature in code review tools such as GitHub, GitLab, Bitbucket, Gitea.

It's a very good point - PRs aren't a feature of Git, but rather of the Git based platforms. But they're not just for code review (albeit, it's an important aspect) - they're a notification that a change is incoming and a discussion forum for the intended change. IMO, they're an important part of what makes GitOps so effective for scalable operations. When utilised properly, a PR takes away the need to maintain certain traditional ITIL change management practices (such as CAB) - ie the Pull Request is the Change Request, and all of its links, discussions and approvals provide the necessary assurances that a change is good to go.
That's not to take away from the main point of this thread, other than to say that understanding of software development practices (like branching, PRs) is an important consideration when adopting GitOps.

mikecali commented 3 years ago

I totally agree. Should we actually have a separate section called "GIT Best Practices" I actually started thinking about it and here is the first stub of what I thought we can add.

GIT Best practice 


 Don’t commit code as an unrecognized author. Git needs to be a source of truth and allow auditing of all actions.
 Ensure all secrets are stored encrypted. Use 3rd party secrets Management tools
 Don't commit huge files or binaries into the repo. Consider using binary repository managers.
 Use git ignore when appropriate and make it clear.
 Clean up stale branches and Repos
 Create a Branch naming convention that's scalable
 Enable security alerts and cleanup old users
 Scan for vulnerabilities and secret commits (Example AWS Keys)```
OpsM0nkey commented 3 years ago

@mikecali I like it 🙂

mikecali commented 3 years ago

@lloydchang @OpsM0nkey

I think this is really good now? I am happy with this list. Should we move this forward?

  1. Start Small!
  2. K8s has to be on the roadmap - how do we get ready for that (i.e. start w infra, then network)
  3. Matrix view of crawl / walk / run (i.e. Independent deployability, Rapid application deployment)
  4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
  5. Standardisation (can’t have 20 x Git sources of truth)
  6. Champions / CoP - a team that can ‘go first’
  7. Executive sponsor
  8. Automation Maturity - Infrastructure as Code at the minimum
  9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risks

With the addition of Git Best Practices maybe? But I am not sure if this fits in the adoption journey document.

  1. Don’t commit code as an unrecognized author. Git needs to be a source of truth and allow auditing of all actions.
  2. Ensure all secrets are stored encrypted. Use 3rd party secrets Management tools
  3. Don't commit huge files or binaries into the repo. Consider using binary repository managers.
  4. Use git ignore when appropriate and make it clear.
  5. Clean up stale branches and Repos
  6. Create a Branch naming convention that's scalable
  7. Enable security alerts and cleanup old users
  8. Scan for vulnerabilities and secret commits (Example AWS Keys)
lloydchang commented 3 years ago

@mikecali

Can

Standardisation (can’t have 20 x Git sources of truth)

change to:

Standardisation (i.e. Choose a standard from the options in Argo CD and Flux CD to structure your repositories)

for the reason at https://github.com/open-gitops/documents/issues/19#issuecomment-949388020 ?

Thank you!

mikecali commented 3 years ago

@lloydchang yes of course.

How about this?

  1. Start Small!
  2. K8s has to be on the roadmap - how do we get ready for that (i.e. start w infra, then network)
  3. Matrix view of crawl / walk / run (i.e. Independent deployability, Rapid application deployment)
  4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
  5. Standardisation (i.e. Choose a standard from the options in Argo CD and Flux CD to structure your repositories)
  6. Champions / CoP - a team that can ‘go first’
  7. Executive sponsor
  8. Automation Maturity - Infrastructure as Code at the minimum
  9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risks

@lloydchang @OpsM0nkey what do you think about this?

With the addition of Git Best Practices maybe? But I am not sure if this fits in the adoption journey document.

  1. Don’t commit code as an unrecognized author. Git needs to be a source of truth and allow auditing of all actions.
  2. Ensure all secrets are stored encrypted. Use 3rd party secrets Management tools
  3. Don't commit huge files or binaries into the repo. Consider using binary repository managers.
  4. Use git ignore when appropriate and make it clear.
  5. Clean up stale branches and Repos
  6. Create a Branch naming convention that's scalable
  7. Enable security alerts and cleanup old users
  8. Scan for vulnerabilities and secret commits (Example AWS Keys)
lloydchang commented 3 years ago

@mikecali LGTM re: Adoption Journey Document: "1. Start Small! [...] 9. Communication Strategy..."

As for Git best practices, I wouldn't create it because less is more.

There is already a plethora of Git documentation, such as:

  1. https://git-scm.com/
  2. https://github.com/git/git/tree/master/Documentation
  3. https://resources.github.com/videos/github-best-practices/
  4. https://docs.microsoft.com/learn/modules/maintain-secure-repository-github/

To @OpsM0nkey's point about the right skill set... An audience of software engineers may want "a bottom-up explanation of Git, starting with its data model and later covering the command-line interface" at

  1. https://missing.csail.mit.edu/2020/version-control/

Git is flexible and context dependent.

Many Git best practices will become out of date as software and organizations change. For example, https://nvie.com/posts/a-successful-git-branching-model/ was conceived in 2010, with a note of reflection in 2020 about "panaceas don't exist. Consider your own context."

lloydchang commented 3 years ago

@OpsM0nkey wrote:

When utilised properly, a PR takes away the need to maintain certain traditional ITIL change management practices (such as CAB) - ie the Pull Request is the Change Request, and all of its links, discussions and approvals provide the necessary assurances that a change is good to go.

Git based platforms have different learning curves than Service Now CAB Workbench and Jira Service Desk.

re: CAB meetings in https://docs.servicenow.com/bundle/rome-it-service-management/page/product/change-management/concept/cab-workbench.html

It's a cultural change from synchronous to asynchronous communication.

re: Default form fields for change requests in https://confluence.atlassian.com/servicedeskcloud/managing-changes-with-your-it-service-desk-817562147.html

Depending on Git based platforms used, pull request templates may be desired, such as https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates and https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository

lloydchang commented 3 years ago

Related, docs(ADOPTION-JOURNEY.md): considerations #43

Thanks @mikecali @OpsM0nkey

lloydchang commented 2 years ago

@mikecali Would you please take it from here? Thank you!

Hello @mikecali @OpsM0nkey @SimonDelord @waynedovey @glimpsovstar

Today, @scottrigby opened this topic during GitOps Working Group meeting.

Here is the outcome:

Feedback on Adoption Journey Document

https://github.com/open-gitops/documents/issues/19 https://github.com/open-gitops/documents/pull/43 note commit co-authors

  • @chris-short: numbering these doesn't make sense
  • @chris-short: There is no one way to adopt GitOps
  • @todaywasawesome: could make a good Green paper
  • @JaimeMagiera: perhaps better as a blog
  • @scottrigby: I would like to see more paragraph content

Thanks all!