search

one-touch-provisioning / otp-gitops

Apache License 2.0
32 stars 25 forks source link
acm ansible argocd aws azure gcp gitops ibm-cloud infrastructure kubernetes multi-cloud openshift rhacm vmware

readme

Logo [![Contributors][contributors-shield]][contributors-url] [![Forks][forks-shield]][forks-url] [![Stargazers][stars-shield]][stars-url] [![Issues][issues-shield]][issues-url] [![MIT License][license-shield]][license-url]

One Touch Provisioning (OTP) is a pattern that enables the seamless end-to-end provisioning of Red Hat OpenShift clusters, their applications, governance and policies to Public, Private, On-Premises and both Near and Far Edge Clouds all via Code.

OTP Capabilities | Report Bug | Request Feature

Table of Contents


About the Project

One Touch Provisioning

This method/pattern is our opinionated implementation of the GitOps principles, using the latest and greatest tooling available, to enable the hitting of one big red button (figuratively) to start provisioning a platform that provides Cluster and Virtual Machine Provisioning capabilities, Governance and policy management, observability of Clusters and workloads and finally deployment of applications, such as IBM Cloud Paks, all within a single command*.

The pattern leans very heavily on technologies such as Red Hat Advanced Cluster Management (RHACM) and OpenShift GitOps (ArgoCD), enabling a process upon which OpenShift clusters, their applications, governance and policies are defined in Git repositories and by leveraging RHACM as a function of OpenShift GitOps, enables the seamless end to end provisioning of those environments.

(back to top)

Introduction to Concepts and Technologies leveraged

Before Getting Started with this pattern, it's important to understand some of the concepts and technologies used. This will help reduce the barrier of entry when adopting the pattern and help understand why certain design decisions were made.

(back to top)

Getting Started

Prerequisites

Red Hat OpenShift Cluster

Deploy a "vanilla" Red Hat OpenShift cluster using one of the methods below:

CLI tools

OpenShift CLI
Helm CLI
KubeSeal CLI (Optional)

Repository considerations

There are two repository patterns to consider when leveraging GitOps: Monorepo or Polyrepo. For OTP, we have leveraged a Polyrepo structure, which consists of six git repositories within the GitOps workflow. You can learn more on why we chose a Polyrepo here.

Setup of git repositories

  1. Create a new GitHub Organization using instructions from this GitHub documentation.

  2. Create the repositories within your new GitHub Organization and clone them locally.

    GIT_ORG=<new-git-organization> OUTPUT_DIR=<gitops-repos> ./scripts/create-repos.sh

  3. (Optional) Many users may wish to use private Git repositories on GitHub to store their manifests, rather than leaving them publically readable. The steps for setting up OpenShift GitOps for Private repositories can be found here.

  4. Update the default Git URL and branch references in your otp-gitops repository by running the provided script ./scripts/set-git-source.sh script.

    GIT_ORG=<GIT_ORG> GIT_BRANCH=master ./scripts/set-git-source.sh
    • You can unset the changes you made above by running the ./scripts/unset-git-source.sh.

IBM Entitlement Key for IBM Cloud Paks

If you intend to deploy the Infrastructure Automation component of IBM Cloud Pak for Watson AIOps, then please follow the instructions here.

(back to top)

Installation

Install and configure OpenShift GitOps

  1. Install the OpenShift GitOps Operator and create a ClusterRole and ClusterRoleBinding. 

    oc apply -k setup/argocd-operator
while ! oc wait crd applications.argoproj.io --timeout=-1s --for=condition=Established  2>/dev/null; do sleep 30; done
while ! oc wait pod --timeout=-1s --for=condition=Ready -l '!job-name' -n openshift-gitops > /dev/null; do sleep 30; done

  2. Create a custom ArgoCD instance with custom checks. To customise which health checks, comment out those you don't need in setup/argocd-instance/kustomization.yaml.

    oc apply -k setup/argocd-instance
while ! oc wait pod --timeout=-1s --for=condition=ContainersReady -l app.kubernetes.io/name=openshift-gitops-otp-server -n openshift-gitops > /dev/null; do sleep 30; done

  3. (Optional) If using IBM Cloud ROKS as a RHACM Hub Cluster, then you will need to configure TLS.

    ./scripts/patch-argocd-tls.sh

Configure Storage and Infrastructure nodes

On AWS, Azure, GCP, vSphere and Baremetal you can run the following script to configure the machinesets, infra nodes and storage definitions for the Cloud you are using for the RHACM Hub Cluster.

This will deploy additional nodes to support OpenShift Data Foundation (ODF) for Persistant Storage, as well as additional nodes to support Infrastructure (aka infra) components, such as RHACM, Quay, Ingress Controllers, OpenShift Internal Registry and ACS.

This is a design choice to reduce OpenShift licensing requirements as running these components on Infrastructure nodes does not consume a subscription cost.

When running on Baremetal, it will utilise Local Storage for deploying ODF. It will autoselect all workerIt will not deploy additional nodes for storage or Infra, this will be improved upon in later versions.

   ./scripts/infra-mod.sh

IBM Cloud - ROKS

If you are running a managed OpenShift cluster on IBM Cloud, you can deploy OpenShift Data Foundation as an add-on. You will also need to label some of your worker nodes as Infra nodes, otherwise RHACM will fail to deploy.

Attach the following label to the worker nodes you intend to use as Infra nodes.

node-role.kubernetes.io/infra: ''

Install a Local Hashicorp Vault Instance (Optional)

OTP works best when connected to an Secret Store like Hashicorp Vault, if you already have a pre-existing Vault-like instance available, for example IBM Secrets Manager, you can skip this step and move onto installing the External Secrets Operator, however if you'd like to install a local Hashicorp Vault Instances into the Hub Cluster, then follow the below steps.

   oc apply -k setup/hashicorp-vault-chart

Install External Secrets Operator

  1. Install the External Secrets Operator to enable OTP to connect to either a pre-existing Vault-like instance or to the Local Hashicorp Vault instance deployed in the previous step.

    oc apply -k setup/external-secrets-operator

  2. Apply the API Key as a secret that will allow OTP to connect to your Vault-like instance via the External Secret Operator.

    oc create secret generic ibm-secret --from-literal=apiKey='<APIKEY>' -n kube-system

  3. Configure the ClusterSecretStore with the API Key secret and URL of your Vault-like instance.

    apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
 name: cluster
 namespace: external-secrets
spec:
 provider:
   ibm:
     auth:
       secretRef:
         secretApiKeySecretRef:
           name: ibm-secret
           namespace: kube-system
           key: apiKey
     serviceUrl: >-
       https://3f5f4d5b-6179-4d7c-a7a2-72dc28eb4a81.au-syd.secrets-manager.appdomain.cloud

  4. Apply the updated ClusterSecretStore.

    oc apply -f setup/external-secrets-instance/cluster-secret-store.yaml

Bootstrap the OpenShift RHACM Hub cluster

The bootstrap YAML follows the app of apps pattern.

  1. Retrieve the ArgoCD/GitOps URL and admin password and log into the UI

    oc get route -n openshift-gitops openshift-gitops-otp-server -o template --template='https://{{.spec.host}}'

# Password is not required if using the OpenShift as an authorisation provider
oc extract secrets/openshift-gitops-otp-cluster --keys=admin.password -n openshift-gitops --to=-

  2. The resources required to be deployed for this pattern have been pre-selected. However, you can review and modify the resources deployed by editing the following.

     0-bootstrap/hub/1-infra/kustomization.yaml
 0-bootstrap/hub/2-services/kustomization.yaml
 0-bootstrap/hub/3-policies/kustomization.yaml
 0-bootstrap/hub/4-clusters/kustomization.yaml
 0-bootstrap/hub/5-apps/kustomization.yaml

    Any changes to the kustomization files before the Initial Bootstrap, will need to be committed back to your Git repository, otherwise they will not be picked up by OpenShift GitOps.

  3. Deploy the OpenShift GitOps Bootstrap Application.

    oc apply -f 0-bootstrap/hub/bootstrap.yaml

  4. ArgoCD Sync waves are used to managed the order of manifest deployment, this is required as some objects have parent-child relationships and are expected to exist within the RHACM Hub before they can be successfully deployed. We have seen occassions where applying both the Infrastructure, Services and Policies layers at the same time can fail. This typically occurs when there are issues with provisioning of additional nodes to support Storage and Infrastructure components. YMMV.

Once the Infrastructure, Services and Policies layers have been deployed, update the 0-bootstrap/hub/kustomization.yaml manifest to enable the Clusters and Apps layer and commit to Git. OpenShift GitOps will then automatically deploy any resources listed within those Kustomise files.

   resources:
   - 1-infra/1-infra.yaml
   - 2-services/2-services.yaml
   - 3-policies/3-policies.yaml
   ## Uncomment once the above layers have completed.
   # - 4-clusters/4-clusters.yaml
   # - 5-apps/5-apps.yaml

Installation is successful once all ArgoCD Applications are fully synced without errors.

You will be able to access the RHACM Hub console via the OpenShift console.

(back to top)

Usage

Deploying and Destroying Managed (aka Spoke) OpenShift Clusters via OpenShift GitOps

This pattern treats Managed (aka Spoke) Clusters as OpenShift GitOps Applications. This allows us to Create, Destroy, Hibernate and Import Managed Clusters into Red Hat Advanced Cluster Management via OpenShift GitOps.

Creating and Destroying Managed OpenShift Clusters

The pattern provides full end to end deployment of not only Clusters, but also Policies, Governance and Applications.

For more usage examples, please refer to the Documentation

(back to top)

Roadmap

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

(back to top)

License

Distributed under the APACHE 2.0 License. See LICENSE for more information.

(back to top)

Contact

Project Link: https://github.com/one-touch-provisioning/otp-gitops

(back to top)

Acknowledgments

This asset has been built on the shoulders of giants and leverages the great work and efforts undertaken by the teams/individuals below. Without these efforts, then this pattern would have struggled to get off the ground.

The reference architecture for this GitOps workflow can be found here.

(back to top)