search

apac-mcm-aiops-asset / mcm-aiops-gitops

Apache License 2.0
6 stars 1 forks source link

readme

Note 💡

This repository has been archived and will no-longer be maintained.

You can find it's new location at : https://github.com/one-touch-provisioning/otp-gitops

One Touch Provisioning for IBM Cloud Pak across Multi Cloud

Elevator Pitch

This asset is our opinionated implementation of the GitOps principles, using the latest and greatest tooling available, to enable our customers to hit one big red button (figuratively) to start provisioning a platform that provides Cluster and Virtual Machine Provisioning capabilities, Governence and policy management, observability of Clusters and workloads and finally deployment of IBM Cloud Paks, all within a single command*. Codified, Repeatable and Auditable.

OTP *Disclaimer, may actually be more than just one command to type. ;)

The asset is not intended to be used straight into Production, and a lot of assumptions have been made when putting this together. It's main intention is to show the Art of the Possible, but it can be used a base to roll your own.

Whilst all efforts have been made to provide a complete One Touch Provisioning asset, it may not suit every environment and your mileage may vary.

Shout outs 📣

This asset has been built on the shoulders of giants and leverages the great work and effort undertaken by the Cloud Native Toolkit - GitOps Production Deployment Guide and IBM Garage TSA teams. Without those efforts, then this asset would have struggled to get off the ground.

The reference architecture for this GitOps workflow can be found here.

Table of contents

Note ✋

This repository provides an opinionated point of view on how tooling and principles such as Terraform, Ansible and GitOps can be used to manage the infrastructure, services and application layers of OpenShift/Kubernetes based systems. It takes into account the various personas interacting with the system and accounts for separation of duties.

It is assumed that you have already configured the compute, networks, storage, Security Groups, Firewalls, VPC, etc to enable the platform to be deployed. The asset will not perform those actions for you, and it will fail if you attempt to deploy it without those all pre-configured.

This respository is not intended to be a Step-by-Step Guide and some prior knowledge in OpenShift/Kubernetes/VM Provisioning is expected.

Asset Capabilities 🚀

Coming Soon

Pre-requisites ⬅️

Red Hat OpenShift cluster ⭕

Minimum OpenShift v4.8+ is required.

Firstly, build a "bare-bones" Red Hat OpenShift cluster using either IPI (Installer Provisioned Infrastructure), UPI (User Provisioned Infrastructure) methods or a Managed OpenShift offering like IBM Cloud - ROKS.

IPI Methods

UPI Methods

Leveraging the work undertaken by the Cloud Native Toolkit team, you can utilise the following Github repositories to assist you with your UPI install of OpenShift.

Managed OpenShift

CLI tools 💻

IBM Entitlement Key 🔑

To get an entitlement key:

  1. Log in to MyIBM Container Software Library with an IBMid and password associated with the entitled software.
  2. Select the View library option to verify your entitlement(s).
  3. Select the Get entitlement key to retrieve the key.

Setup git repositories

  1. Create a new GitHub Organization using instructions from this GitHub documentation.

  2. From each template repository, click the Use this template button and create a copy of the repository in your new GitHub Organization. Note: Make sure the repositories are public so that ArgoCD can access them.

    Create repository from a template

  3. Clone the repositories locally.

    mkdir -p gitops-repos
cd gitops-repos

# Example: set default Git org for clone commands below
GIT_ORG=apac-mcm-aiops-asset

# Clone using SSH
git clone git@github.com:$GIT_ORG/mcm-aiops-gitops.git
git clone git@github.com:$GIT_ORG/mcm-aiops-gitops-infra.git
git clone git@github.com:$GIT_ORG/mcm-aiops-gitops-services.git
git clone git@github.com:$GIT_ORG/mcm-aiops-gitops-apps.git

  4. Update the default Git URl and branch references in your mcm-aiops-gitops repository by running the provided script ./scripts/set-git-source.sh script.

    cd mcm-aiops-gitops
GIT_ORG=<GIT_ORG> GIT_BRANCH=master ./scripts/set-git-source.sh
git add .
git commit -m "Update Git URl and branch references"
git push origin master

Install and configure OpenShift GitOps

  1. Install the OpenShift GitOps Operator and create a ClusterRole and ClusterRoleBinding. 

    oc apply -f setup/ocp/
while ! oc wait crd applications.argoproj.io --timeout=-1s --for=condition=Established  2>/dev/null; do sleep 30; done
while ! oc wait pod --timeout=-1s --for=condition=Ready -l '!job-name' -n openshift-gitops > /dev/null; do sleep 30; done

  2. Create a custom ArgoCD instance with custom checks

    oc apply -f setup/ocp/argocd-instance/ -n openshift-gitops
while ! oc wait pod --timeout=-1s --for=condition=ContainersReady -l app.kubernetes.io/name=openshift-gitops-cntk-server -n openshift-gitops > /dev/null; do sleep 30; done

    Note: We use a custom openshift-gitops-repo-server image to enable the use of Plugins within OpenShift Gitops. This is required to allow RHACM to utilise Policy Generator plugin. We have included the Dockerfile used to build this under setup/custom-openshift-gitops-repo-server folder.

  3. Configure TLS on IBM Cloud ROKS

    scripts/patch-argocd-tls.sh

Configure manifests for Infrastructure

If you are running a managed OpenShift cluster on IBM Cloud, you can deploy OpenShift Data Foundation as an add-on. Otherwise, on AWS, Azure, GCP and vSphere run the following script to configure the machinesets, infra nodes and storage definitions for the Cloud you are using for the Hub Cluster

   ./scripts/infra-mod.sh

Bootstrap the OpenShift cluster 🥾

  1. Retrieve the ArgoCD/GitOps URL and admin password and log into the UI

    oc get route -n openshift-gitops openshift-gitops-cntk-server -o template --template='https://{{.spec.host}}'

# Passsword is not needed if Log In via OpenShift is used (default)
oc extract secrets/openshift-gitops-cntk-cluster --keys=admin.password -n openshift-gitops --to=-

  2. The resources required to be deployed for this asset have been pre-selected, and you should just need to clone the mcm-aiops-gitops repository in your Git Organization if you have not already done so. However, you can review and modify the resources deployed by editing the following.

     0-bootstrap/single-cluster/1-infra/kustomization.yaml
 0-bootstrap/single-cluster/2-services/kustomization.yaml

    If you choose to disable any Infrastructure or Services resources before the Initial Bootstrap, you will need to re-commit those changes back to your Git repository, otherwise they will not be picked up by OpenShift GitOps.

  3. Deploy the OpenShift GitOps Bootstrap Application.

    oc apply -f 0-bootstrap/single-cluster/bootstrap.yaml

  4. Its recommended to deploy the Infrastructure components, then the Service Componenets once complete. ArgoCD Sync waves are used to managed the order of manifest deployments, but we have seen occassions where applying both the Infrastructure and Services layers at the same time can fail. YMMV.

Once the Infrastructure layer has been deployed, update the 0-bootstrap/single-cluster/kustomization.yaml manifest to enable the Services layer and commit to Git. OpenShift GitOps will then automatically deploy the Services.

   resources:
   - 1-infra/1-infra.yaml
   ## Uncomment 2-services/2-services.yaml once
   ## 1-infra/1-infra.yaml has been completed
   - 2-services/2-services.yaml
   ## Uncomment to deploy Clusters and Applications
   ## Must be done after all steps for 1-infra & 2-services
   ## have been completed.
   # - 3-apps/3-apps.yaml

Credentials

After Infrastructure Automation and RHACM have been installed successfully - all apps are synced in ArgoCD.

Infrastructure Automation

The route to IBM Infrastructure Automation is

oc -n ibm-common-services get route cp-console --template '{{.spec.host}}'

To use Infrastructure Automation, use the following users with the password Passw0rd.

POD=$(oc -n ldap get pod -l app=ldap -o jsonpath="{.items[0].metadata.name}")
oc -n ldap exec $POD -- ldapsearch -LLL -x -H ldap:// -D "cn=admin,dc=ibm,dc=com" -w Passw0rd -b "dc=ibm,dc=com" "(memberOf=cn=operations,ou=groups,dc=ibm,dc=com)" dn

The default admin password is:

oc -n ibm-common-services get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d

Red Hat Advanced Cluster Management

The route to Red Hat Advanced Cluster Management is

oc -n open-cluster-management get route multicloud-console --template '{{.spec.host}}'

Managing OpenShift Clusters via OpenShift GitOps

Within this asset we treat Managed Clusters as OpenShift GitOps Applications. This allows us to Create, Destroy, Hibernate and Import Managed Clusters into Red Hat Advanced Cluster Manmagement via OpenShift GitOps.

Creating and Destroying Managed OpenShift Clusters

Review the Applications layer kustomization.yaml to enable/disable the Clusters that will be deployed via OpenShift GitOps.

  resources:
  ## Create Clusters
  ## Include the Clusters you wish to create below
  ## Examples have been provided
   - argocd/clusters/create/aws/aws-tokyo.yaml
  # - argocd/clusters/create/azure/azure-aus.yaml
  # - argocd/clusters/create/vsphere/vsphere.yaml

Auto-discovery and import of Managed OpenShift Clusters

Hibernating Managed OpenShift Clusters

Managing IaaS Providers within IBM Infrastructure Automation

Deployment of Cloud Paks through OpenShift GitOps

We will use IBM Cloud Pak for Integration (CP4i) as an example Application that can be deployed onto your Managed Clusters. As mentioned previously, we re-use the GitOps approach, and utilise OpenShift GitOps to configure the cluster ready for CP4i, through deploying OpenShift Container Storage, creating Nodes with the correct CPU and Memory, installing the necessary services and tools, and finally, deploy CP4i with MQ and ACE.

There is a few minor manual steps which need to be completed, and that is preparing the CP4i respository with your Cloud details and adding the IBM Cloud Pak Entitlement Secret into the Managed Cluster. In future, we aim to automate this step via SealedSecrets, Vault and Ansible Tower.

We will use the Cloud Native Toolkit - GitOps Production Deployment Guide repositories and it is assumed you have already configured these repostories by following the very comprehensive guide put together by that team. That configuration of those repositories are beyond the scope of this asset.

# Log into Managed Cluster that the CP4i will be deployed too via oc login
oc login --token=<token> --server=<server> 

# Clone the multi-tenancy-gitops repository you configured via the Cloud Native Toolkit - GitOps Production Deployment Guide
git clone git@github.com:cp4i-cloudpak/multi-tenancy-gitops.git

cd multi-tenancy-gitops
# Run the infra-mod.sh script to configure the Infrastruture details of the Managed Cluster
./scripts/infra-mod.sh

# Create an IBM Entitlement Secret within the tools namespace

## To get an entitlement key:
## 1. Log in to https://myibm.ibm.com/products-services/containerlibrary with an IBMid and password associated with the entitled software.  
## 2. Select the **View library** option to verify your entitlement(s). 
## 3. Select the **Get entitlement key** to retrieve the key.

oc new-project tools || true
oc create secret docker-registry ibm-entitlement-key -n tools \
--docker-username=cp \
--docker-password="<entitlement_key>" \
--docker-server=cp.icr.io

Note: Our aim is to reduce these steps in future releases of the asset.

You will need to update the tenents/cloudpaks/cp4i/cp4i-placement-rule.yaml file within the mcm-aiops-gitops-apps repository to match the cluster you wish to deploy the Cloud Pak to and commit to Git.

apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: ibm-cp4i-argocd
  namespace: openshift-gitops
  labels:
    app: ibm-cp4i-argocd
spec:
  clusterConditions:
  - status: "True"
    type: ManagedClusterConditionAvailable
  clusterSelector:
    matchExpressions: []
    matchLabels:
      # Replace value with Cluster you wish to provision too.
      name: aws-ireland

Uncomment the CP4i Application within Application kustomization.yaml file and commit to Git.

resources:
# Deploy Applications to Managed Clusters
## Include the Applications you wish to deploy below
## An example has been provided
 - argocd/cloudpaks/cp4i/cp4i.yaml

OpenShift GitOps will create a RHACM Application that subscribes to the multi-tenancy-gitops repository you configured and apply the manifests to the Managed Cluster's OpenShift-GitOps controller.