Tazama Infrastructure

[scott45]

Identify any issues, errors and opportunities for improvement in the installation guide.

Issues & errors

1. Step-1 Explicityly state the namespace where (optionals) vault, keycloak, prometheus and grafana are to be deployed
2. Step-3 Secure your ingress with TLS, show how to generate the tls.key and tls.crt command using

mkdir certs 

openssl req -x509 -nodes -days 9999 -newkey rsa:2048 -keyout /Users/scott/Desktop/tazama/EKS-helm/certs/tls.key -out /Users/scott/Desktop/tazama/EKS-helm/certs/tls.crt 

kubectl create secret tlscomsecret ingress-cert --namespace development --key=/Users/scott/Desktop/tazama/EKS-helm/certs/tls.key --cert=/Users/scott/Desktop/tazama/EKS-helm/certs/tls.crt -o yaml

3. Provide more commands in jenkins section e.g getting password, port forwarding etc

Get your 'admin' user password by running:

kubectl exec --namespace cicd -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

Get the Jenkins URL to visit by running these commands in the same shell:

echo http://127.0.0.1:8080
  kubectl --namespace cicd port-forward svc/jenkins 8080:8080

4. State the need to install the aws credentials plugin before configuring ecr credentials
5. Before running through the installation commands in Section Building the Jenkins Agent Locally Image

Prereqs:

Create image repo in aws;

aws ecr create-repository --repository-name hello-repository --region region. The image repository on aws will be aws_account_id.dkr.ecr.region.amazonaws.com

Login to registry

aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

Bash docker script provided here

The script that has a dockerfile didn't work for me. Made a few changes to it for it to successfully run. I'll submit the changes.

6. Couple of jenkins notes to be aware of; Step 4

• Make sure you check your executors otherwise your builds will be stuck in queues. Go to manage jenkins, system and change executor number from 0 to 2. There is a good practice recommended by jenkins docs that don't require any scaling of executors. This will be submitted e.g working with label matching for running builds

• Ensure github token has been used doe github creds if you have mfa set.

• Configuring Rule Processors : choose rule processor then go to configuration

Opportunities for improvement

Workflow Automations / Security

• Write the terraform scripts needed to spin up the k8s clusters on EKS and AKS before following docs to install tazama. I'm currently doing this for EKS. This was done for EKS with terraform and the PR is here

• Atlantis to improve terraform workflows

• Nice to have - create a tazama dockerhub account and add centralised github action workflows to build and push different services docker images of major releases / branches for public use.

• Github workflows for running tests, cicd to testing environments, check code quality, build docker images, check github commits and naming convetions etc

• Secrets management

• VPN ?? if there is a valid use case

• Github branches protection

Documentation The currently archived eks setup wiki / doc in confluence is best suited as an installation guide

• New Engineer Onboarding / Machine setup
• New DevOps / SRE Specific Setup / Onboarding / Machine setup
• Infra Architecture Diagram
• Infra cheat sheet (services and different integrations we have in place)
• Terraform Doc (installation, versions and commands to run)
• Jenkins configuration
• Kubectl basics / Kubectl Setup for K8s EKS Cluster
• Document Github Workflow (GitOps)
• Document how to request access to certain tools / platforms
• K8s (eks etc) Uprgade guides for managed clusters to ensure the software is running supported versions
• Project Mgt / Infra Jira Epic

CICD & Deployments

• Explore ArgoCD as an option to jenkins. It's a great k8s deployment tool
• ArgoCD (to co exist with jenkins or replace it) - https://helm.sh/blog/tools-to-manage-helm-declaratively
• Github actions

Monitoring Stack

• Datadog
• Slack notification channels and webhook configuration\

Artifactory mgt

• Jfrog artifactor to store libraries, packages images and build artefacts