search

littlemanco / www.littleman.co

Website
1 stars 1 forks source link

As a user I'd like to know more about setting up CI/CD with Drone CI #33

Closed andrewhowdencom closed 5 years ago

andrewhowdencom commented 5 years ago

Article;

// Todo: Note Kubernetes is still in flux // Todo: Put in separate network first

Lables:

co.littleman.k8s.tolerates: fault-intolerant co.littleman.k8s.tolerates: fault-tolerant

andrewhowdencom commented 5 years ago

Goals

Requirements

// Todo: Do we split requirements otu to a different post?

TIp: We are opinionated about our infrastructure. However, many of these builds steps can be run on AWS, Azure or your own bare metal infrastructure.

GKE

Create the main cluster 

gcloud container clusters create \
  --zone=europe-west1-d \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=1 \
  --num-nodes=1 \
  --enable-autorepair \
  --enable-autoupgrade  \
  --disk-size=30GB \
  --disk-type=pd-standard \
  --machine-type=n1-standard-2 \
  --no-enable-basic-auth \
  --no-issue-client-certificate \
  build-farm

Note: Looks like we need cluster level autoscaling on to get pod autoscaler. Network Policy is just to support it, autorepair and autoupgrade for low maintenance ,machine type the minimal required to schedule all pods, basic auth and client certificate deprecated auth mechanisms.

Also, even though it's autoscaled between 1 and 1 it looks like it creates a 3 node cluster by default. Added num-nodes to fix that.

// Todo: Revisit this Removed network policy to try and cut down number of pods required for control plane.

Create the build workers

gcloud container node-pools create buildworkers \
  --cluster build-farm \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=3 \
  --num-nodes=1 \
  --zone=europe-west1-d \
  --preemptible \
  --node-taints 'role=build-worker:NoSchedule' \
  --enable-autoupgrade \
  --enable-autorepair \
  --machine-type=n1-highcpu-2 \
  --disk-size=30GB \
  --disk-type=pd-ssd

Enable autoscaling allows scaling to 0, preemptable is cheap compute. Might die but eh. Node taints mean nothing else will get scheduled here. High CPU because builds are typically CPU limited.

Verify cluster works

gcloud container clusters get-credentials buildfarm --zone=europe-west1-d

Need ot bump size of cluster small node; need to deploy autoscaler

$ kubectl get events --namespace=kube-system | grep cluster-autoscaler

 6m          6m           1         cluster-autoscaler-status.159421c15d004832                              ConfigMap                                                 Normal    ScaleDownEmpty      cluster-autoscaler                                   Scale-down: removing empty node gke-build-farm-buildworkers-2f591636-5k3h

Scaled down! Means we have up to ${N} build machines, and they're cheap because they're preemptable.

Helm

// Todo: Put tiller into helm namespace

RBAC

Create the RBAC accounts:

Ref: https://github.com/helm/helm/blob/master/docs/rbac.md#example-service-account-with-cluster-admin-role

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF

TLS

Ref: https://github.com/helm/helm/blob/master/docs/tiller_ssl.md#generating-certificate-authorities-and-certificates

Generate CA

mkdir ~/.TLS
chmod 700 ~/.TLS
cd ~/.TLS

// Todo: Reference TLS article // Todo: Reference article on setting up a CA with OpenSSL, and mention can also do this with Vault et. al

  1. Privkey
openssl genrsa -out ./ca.key.pem 4096
  1. CA WARNING: Customize to your own liking 
    
openssl req -key ca.key.pem -new -x509 -days 7300 -sha256 -out ca.cert.pem -extensions v3_ca

You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank.

Country Name (2 letter code) [AU]:DE State or Province Name (full name) [Some-State]:Berlin Locality Name (eg, city) []:Berlin Organization Name (eg, company) [Internet Widgits Pty Ltd]:littleman.co Organizational Unit Name (eg, section) []:Software Engineering Common Name (e.g. server FQDN or YOUR name) []:build-farm.k8s.littleman.com
Email Address []:andrew.howden@littleman.co


3. Privkeys for srv/client

openssl genrsa -out ./tiller.key.pem 4096 openssl genrsa -out ./helm.key.pem 4096


4. Tiller CSR (srv)

openssl req -key tiller.key.pem -new -sha256 -out tiller.csr.pem

You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank.

Country Name (2 letter code) [AU]:DE State or Province Name (full name) [Some-State]:Berlin Locality Name (eg, city) []:Berlin Organization Name (eg, company) [Internet Widgits Pty Ltd]:littleman.co Organizational Unit Name (eg, section) []:Software Engineering Common Name (e.g. server FQDN or YOUR name) []:tiller.build-farm.k8s.littleman.co Email Address []:andrew.howden@littleman.co

Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []:


5. Helm CSR (client)

openssl req -key helm.key.pem -new -sha256 -out helm.csr.pem You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank.

Country Name (2 letter code) [AU]:DE State or Province Name (full name) [Some-State]:Berlin Locality Name (eg, city) []:Berlin Organization Name (eg, company) [Internet Widgits Pty Ltd]:littleman.co Organizational Unit Name (eg, section) []:Software Engineering Common Name (e.g. server FQDN or YOUR name) []:Andrew Howden
Email Address []:andrew.howden@littleman.co

Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []:


6. generate certificates

openssl x509 -req -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -in tiller.csr.pem -out tiller.cert.pem -days 365 openssl x509 -req -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -in helm.csr.pem -out helm.cert.pem -days 365


7. Verify certificates:

ca.cert.pem ca.key.pem ca.srl helm.csr.pem helm.key.pem tiller.cert.pem tiller.csr.pem tiller.key.pem


8. Protect certificates on local machine:

chmod 600 ~/.TLS/*


## Initialize Helm

helm init --override 'spec.template.spec.containers[0].command'='{/tiller,--storage=secret}' --tiller-tls --tiller-tls-verify --tiller-tls-cert=tiller.cert.pem --tiller-tls-key=tiller.key.pem --tls-ca-cer t=ca.cert.pem --service-account=tiller --history-max=200 --upgrade


Move PKI material into $HELM_HOME

cp ca.cert.pem $(helm home)/ca.pem cp helm.cert.pem $(helm home)/cert.pem cp helm.key.pem $(helm home)/key.pem


Verify it works

helm ls --tls


# Setup PKI tooling

- Why PKI
- Why JetStack

See https://github.com/helm/charts/tree/master/stable/cert-manager#installing-the-chart

Create namespace

kubectl create ns cert-manager


Apply CRDs

kubectl apply \ --namespace=cert-manager \ --filename https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml


Disable validatoin for reasons

kubectl label ns cert-manager certmanager.k8s.io/disable-validation="true"


Deploy chart

helm install --tls --namespace=cert-manager --name=cert-manager stable/cert-manager


Create service account:

$ gcloud iam service-accounts create build-farm-cert-manager --display-name "Build Farm Cert Manager" Created service account [build-farm-cert-manager].


Allow that service account to access and update DNS records:

$ gcloud projects add-iam-policy-binding littleman-co --member serviceAccount:build-farm-cert-manager@littleman-co.iam.gserviceaccount.com --role roles/dns.admin Updated IAM policy for project [littleman-co]. ... (omitted for brevity) ...


Create a key for that service account:

gcloud iam service-accounts keys create ~/.TLS/service-account.json --iam-account=build-farm-cert-manager@littleman-co.iam.gserviceaccount.com created key [24aac2d8e3e608d6d8ded64178b44ea7f46c78a7] of type [json] as [/home/hello/.TLS/service-account.json] for [build-farm-cert-manager@littleman-co.iam.gserviceaccount.com]


Create a secret that will be consumed by cert manager to issue certificates:

$ kubectl create secret generic cert-manager-gcloud-service-account --from-file=$HOME/.TLS/service-account.json --namespace=cert-manager secret/cert-manager-gcloud-service-account created


Create issuer; 

WARNING: This requires Google Cloud as its DNS validator, however you can amend it to whatveer. See docs 

// Todo: Insert docs link

cat <<EOF | kubectl apply -n cert-manager -f - apiVersion: certmanager.k8s.io/v1alpha1 kind: ClusterIssuer metadata: name: cert-manager-letsencrypt-production spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: pki@littleman.co

Name of a secret used to store the ACME account private key

privateKeySecretRef:
  name: cert-manager-letsencrypt-production
# ACME DNS-01 provider configurations
dns01:
  # Here we define a list of DNS-01 providers that can solve DNS challenges
  providers:
    - name: prod-dns
      clouddns:
        # A secretKeyRef to a google cloud json service account
        serviceAccountSecretRef:
          name: cert-manager-gcloud-service-account
          key: service-account.json
        # The project in which to update the DNS zone
        project: littleman-co


How

- Autoscaling (but is soft because drone doesnt support annotations and i cbf writing mutating admission controller)
- Kubernetes
- Drone
- Google Cloud
andrewhowdencom commented 5 years ago

Install Ingress Server

helm install --tls --name=drone-ci-ingress \
  --namespace=drone-ci \
  stable/nginx

Get the IP for the build server:

$ kubectl get svc --watch -n drone-ci
NAME                                             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
drone-ci-ingress-nginx-ingress-controller        LoadBalancer   10.59.243.247   34.76.20.96   80:31614/TCP,443:30321/TCP   1m
drone-ci-ingress-nginx-ingress-default-backend   ClusterIP      10.59.250.255   <none>        80/TCP                       1m

Get the Zones from Google:

$ gcloud dns managed-zones list
NAME                DNS_NAME             DESCRIPTION           VISIBILITY
littleman-co        littleman.co.

Start a DNS transaction

$ gcloud dns record-sets transaction start --zone littleman-co
Transaction started [transaction.yaml].

Add the A record to the zone

gcloud dns record-sets transaction add "10.59.243.247" --name build-farm.littleman.co --ttl 300 --type A --zone littleman-co
Record addition appended to transaction at [transaction.yaml].

Execute the transaction:

$ gcloud dns record-sets transaction execute --zone=littleman-co
Executed transaction [transaction.yaml] for managed-zone [littleman-co].
Created [https://www.googleapis.com/dns/v1/projects/littleman-co/managedZones/littleman-co/changes/87].
ID  START_TIME                STATUS
87  2019-04-11T15:44:08.308Z  pending

Wait a while for the DNS record to propagate:

$ watch -n1 dig build-farm.littleman.co. +short

10.59.243.247

// Todo: Verify NGINX

Install Drone

TIP: This is where the required oAuth account will be consumed. If you haven't done so, please set it up now: https://docs.drone.io/installation/github/kubernetes/

Create secret for GitHub authneetication

WARNING: Replace XXX with your own values.

// Todo: Should probably do these as "export".

kubectl create secret generic drone-ci-github-application \
      --namespace=drone-ci \
      --from-literal=clientSecret="XXXXXXXXXXXXXXXXXXXXXXXX"

Write drone configuration to disk:

cat <<EOF > values.yml
---
sourceControl:
  provider: "github"
  secret: "drone-ci-github-application"
  github:
    clientID: "XXX"

server:
  host: "build-farm.littleman.co"
  protocol: "https"

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx"
  hosts:
    - build-farm.littleman.co
  tls:
    - hosts:
        - build-farm.littleman.co
      secretName: drone-ci-tls
EOF

agent:
  affinity:

Install drone

# Needs the --devel as it's in RC at the minute
helm install --name drone-ci --tls --namespace=drone-ci --devel --atomic --values=values.yml stable/drone

Visit https://build-farm.littleman.co/ (${YOUR_URL})

Follow the prompt.

Tada! Build farm.

CURRENTLY INCOMPLETE:

https://github.com/drone/drone/blob/bd23f21ff47766d3c8a5e696e9608341b9e6a6e2/scheduler/kube/kube.go#L141

As well as the chart.

andrewhowdencom commented 5 years ago

https://discourse.drone.io/t/is-it-possible-to-schedule-drone-task-runners-with-specific-kubernetes-tolerations/4282

andrewhowdencom commented 5 years ago

Need to automatically clean up the jobs.

andrewhowdencom commented 5 years ago

Use ArgoCD to manage releases.

stale[bot] commented 5 years ago

This issue has not received any updates in quite some time. It's not clear whether it's still relevant, and should be kept. If there is no more activity in the next 14 days on this ticket, I will close it. It can always be reopened later, but for now in the interests of keeping the project as clear and simple as possible it will be closed. If you'd like to keep it open simply comment:

"This needs to remain open"

And I will leave it alone for the next 90 days or so. Alternatively, if you are already sure that this is no longer relevant you can also close it directly. For more information see:

https://l.littleman.co/2XnF2B3