CI-CD-PIPELINE-SETUP

[sanket1211996]

• [x] Create Instance

[sanket1211996]

• [x] installed needed software on instance

[sanket1211996]

• [x] installed jenkins

[sanket1211996]

• [x] configure jenkins

[sanket1211996]

• [x] Cofigure Nodejs plugin

[sanket1211996]

• [x] Cofigure Maven plugin

[sanket1211996]

• [x] Cofigure jdk8 plugin

[sanket1211996]

• [x] Cofigure git plugin

[sanket1211996]

• [x] Cofigure email smtp plugin

[sanket1211996]

• [x] Cofigure Build Pipeline Structure

[sanket1211996]

• [x] Complete Jenkin Setup course online - 5hr
• [x] 1. Pluralsight: Getting Started with jenkins
• [x] 2. Pluralsight: Designing Modern CI/CD pipeline

[sanket1211996]

• [x] Configure UI Public API to accommodate public api url
• [x] Add public url in google cloud console for social login access

[sanket1211996]

• [x] setup Confluent Kafka on jetstream ref: https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html

[sanket1211996]

https://dzone.com/articles/ultimate-guide-to-installing-kafka-docker-on-kuber

[sanket1211996]

Dev Notes:

Delete Stuck Namespace: In one terminal:

kubectl proxy In another terminal:

kubectl get ns delete-me -o json | \ jq '.spec.finalizers=[]' | \ curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data @-

[sanket1211996]

Dev Notes: Dev Notes Instance Setup

kubernetes k8s install

https://microk8s.io/?_ga=2.251306115.466575059.1648492247-887695204.1648492247

docker local registry connect

https://stackoverflow.com/questions/58654118/pulling-local-repository-docker-image-from-kubernetes

Enable this service to enable external IP to kubernetes

sudo microk8s enable metallb

[sanket1211996]

Dev Notes Kafka Kube Install: https://docs.bitnami.com/tutorials/deploy-scalable-kafka-zookeeper-cluster-kubernetes/

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install -n space-dev zookeeper bitnami/zookeeper --set replicaCount=1 --set auth.enabled=false --set allowAnonymousLogin=true

helm install -n space-dev kafka bitnami/kafka --set zookeeper.enabled=false --set replicaCount=1 --set externalZookeeper.servers=zookeeper.space-dev.svc.cluster.local

helm upgrade -n space-dev zookeeper bitnami/zookeeper --set replicaCount=1 --set auth.enabled=false --set allowAnonymousLogin=true

helm upgrade -n space-dev kafka bitnami/kafka --set zookeeper.enabled=false --set replicaCount=1 --set externalZookeeper.servers=zookeeper.space-dev.svc.cluster.local

[sanket1211996]

Zookeeper help instructions: Please be patient while the chart is being deployed

ZooKeeper can be accessed via port 2181 on the following DNS name from within your cluster:

zookeeper.space-dev.svc.cluster.local

To connect to your ZooKeeper server run the following commands:

export POD_NAME=$(kubectl get pods --namespace space-dev -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $POD_NAME -- zkCli.sh

To connect to your ZooKeeper server from outside the cluster execute the following commands:

kubectl port-forward --namespace space-dev svc/zookeeper 2181: &
zkCli.sh 127.0.0.1:2181

[sanket1211996]

Kaafka Help Instructions: Please be patient while the chart is being deployed

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

kafka.space-dev.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

kafka-0.kafka-headless.space-dev.svc.cluster.local:9092

To create a pod that you can use as a Kafka client run the following commands:

kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.1.0-debian-10-r60 --namespace space-dev --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace space-dev -- bash

PRODUCER:
    kafka-console-producer.sh --broker-list kafka-0.kafka-headless.space-dev.svc.cluster.local:9092 --topic test

CONSUMER:
    kafka-console-consumer.sh --bootstrap-server kafka.space-dev.svc.cluster.local:9092 --topic test --from-beginning

[sanket1211996]

Kafka Topic : kafka-topics.sh --create --create --topic nexrad_incoming --bootstrap-server kafka-0.kafka-headless.space-dev.svc.cluster.local:9092

kafka-topics.sh --create --create --topic nexrad_outgoing --bootstrap-server kafka-0.kafka-headless.space-dev.svc.cluster.local:9092

kafka-topics.sh --create --create --topic merra_incoming --bootstrap-server kafka-0.kafka-headless.space-dev.svc.cluster.local:9092

kafka-topics.sh --create --create --topic merra_outgoing --bootstrap-server kafka-0.kafka-headless.space-dev.svc.cluster.local:9092

kafka-topics.sh --list --bootstrap-server kafka-0.kafka-headless.space-dev.svc.cluster.local:9092

[sanket1211996]

Kafka Manager Setup:

helm install stable/kafka-manager zkHosts zookeeper.space-dev.svc.cluster.local

[sanket1211996]

helm upgrade -n space-dev zookeeper bitnami/zookeeper --set replicaCount=2 --set auth.enabled=false --set allowAnonymousLogin=true

helm upgrade -n space-dev kafka bitnami/kafka --set zookeeper.enabled=false --set replicaCount=2 --set externalZookeeper.servers=zookeeper.space-dev.svc.cluster.local

kafka-0.kafka-headless.space-dev.svc.cluster.local:9092 kafka-1.kafka-headless.space-dev.svc.cluster.local:9092