THIS GITHUB REPO IS FOR TO CLOSE SKILLS GAPS BY SELF LEARNING WITH AI

Thanos Setup for Kubernetes with Prometheus and Grafana

This repository contains a step-by-step guide and configuration files to set up a highly available monitoring solution using Thanos, Prometheus, and Grafana. The setup includes multiple Kubernetes clusters, remote write and read capabilities, and long-term storage with object stores like AWS S3 or Minio.

Overview

Main Components

Prometheus: Collects and stores real-time metrics.
Thanos: Adds long-term storage and global view across multiple clusters.
Grafana: Visualizes metrics from Prometheus and Thanos.

Key Features

Monitoring across multiple Kubernetes clusters.
Remote read/write for Prometheus metrics.
Long-term storage in object stores.
Secure mutual TLS between components.
Centralized Grafana dashboard for all environments.

Setup

Prerequisites

Kubernetes Cluster Minikube.
Installed tools: kubectl, helm, docker, minio (optional for local object store).
Basic understanding of Kubernetes, Prometheus, and Grafana.

Components Deployed

Prometheus Operator: For managing Prometheus instances in Kubernetes.
Thanos Sidecar: Provides remote read/write capabilities.
Thanos Querier: Central query point for multiple Prometheus instances.
Thanos Receiver: Handles remote write data from Prometheus.
Thanos Store Gateway: Provides access to historical metrics.
Thanos Compactor: Downsamples long-term data to save storage.
Grafana: Connects to Thanos for visualizing metrics.

Getting Started

Step 1: Install Minikube

Start by setting up a Kubernetes environment using Minikube on Windows or Mac. This repository contains scripts and configurations for both local environmets.

choco install minikube -y

Step 2: Deploy Prometheus and Thanos

Clone this repository:

git clone https://github.com/rifaterdemsahin/prometheusremotewrite.git
cd prometheusremotewrite

Create the monitoring namespace:
```
kubectl create namespace monitoring
```

Install Prometheus Operator CustomResourceDefinitions (CRDs)

kubectl apply -f ./prometheus-operator-crds/

Apply the custom resource definitions and deploy the Prometheus Operator RECURSIVE -R:
```
kubectl apply -R -f prometheus-operator/
```
Deploy Prometheus and Thanos components: (10:07)
```
kubectl apply -f prometheus
```

Check Pods Going Online

kubectl get pods -n monitoring

Check Logs

kubectl logs -l app.kubernetes.io/name=prometheus -n monitoring -f

Port Forward

kubectl get svc -n monitoring

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Step 3: Configure Object Store (Minio or S3)

For local storage, use Minio as an S3-compatible object store. Modify object-store.yaml with your S3 or Minio credentials.

If using Minio, deploy Minio in Kubernetes:
```
kubectl apply -f minio/
```

Step 4: Grafana Setup

Deploy Grafana:
```
kubectl apply -f grafana/
```
Access Grafana UI using port-forwarding:
```
kubectl port-forward svc/grafana 3000:3000 -n monitoring
```
Default credentials are:
- Username: admin
- Password: grafana
Add Thanos as a data source in Grafana.

Step 5: Mutual TLS Setup

For secure communication between Thanos components, mutual TLS is implemented. Follow the TLS configuration steps provided in the mutual-tls/ folder.

Step 6: Scaling and High Availability

Thanos can scale horizontally for high availability. Update the thanos-receiver, querier, and store-gateway configurations to scale the deployment.

Step 7: Clean Up

To clean up the environment, delete the deployed resources:

kubectl delete -f prometheus-thanos/
kubectl delete -f grafana/
kubectl delete namespace monitoring

Next Steps

Remote Write Configuration: Switch to remote write for lower latency and real-time data streaming from Prometheus to Thanos.
Production Deployment on AWS: Follow the next steps in the tutorial to deploy Thanos on AWS using EKS and S3.

rifaterdemsahin / prometheusremotewrite

readme