Add Prometheus and Grafana to dev

[Coldboltage]

Description

Currently, the development environment is set up and functioning well. The next step is to expand the monitoring capabilities by adding Prometheus and Grafana to the system. Grafana should be accessible on port 3001 externally while continuing to use port 3000 internally. This change should coexist with the existing API setup, which also uses port 3000, with namespace isolation to ensure no conflicts.

Objective

• Add Prometheus and Grafana
  Set up Prometheus for metrics collection and Grafana for visualization within the development environment. Ensure these tools are integrated smoothly, with proper access and port configurations.

• Port Configuration for Grafana

  • Grafana should be exposed externally on port 3001.
  • Internally, Grafana will continue using port 3000 to maintain simplicity, considering namespace isolation to avoid conflicts with the API.

Plan

• Add Prometheus and Grafana to Dev Environment
  Expand the current Terraform configuration to include Prometheus and Grafana. Ensure Prometheus is set up for metrics collection, and Grafana can visualize these metrics effectively.

Deployment Phases

1. Development:
   Focus on adding Prometheus and Grafana to the development environment. Validate that the monitoring stack works without impacting existing components.

2. Future Considerations:
   These changes will help form the basis for future staging and production environments. However, the focus is on development for now.

Steps to Implement

• [ ] Add Prometheus to Development

  • Configure Prometheus to run in the development Kubernetes cluster.
  • Verify that Prometheus collects metrics from Kubernetes and other services (like the API).

• [ ] Add Grafana to Development

  • Configure Grafana to run in the development Kubernetes cluster.
  • Ensure Grafana uses port 3000 internally, but expose it on port 3001 externally to avoid conflicts with the API.
  • Use namespaces to manage service isolation and avoid port conflicts.

• [ ] Testing and Validation

  • Verify that Prometheus is collecting metrics as expected.
  • Verify that Grafana is accessible on port 3001 externally and port 3000 internally without conflicts.
  • Confirm metrics from the development setup are correctly visualized in Grafana.

• [ ] Document the Process

  • Document the setup process for Prometheus and Grafana, detailing how to access them and validate metrics.
  • Include troubleshooting steps if Grafana or Prometheus encounters issues.

Secrets Management

• No changes to secrets management are required. The existing secret management approach will be retained, as it is functioning well for this purpose.

Reason

Adding Prometheus and Grafana to the development environment enhances monitoring and observability, ensuring that all components are visible and working as expected. This helps in proactive identification of issues and prepares the groundwork for robust staging and production environments in the future.

Provider Configuration

provider "kubernetes" {
  config_path = "~/.kube/config" # Adjust to match the user's local Kubernetes configuration
}

[Coldboltage]

This issue has been updated. As noted from the repository, we now have a system that is ready to operate with the previous version of the jobsbolt-api. Moving forward, we will take an infrastructure-first approach, which aims to sort out Terraform before anything else. This can be tested with Docker Compose to ensure all components work well together.

The focus now is to add Prometheus and Grafana, as the API already integrates with Sentry. This addition will allow us to track the application's behavior effectively while also testing our approach of setting up development infrastructure first, followed by adjusting the code to meet those infrastructure constraints.