Expose metrics during tests execution

[veaceslavdoina]

We already deployed Prometheus in Dist-Tests cluster and tested how we can get Codex metrics.

screenshots

Configuration

1. Configure Codex to expose the metrics
2. Expose Codex container metrics port to be accessible for scraping
3. Add Codex Pod annotations to use Prometheus Service Discovery for metrics scraping

Manifest

---
apiVersion: v1
kind: Pod
metadata:
  name: codex-metrics
  namespace: default
  annotations:
    prometheus.io/scrape: "true"
  labels:
    app: codex
    tests-type: dist-tests
spec:
  containers:
  - name: codex
    image: codexstorage/nim-codex:latest-dist-tests
    imagePullPolicy: Always
    env:
    - name: CODEX_METRICS
      value: "true"
    - name: CODEX_METRICS_ADDRESS
      value: "0.0.0.0"
    - name: CODEX_METRICS_PORT
      value: "8008"
    ports:
      - name: metrics
        containerPort: 8008
        protocol: TCP

[veaceslavdoina]

@benbierens, thank you for the assistance.

We did the following

• Pod Annotations are handled via ContainerRecipeFactory and %App%ContainerRecipe

Current state

• Annotations adding failed because of System.ArgumentException : An item with the same key has already been added. Key: prometheus/scrape And fix for that was commented because it doesn't work as expected

To do

• Fix annotations assignment for all Codex Pods
• We should be able to set "prometheus.io/port", "8008" in a dynamic way based on the already defined value
• Pod Labels assignment should be handled via ContainerRecipeFactory as we did it for Pod Annotations