AICoE / prometheus-anomaly-detector

A newer more updated version of the prometheus anomaly detector (https://github.com/AICoE/prometheus-anomaly-detector-legacy)
GNU General Public License v3.0
597 stars 151 forks source link

Pod Constantly Crashes And Restarts Silently (In DEBUG) #161

Closed Carmezim closed 3 years ago

Carmezim commented 3 years ago

I seem to not be able to get the Tornado serving the endpoints. I am able to train the data and see the results though.

It seems the pod is crashing and restarting each few minutes without any logs in DEBUG mode so it never finishes training and get to start the server.

This is the service

spec:
  ports:
    - name: metrics
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app.kubernetes.io/name: prometheus-anomaly-detector
  clusterIP: *******
  clusterIPs:
    - ******
  type: ClusterIP
  sessionAffinity: None
status:
  loadBalancer: {}

The deployment ports:

spec:
      containers:
        - name: prometheus-anomaly-detector
          image: prometheus-anomaly-detector-image
          ports:
            - containerPort: 8080
              protocol: TCP
          hostNetwork: false

Running master.

Carmezim commented 3 years ago

That was given under-provisioning the deployment.