search

FNNDSC / pman

A process management system written in python
MIT License
22 stars 33 forks source link

feat: add prometheus scrape annotations to pman pod #218

Closed sakshi1215 closed 1 year ago

sakshi1215 commented 1 year ago

refers https://github.com/FNNDSC/pman/issues/215

With this we will be able to scrape resource metrics on kubernetes.

jennydaman commented 1 year ago

215 is about annotating the containers created by pman, not the pman container itself.

pman doesn't have a /metrics endpoint so I am not sure this would work?

The idea here is, pman creates containers which are ChRIS plugin instances. We want to scrape Kubernetes/cAdvisor to get resource usage of ChRIS plugins.

sakshi1215 commented 1 year ago

You don’t any annotations on the container for the kubelet and cAdvisor to get the metrics, cAdvisor has all the usage records per pod as well as per container. Same goes in the case of prometheus, to scrape the usage metrics you can directly use cAdvisor endpoint in cluster without any annotations to a pod. In the issue it was mentioned that annotations needs to be added to the containers so I assumed there would be a prometheus server running which would expose the application metrics at 9092 port(which again is a general convention), but looks like there are no custom metrics as well as instrumentation in the application to get per plugin details about usage.

So if it is just about the resource usage by the containers then you dont need any extra annotations on container in the kubernetes worls.

On Wed, 9 Nov 2022 at 19:55, Jennings Zhang @.***> wrote:

Closed #218 https://github.com/FNNDSC/pman/pull/218.

— Reply to this email directly, view it on GitHub https://github.com/FNNDSC/pman/pull/218#event-7774072761, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANF3CEAIXFVVBRNARNDZ3TTWHOX5TANCNFSM6AAAAAARTWTBOI . You are receiving this because you authored the thread.Message ID: @.***>

jennydaman commented 1 year ago

@sakshi1215 the reason why we would want to annotate the pods being created by pman is because we want to use something like cAdvisor to get information about pods which are ChRIS plugin instances. There are probably many pods running, and also some pods which are the ChRIS backend themselves, though we want to be able to collect performance data on just the ChRIS plugin instances and excluding everything else.