hemajv
commented
3 years ago Thank you for opening the issue @drbwa! :smiley: These are some great questions and I'm not sure if we have the answers for all of them yet, but here are my thoughts.
- Are the tickets currently generated indicative of a variety of issues that can occur in MOC/OCP?
The tickets currently generated are based on the availability alerts we have so far defined here: https://github.com/operate-first/apps/blob/master/odh/base/monitoring/overrides/prometheus-operator/overlays/alerts/prometheus-rules.yaml Right now, we have basic availability alerts which detect when any of the applications deployed on MOC go down, but we are planning to add more alerts as well.
- Can the issues for which we generate tickets be expected to be visible in log messages?
The issues are being created by the GitHub receiver, which does generate some logs such as:
- Does the representation of the issue tickets facilitate automated processing?
I think the following details may help answer this question:
-
Are alerts configured based on data available in Prometheus? Are alerts based on metrics only (i.e. not on logs)?
Yes, currently alerts are configured only based on the metrics we have available in Prometheus.
-
Which components are being monitored and have alerts defined for them?
As I pointed out, you can find our alerting rules defined here. The components we have them defined so far are for JupyterHub, ArgoCD, Grafana, Prometheus, and Observatorium .
-
As far as I understand, the alerts currently configured will generate issue tickets for un/availability events. Is this correct? Do we currently alert on any other issues?
Yes, currently we have defined only basic availability alerts, but we aim to define other alerts as well.
-
Do we have, or is there, some kind of fault model or something like a set of alert templates for monitoring an OCP deployment that we could use to expend the set of issues tickets are created for?
We currently do not have any such template created, but there are other teams at Red Hat such as the OSD team, app-SRE team who have some well defined alerts for their monitoring that we are looking into and would like to incorporate.
-
What metadata do tickets make available? For example, I suppose that the outage start time is represented by the ticket creation time. How are we going to track the end time of an outage?
Here is an example of what an issue looks like: https://github.com/operate-first/SRE/issues/49. There are some labels associated with the alert. For the timestamps, we will probably have to look at the logs from the GitHub receiver since they seem to have timestamps of when an alert was resolved.
-
Which metadata fields are going to be generated automatically and which other fields do we expect to be added manually, possibly as free text? In other words, is there a 'schema' for the fields in an issue ticket?
Currently, it seems that the GitHub receiver issue templates are static and are defined here: https://github.com/m-lab/alertmanager-github-receiver/blob/master/alerts/template.go#L54. If we want to provide additional fields, I believe we can send the changes upstream and they will review to accept the changes. I had a similar discussion with the upstream folks here.
-
Are all tickets going to be generated automatically or will we also have manually created tickets (e.g., based on user complaints)? What is the template for manually created tickets?
We aim to have most tickets generated automatically, but as you pointed out things like user complaints will need to be created manually. I think we are planning to follow templates similar to what we have defined here for our support repo, but we haven't defined it yet.
I hope this helps and please do let us know if you have any other questions! @4n4nd @HumairAK feel free to add anything else I may have missed.
drbwa
sesheta
@hemajv @4n4nd Please allow me a number of questions regarding the issue tickets.
Consider an anomaly detector that learns to detect incidents/outages/failures based on an analysis of a stream of log messages being produced by a system. This anomaly detector learns in an unsupervised manner. However, we need some labels (sets of log messages that are indicative of specific issues) to tune and test the anomaly detector.
The overarching questions I would like to try and clarify are:
Let me try to confirm my understanding.
Are alerts configured based on data available in Prometheus? Are alerts based on metrics only (i.e. not on logs)?
Which components are being monitored and have alerts defined for them?
As far as I understand, the alerts currently configured will generate issue tickets for un/availability events. Is this correct? Do we currently alert on any other issues?
Do we have, or is there, some kind of fault model or something like a set of alert templates for monitoring an OCP deployment that we could use to expend the set of issues tickets are created for?
What metadata do tickets make available? For example, I suppose that the outage start time is represented by the ticket creation time. How are we going to track the end time of an outage?
Which metadata fields are going to be generated automatically and which other fields do we expect to be added manually, possibly as free text? In other words, is there a 'schema' for the fields in an issue ticket?
Are all tickets going to be generated automatically or will we also have manually created tickets (e.g., based on user complaints)? What is the template for manually created tickets?
/cc @davidohana @eranra @ronenschafferibm