max-rocket-internet / k8s-event-logger

Watches k8s cluster events and logs them to stdout in JSON
GNU General Public License v3.0
219 stars 30 forks source link

Bug in Alert Translation with backticks #39

Closed navinpai closed 1 year ago

navinpai commented 1 year ago

Had an alert in SQLite set as below:

> select data from alert_rule;
[{"refId":"A","queryType":"instant","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"loki","model":{"datasource":{"type":"loki","uid":"loki"},"editorMode":"builder","expr":"{container_name=\"/app-demo\"} |= `debug`","hide":false,"intervalMs":1000,"maxDataPoints":43200,"maxLines":null,"queryType":"instant","refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[],"type":"gt"},"operator":{"type":"and"},"query":{"params":["B"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"min","refId":"B","settings":{"mode":""},"type":"reduce"}},{"refId":"C","queryType":"","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"B","hide":false,"intervalMs":1000,"maxDataPoints":43200,"refId":"C","type":"threshold"}}]

Note the `debug` in the rule

When this was migrated to postgres, it converted it as:

> select data from alert_rule;
[{"refId":"A","queryType":"instant","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"loki","model":{"datasource":{"type":"loki","uid":"loki"},"editorMode":"builder","expr":"{container_name=\"/app-demo\"} |= "debug"","hide":false,"intervalMs":1000,"maxDataPoints":43200,"maxLines":null,"queryType":"instant","refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[],"type":"gt"},"operator":{"type":"and"},"query":{"params":["B"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"min","refId":"B","settings":{"mode":""},"type":"reduce"}},{"refId":"C","queryType":"","relativeTimeRange":{"from":300,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"B","hide":false,"intervalMs":1000,"maxDataPoints":43200,"refId":"C","type":"threshold"}}]

which gives the following error in Grafana:

logger=ngalert.scheduler t=2023-09-11T05:59:10.015884274Z level=error msg="Failed to update alert rules" error="failed to get alert rules: failed to fetch alert rules: invalid character 'd' after object key:value pair"

The error seems to be how ` is converted to " without escaping it. To solve it, I had to manually update my DB to change the "debug" to \"debug\"

navinpai commented 1 year ago

Invalid report. Closing