Open mousimin opened 2 months ago
Hi @friedrichg & @yeya24 , I guess the error message "bad response status 422 Unprocessable Entity" was from altermanager, right? But I didn't find any error log from altermanager even I used debug log level, any suggestion from you will be appreciated!
I want to answer myself so that the others can refer to it.
I manually sent the HTTP request using curl and got the detailed response from alertmanager:
maxFailure (quorum) on a given error family, rpc error: code = Code(422) desc = addr=10.120.131.81:9071 state=ACTIVE zone=z1, rpc error: code = Code(422) desc = {"code":601,"message":"0.generatorURL in body must be of type uri: \"staging-cortex-ruler.org/graph?g0.expr=up%7Bapp%3D%22cert-manager%22%7D+%3E+0\u0026g0.tab=1\""}
So I added the schema "https://" at the beginning of the value -ruler.external.url and then it worked.
Map this to the code:
func (n *Manager) sendOne(ctx context.Context, c *http.Client, url string, b []byte) error {
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(b))
if err != nil {
return err
}
req.Header.Set("User-Agent", userAgent)
req.Header.Set("Content-Type", contentTypeJSON)
resp, err := n.opts.Do(ctx, c, req)
if err != nil {
return err
}
defer func() {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}()
// Any HTTP status 2xx is OK.
//nolint:usestdlibvars
if resp.StatusCode/100 != 2 {
return fmt.Errorf("bad response status %s", resp.Status)
}
return nil
}
Maybe we should add the response body into the error message as well? Currently we only add the status which is not easy for debugging.
@friedrichg @yeya24 should we go ahead and start logging the body of the response? It makes sense IMHO.
@rapphil Agree. Would you like to work on it? Just want to make sure AM doesn't send something crazy in the response body. Maybe we can truncate the message with a limit
Describe the bug We are running micro services for cortex, we were using v1 version for alertmanager api by specifying flag -ruler.alertmanager-use-v2=false(used cortex v1.16.0), now we upgrade cortex to v1.17.1, from the log, I see we are using v2 version for alertmanager, when I create some alert rules, I see the alerts fire, but we can't get any email notification, meanwhile we are getting some error messages like:
caller=notifier.go:544 level=error user=Test alertmanager=https://cortex-alertmanager.org/alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 422 Unprocessable Entity"
To Reproduce Steps to reproduce the behavior:
Expected behavior we should get the notifications and no error log should appear.
Environment:
Additional Context configuration file for cortex ruler:
configuration file for cortex alertmanager:
the configuration for alertmanager: