Closed friedrichg closed 1 year ago
If we setup the rule_format_version to 2, looks like the problem is solved
$ curl -v -d '{"rule_format_version": "2"}' -H 'X-Scope-OrgID: fake' http://127.0.0.1:8000/api/prom/configs/alertmanager
* Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> POST /api/prom/configs/alertmanager HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.87.0
> Accept: */*
> X-Scope-OrgID: fake
> Content-Length: 28
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Date: Mon, 05 Jun 2023 18:21:57 GMT
<
* Connection #0 to host 127.0.0.1 left intact
the other problem related to this is that if we upload rules using configs from v1.13.2 and we upgrade later configs. Rulers will fail for all tenants quietly to
level=error ts=2023-06-05T18:30:30.919144332Z caller=ruler.go:490 msg="unable to list rules" err="unsupported rule format version 0"
Which renders the ruler unusable for all tenants, not just the one that an unsupported rule format.
@rajagopalanand
one can find offending configurations with
select id,owner_id,config,deleted_at from configs where config::text like '%"rule_format_version": "1"}%';
Please notice this query returns also historical configurations. To check current configuration one has to check the latest with something like
select id,config from configs where owner_id='tenant' order by id desc limit 1;
After deleting the offending configurations, everything is working is fine.
Describe the bug it's not possible to read or modify alertmanager config or rules in configdb after https://github.com/cortexproject/cortex/commit/5578eeb72f322f012fc3c98088d6a0770daf8d37
cortex operators using configdb cannot upgrade to v1.14.1 or v1.15.2
To Reproduce Steps to reproduce the behavior: 1.
2.
Returns
Expected behavior
Previous commit works fine: https://github.com/cortexproject/cortex/commit/c930bd6f344f4560897eb5729a95737fb2995427
Environment:
Additional Context Problem is still present in latest cortex
The following simple form fails too