Ruler: backend local support glob format for rules path

[jmichalek132]

Is your feature request related to a problem? Please describe.

Context: We have a central git repo for storing alerts and recording rules, this enables us to run a validation pipeline to make sure the rules are valid, enforce certain company-specific requirements using tools such as promruval, and run the unit test for the alerts/recording rules if present. We have a folder per tenant in the repo to match what the ruler expects. We deploy these rules on the filesystem using git-sync sidecar.

Problems:

• In the case of a tenant having dozens of rule files, it would be good to be able to organize them using nested folders within the tenant folder but the ruler doesn't support that, files not present in the tenant top-level folder are not read.
• We wanted to store the unit tests for rules alongside the actual rules, in files with a .test suffix, however, the ruler ended up reading those files too and logging error messages about those files having unexpected formats.

Describe the solution you'd like

I would like the ruler to support the glob format in the argument for the rules path when using the backend local. Optionally the argument could be repeated. How could this look like:

  - /etc/prometheus/alerts/*.yaml
  - /etc/prometheus/alerts/*/*.yaml
  - /etc/prometheus/alerts/*/*/*.yaml
  - /etc/prometheus/alerts/*/*/*.yaml

This would allow a certain level of nested folders, and also avoid loading in files with the .test suffix. As discussed in the original Slack thread there are some pros and cons to this approach. Pros:

• More flexibility when loading rules
• Thanos already supports this too Cons:
• Diverging from Prometheus behavior which doesn't support this fully details in https://github.com/prometheus/prometheus/issues/11932
• Less predictable behavior

For the cons, I would argue that there is already divergence from Prometheus due to the support for tenants, and the less predictable behavior could be mitigated by documentation / hiding the feature behind a feature flag/config option. The arguments to not support this in Prometheus are mainly to ensure consistent behavior in Prometheus with a different configuration loading option.

Describe alternatives you've considered

Git sync supports executing a command on successful sync, this could trigger a script that flattens a nested folder structure and removes files with the .test suffix. This does feel a bit fragile. For tests, we could also just store them in a folder not read by the ruler.

Additional context

Originally raised on Grafana labs slack .

As mentioned thanos already supports this with:

      --rule-file=rules/ ...     Rule files that should be used by rule
                                 manager. Can be in glob format (repeated).
                                 Note that rules are not automatically detected,
                                 use SIGHUP or do HTTP POST /-/reload to re-read
                                 them.

This might be relevant for loki too given it has the same behavior implemented.

I would be also willing to contribute this feature.

[dimitarvdimitrov]

thanks for opening the issue. The proposal makes sense to me (as we discussed in the #mimir Slack channel). I'm copying my response here as well

do we want mimir to feel more like prometheus or do we want to be more useful and less predictable at times I’d say I’m ok with adding more globs to the ruler’s support. You mentioned Thanos supports this too.

I think we don't have the consistency problem that prometheus does. In addition to that, more supported globs is not a breaking change and does not change the compatibility (more likeness than compatibility) with prometheus config

[dimitarvdimitrov]

As you said you're open to implementing it, if we don't hear any objections in the next few days I think it's ok to start working on this