rule_files should support all globs supported by Go's path/filepath

[kustodian]

Proposal

rule_files should support * and all other glob characters supported by Go's path/filepath in the whole file path. Currently, you can only use * in the filename, but not in the path, so this doesn't work:

rule_files:
  - /etc/prometheus/alerts/*/*.yaml

Having it possible to pass something like this:

rule_files:
  - /etc/prometheus/alerts/*.yaml
  - /etc/prometheus/alerts/*/*.yaml
  - /etc/prometheus/alerts/*/*/*.yaml
  - /etc/prometheus/alerts/*/*/*.yaml

would make it ways easier to generate these alert paths dynamically without thinking of changing the Prometheus configuration if new sub-folders are generated.

The reason why * is not supported in the file path is that there is a regex that checks if there is a * in the path: https://github.com/prometheus/prometheus/blob/6fc5305ce904a75f56ca762281c7a1b052f19092/config/config.go#L40

What doesn't make much sense is that you can use other glob special characters like ?, so something like this works:

rule_files:
  - /etc/prometheus/alerts/?????/*.yaml

There were two related issues opened a few years ago (https://github.com/prometheus/prometheus/issues/3546 and https://github.com/prometheus/prometheus/issues/3954) and there is a discussion here. All of them were closed with arguments that file_sd_config uses this same pattern matching and that those two shouldn't be different. This doesn't make much sense, at least at this moment (don't know how things were a few years ago). The reason why it doesn't make much sense is that:

file_sd_config uses a different regex in a different file https://github.com/prometheus/prometheus/blob/15ba7a0d2dc4a76b5400b5e6f1e87f244fb42d95/discovery/file/file.go#L60 And the documentation is written differently. For file_sd_config it states:

files: [ - ... ] Where may be a path ending in .json, .yml or .yaml. The last path segment may contain a single that matches any character sequence, e.g. my/path/tg_.json.

While the rule_files docs states:

rule_files: [ - ... ]

So even from the documentation it states the rule_files support file path globs (which is not true), while the file SD supports only file patterns.

Is there still any reason to not allow all glob characters supported by path/filepath in the rule_files?

[roidelapluie]

The previous consensus regarding consistency between file_sd and rule_files still holds, because there is a limitation with using file globs in file_sd due to inotify limitations.

[kustodian]

Do rule_files also use inotify? I'm only asking for the glob limitation to be lifted from rule_files.

[roidelapluie]

The limitation in rule_files is there to ensure consistency.

[kustodian]

Sorry about this, but I don't understand what consistency are you talking about? If I understand correctly rule_files don't use inotify, while file_sd_config uses inotify. If that is true, I don't understand why is consistency needed between the two when they work differently?

[roidelapluie]

The consistency is that between the file_sd_config and rule_files features, both features only accept globs in the last part of the file path. This is because I want to encourage users to place files in single directories, so that both features behave in a similar manner, despite being technically different.

It's worth noting that we receive very few requests to lift this limitation from rule_files and allow all glob characters in the file path.

You can also do:

rule_files:
- /a/*.yml
- /b/*.yml

which would work for multiple directories.

[kustodian]

Why do you want to encourage users to use them the same way? I guess you have some reasoning behind it. Because if it's just for the sake of them being used the same, it doesn't feel like a good reason :)

I know about the proposed solution you gave, the problem is that in the case Prometheus configuration needs to know in advance all the possible subdirectories, but our rules are deployed separately and each team manages their own rules, so I don't know what directories will there be. I just want to tell Prometheus to read all the *.yml files in any path of the rules directory.

[GerrAlexLTD]

I let myself to support author. We have the same problem - we want to organize our configs as mirror of our engineer infrastructure, for example:

rules
|__building1
     |__room1
          |__b1.r1.alerts.yaml
          |__b1.r1.rules.yaml

It's important for automatic control of config files of monitoring system and there are two ways to solve this problem now: 1) gateway between config store and VMs with Prometheus, which will move files from many non-determinate folders to some specific folders e. g. Alerts and Rules 2) autocompletion of prometheus.yaml, where program will generate this part of config as many strings with names of folders:

rule_files:
  - /etc/prometheus/building1/room1/*.yaml
  - ...
  - /etc/prometheus/building1/roomN/*.yaml

But maybe, in one of next releases, could you add ability to use all glob-symbols as cmd-option e.g. --all-glob-symbols?