[FEATURE] Reload (instead of restart) Prometheus service if configs change and service is already running

[mdschmitt]

Is your feature request related to a problem?

There's actually no need to restart the prometheus service if you're just updating the configs. A simple reload will do the trick. https://prometheus.io/docs/introduction/faq/#can-i-reload-prometheuss-configuration

Describe the solution you'd like

IF

1. The prometheus service and configurations files already exist (i.e. are just being updated/changed, as opposed to being freshly created)

AND

2. The prometheus service is already running

THEN

Reload the prometheus service instead of "hard" restarting it.

Describe alternatives you've considered

I guess maybe enabling --enable-feature=memory-snapshot-on-shutdown https://prometheus.io/docs/prometheus/latest/feature_flags/#memory-snapshot-on-shutdown would help minimize restart time, but it's technically an experimental feature which makes me think I shouldn't run it in prod..

Additional context

If your prometheus installation is large-scale (i.e. scrapes a LOT of data), restarting it can take minutes while it replays the WAL logs. Reloading involves no downtime.

[mdschmitt]

ugh, it seems that if the rules are changed, a reload is not enough, the service must be restarted. Back to the memory snapshot feature, I guess, unless I create a whole separate service.running resource for restarting instead of reloading.

https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1662430673735549