How to modify prometheus rules and other config files and apply those changes?

VivekMBhasi commented 4 years ago

My actions before raising this issue

[ ] Followed the troubleshooting guide
[x ] Read/searched the docs
[ x] Searched past issues

I have currently installed OpenFaaS and have it running on a 3-Node cluster. I wished to tweak the alert.rules.yml file in the prometheus folder to change the threshold at which the alertmanager gets invoked. This attempt failed as I was not allowed to touch the config files to make changes. After seeing a similar issue on another github page, I tried creating a new rule file and a new prometheus.yml file with the exact same contents of the previous config files (namely alert.rules.yml and prometheus.yml) and just changed the value for which the alertmanager gets trigerred in the new rules file that I made. To ensure these files are considered when updating the services, I made changes in the docker-compose.yml file and made sure the 'prometheus_config' and 'prometheus_rules' were set to the new files that I created. After trying to run deploy_stack.sh, it still doesn't let me change the configs. Is that supposed to be like that or am I doing something wrong here?

Expected Behaviour

The various services should've gotten updated after running deploy_stack.sh

Current Behaviour

It keeps showing me this error: failed to update config func_prometheus_config: Error response from daemon: rpc error: code = InvalidArgument desc = only updates to Labels are allowed

Possible Solution

Steps to Reproduce (for bugs)

Try changing alert.rules.yml
run deploy_stack.sh OR
Try making new config files
Update docker-compose.yml with these config files
Run deploy_stack.sh

Context

I wanted to change the threshold at which the alertmanager would get invoked (and possibly try playing around with the config files to see what can be done).

Your Environment

FaaS-CLI version ( Full output from: faas-cli version ): 0.11.8
Docker version docker version (e.g. Docker 17.0.05 ): 19.03.8
Are you using Docker Swarm or Kubernetes (FaaS-netes)? Docker Swarm
Operating System and version (e.g. Linux, Windows, MacOS): Linux
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs from troubleshooting guide

Next steps

You may join Slack for community support.

alexellis commented 4 years ago

You can read the docs for Docker, or join Slack to seek out technical support.

I would also suggest you switch over to Kubernetes which allows you to edit configs dynamically. Just read the OpenFaaS docs on how to deploy, or use the workshop.

alexellis commented 4 years ago

/msg: slack

derek[bot] commented 4 years ago

-- Join Slack to connect with the community https://docs.openfaas.com/community

openfaas / faas