Document: persistant metrics storage in production

[LucasRoesler]

My actions before raising this issue

• [ ] Followed the troubleshooting guide
• [x] Read/searched the docs
• [x] Searched past issues

Expected Behaviour

When a cluster is restarted or more specifically if the Prometheus pod is restarted, the function metrics should be unaffected

Current Behaviour

When a cluster is restarted or more specifically if the Prometheus pod is restarted, the function metrics are not persisted to a volume and are therefore reset to 0

Possible Solution

Update the production documentation https://docs.openfaas.com/architecture/production/ to describe how to deploy Prometheus with a persistent volume storage. This may require exposing a new option in the Helm chart.

Context

In the Kubernetes channel in the OpenFaaS Slack, a user noticed a cluster restart or some kind of pod restarts because all of the function stats were reset to zero. This was unexpected and was original seen as a potential crash or bug in the entire OF system.

[s1monw1]

Has this come up again? What are the options to enable persistent storage for OpenFaas' Prometheus?

[alexellis]

No specific configuration is required in OpenFaaS, the operator should configure a volume mount or use a service with long term storage built in like Thanos or Cortex.

[alexellis]

Feel free to ask about Prometheus configuration on the Prometheus slack or in their docs.