Open 56quarters opened 5 hours ago
Memcached server being monitored hits its connection limit, the exporter will no longer be able to create new connections to it.
This is very intentional. Because Prometheus is a monitoring system we explicitly want to exercise the connection functionality in addition to simply extracting metrics.
Currently, the
Exporter
struct creates a newmemcache.Client
instance every time the Collect method is called. This isn't a problem per se, but it means that when the Memcached server being monitored hits its connection limit, the exporter will no longer be able to create new connections to it. The operator will lose metrics to the Memcached server just as it begins to have a problem.Instead, I propose to create a single
memcache.Client
instance when theExporter
is created. Thememcache.Client
already handles creating and closing connections when they are no longer healthy. Using the samememcache.Client
instance for every scrape would mean that metrics for a server would continue to be available even when it hits its connection limit since the client continues to use existing connections. A downside of this approach is that the Memcached server address is only resolved to an IP address once at startup. This could be mitigated performing DNS resolution every N seconds in a separate goroutine and updating the server list of the client.Before I attempt these changes, is there any interest in making this change? The current approach has the advantage of being a bit simpler and less stateful but the disadvantage of losing metrics when the Memcached server hits its connection limit.