quipo / statsd

Golang StatsD client
MIT License
164 stars 51 forks source link

Add functionality to handle socket creation errors with "delayed" retries #41

Open ichekrygin opened 7 years ago

ichekrygin commented 7 years ago

When running in distributed container environments (Kubernetes) resolving connection to statsd service may not always be successful, due to various reasons (outdated or lagging iptables is one of many). The current implementation does not provide good handling for such edge-case. While consumer of the client library can "re-dial" the socket connection, it has no control over metrics functionality (bar overloading send methods) It would be very helpful if client library could "handle" this edge-case periodically retry socket connection while dropping current metrics.

Use-case:

  1. Client attempts to open socket and receives dial TCP/UDP error
  2. Client has the option to log error and continue
  3. On every send invocation, client will attempt to reconnect if connection is nil

Expected outcomes:

A. socket creation is successful upon initial socket creation (current model) B. socket creation failed upon initial attempt, however, subsequently succeeds. Impact: loss of metrics until the socket is re-dialled. C. socket creation failed upon initial and on all successive attempts. Impact: loss of all metrics.