manjuraj
commented
11 years ago @TheRealBill Great idea!
Push mechanism makes a lot of sense when each twemproxy knows how to talk to the collector, but the collector doesn't know all the twemproxy running in your datacenter. Sampling is also easier with push model.
Even though nutcracker is single threaded for main network i/o, we have a background thread for collecting stats -- https://github.com/twitter/twemproxy/blob/master/src/nc_stats.c#L839
We can use the same aggregator thread, to push stats and command samples. Similar work was done for twemcache as klogger module - https://github.com/twitter/twemcache#klogger-command-logger. So besides nginx and apache we have some precedence for how this may be implemented in nutcracker
While this would likely be non-trivial, I would love to see nutcracker publish metrics via sFlow. Probably the host flow route since it it isn't networking. Have the ability to sample commands just as the Nginx and apache sFlow modules can would be simply stellar. Especially when dealing with thousands of nodes and clusters. It would also provide a pure metrics source which could be easily integrated into various data collection and alerting/monitoring systems virtually our if the box as it were.