facebook / mcrouter

Mcrouter is a memcached protocol router for scaling memcached deployments.
MIT License
3.27k stars 539 forks source link

num_clients #49

Closed samek closed 9 years ago

samek commented 9 years ago

We've installed mcrouter on two hosts.

Then in order to monitor them we are periodically calling stats from both of them. All worked ok for some time but then one server started reporting really large number.

We've checked and it's working fine.


Host 1. STAT num_clients 18446744073709549633 Host 2.

STAT num_clients 864

config is: commandargs --config-str={"pools":{"A":{"hash":"crc32","servers":["10.0.0.74:11211","10.0.0.192:11211","10.0.0.135:11211","10.0.0.86:11211","10.0.0.128:11211","10.0.0.129:11211"]}},"route":"PoolRoute|A"} -p 11211

alikhtarov commented 9 years ago

Looks like there's a bug in how we increment/decrement num_clients_stats in server.cpp. It's an unsigned int, so if we decrement more than we increment we end up with an underflow and a large number that you observe.

Can you add some logging to server.cpp near stat_incr and stat_decr to investigate (i.e. LOG(ERROR) << "incremented";) and see if you can catch the mismatched incr/decrs? We're also debugging internally.

samek commented 9 years ago

As you know I could't build the latest version I've used Docker which was 6 months old. After changing to another version (more recent) this never appeared again. I'll now try with the Latest build which I've just made.

alikhtarov commented 9 years ago

@samek Any luck reproducing this bug? We didn't see it on our side either.

samek commented 9 years ago

Not anymore.

I've been running the latest version ever since.