awslabs / elasticache-cluster-config-net

A configuration object for Enyim's MemcachedClient that enables the use of Amazon ElastiCache's Auto Discovery feature
Apache License 2.0
29 stars 17 forks source link

Excessive TCP connections to nodes #2

Closed rhargreaves closed 9 years ago

rhargreaves commented 9 years ago

I've observed that using the ElastiCacheClusterConfig component seems to lead to an excessive number of TCP connections to nodes.

For example, when I run NETSTAT with the default Enyim configuration, I find that each node has 10 connections. When I run NETSTAT with the ElastiCacheClusterConfig, I can see connections in the ESTABLISHED state that exist in order of multiples of 10. I've seen instances of up to 140 connections per node.

Is this expected?

normj commented 9 years ago

That is not expected, I'll take a look and see what I can find.

rhargreaves commented 9 years ago

More detail -

It seems to increase gradually with time, in increments of 10. It also seems to affect a single node more than others. I had thought that it could be because this is the node selected as the configuration endpoint, but my tests weren't conclusive and could be just co-incidental. I find that one node will seem to have a large number, whereas other nodes may have 10 or 20, but never >100. I left an app running overnight and it reached 340 connections for a single node.

My setup is 3 nodes in a cluster. However, when I add a node to the cluster, this seems to 'reset' all the connections back to a nominal level (10 per node) after around a minute or so, although they soon build up again. Removing a node has the same effect.

The code I'm using is:

MemcachedClient client = new MemcachedClient(new ElastiCacheClusterConfig(
    ConfigurationManager.AppSettings["ConfigEndpointHost"], 11211));

while (true)
{
    client.Store(StoreMode.Set, "Test", "Test");
    Console.WriteLine(client.Get("Test").ToString());
    Thread.Sleep(60000);
}
normj commented 9 years ago

I believe I have found the cause in polling logic. I need to do a bit more testing before pushing a fix though.

normj commented 9 years ago

I pushed out version 1.0.0.2 to nuget that fixes the problem. Thanks for pointing out the issue.

rhargreaves commented 9 years ago

Confirmed new build fixes the issue. Thanks for speedy resolution!