Occasionally dropping nodes

[davidsulpy]

I've switched from enyim memchached library over to this library to take advantage of node auto-discovery. At first pass it looks great! However, I've been noticing that occasionally, without identifiable trigger (yet) my connections to any cache node just stop. This issue does not occur when using the enyim memcached directly and specifying all the nodes in the cluster specifically in configuration which leads me to believe the issue is happening somewhere in this client.

Is this issue experienced by anyone else? What can I do to help resolve this?

Cheers, David

[MasonSchneider]

Could you describe what you're experiencing a little more? When they stop do you receive any errors or is data just not being stored/retrieved from the nodes?

[davidsulpy]

I'll try and provide as much detail as I can

Here are some example assets to build context:

ExampleCache wrapper

public class ExampleCache : IExampleCache
{
    private readonly Enyim.Caching.MemcachedClient _client;
    public ExampleCache()
    {
        _client = new Enyim.Caching.MemcachedClient(new Amazon.ElastiCacheCluster.ElastiCacheClusterConfig("clusterClient/memcachedAuth"));
    }

    public void StoreSomeString(string key, string someString, TimeSpan expiry)
    {
        _client.Store(Enyim.Caching.Memcached.StoreMode.Set, key, someString, expiry);
    }

    public string GetSomeString(string key)
    {
        _client.Get<string>(key);
    }
}

configuration

<?xml version="1.0" encoding="utf-8"?>
<configuration>

  <configSections>
    <sectionGroup name="clusterClient">
      <section name="memcachedDifferentCache" type="Amazon.ElastiCacheCluster.ClusterConfigSettings, Amazon.ElastiCacheCluster"/>
      <section name="memcachedExample" type="Amazon.ElastiCacheCluster.ClusterConfigSettings, Amazon.ElastiCacheCluster"/>
    </sectionGroup>
  </configSections>

  <clusterClient>
    <memcachedDifferentCache>
      <!-- configuration endpoint for auto node configuration! -->
      <endpoint hostname="other.xxxxxx.cfg.use1.cache.amazonaws.com" port="11211"/>
    </memcachedDifferentCache>
    <memcachedExample>
      <!-- configuration endpoint for auto node configuration! -->
      <endpoint hostname="example.xxxxxx.cfg.use1.cache.amazonaws.com" port="11211"/>
    </memcachedExample>
  </clusterClient>

</configuration>

// Example Fake Nancy Module
public class MyModule : Module
{
    private readonly IExampleCache _cache;
    public MyApp(IExampleCache cache)
    {
        _cache = cache;

        Post["/cache"] = _ => 
        {
            var someModel = this.Bind<SomeModel>();

            _cache.StoreSomeString(someModel.Key, someModel.Value, TimeSpan.FromDays(1));

            return Response.AsJson(someModel);
        }

        Get["/cache/{key}"] = _ =>
        {
            var key = _.key;

            var item = _cache.GetSomeString(key);

            var someModel = new SomeModel
                            {
                                Key = _.key,
                                Value = item
                            };
        }
    }
}

The above little fake app is a pretend Nancy http app that sets and returns strings by key supported by the memcached cache. The IExampleCache constructor dependency is injected into MyModule with an IoC container that controls it's lifespan as a singleton.

When the app starts up, it lasts for quite some time storing and getting items. But then, at some currently undetermined timeframe, it stops storing and retrieving strings as if the cache has gone offline. I have waited for over 10 min to see if the cache comes back online, but it never does. This is only the case when I'm using the Amazon.ElastiCacheCluster configuration. If I am constructing the MemcachedClient with just the pointer to the configuration settings and the configuration settings look like typical enyim.memcached configuration settings where each node is explicitly specified, I don't experience this problem at all.

I wish I had some errors trapped for you, but I do not. I will try to see if I can trap some in the meantime.

Hope this helps.

[kirtisanghi]

I am facing the same probelm .I configured everything but nothing is storing in memcache. Please provide a solution. Its not working with amazom elasticache or memcache . I struggled a lot. My app.config <?xml version="1.0" encoding="utf-8" ?>

program.cs main method

ElastiCacheClusterConfig config = new ElastiCacheClusterConfig("skillogic-cache1.lzfeej.cfg.use1.cache.amazonaws.com", 11211);

        Console.WriteLine("Creating client...");
       MemcachedClient client = new MemcachedClient(config);
                     if (client.Store(Enyim.Caching.Memcached.StoreMode.Add, "Demo", "Hello World"))
            {
                Console.WriteLine("Stored to cache successfully");
            }
            else
            {
                Console.WriteLine("Did not store to cache successfully");
            }

            Object value;
            if (client.TryGet("Demo", out value))
            {
                Console.WriteLine("Got the value: " + (value as string));
            }
            else
            {
                // Search Database if the get fails
                Console.WriteLine("Checking database because get failed");
            }
        }
        Console.Read();

[davidsulpy]

@kirtisanghi is your issue consistent or does it first work and then stop working?

[kirtisanghi]

Hi David.. My issue is consistent...

[kirtisanghi]

Am I doing anything wrong? I follow all the instruction. Can you please help me?

Thanks in advance.

[davidsulpy]

@kirtisanghi sounds like you're experiencing a different issue then, perhaps.

Does your example code work when you don't use the ElastiCacheClusterConfig and instead use the default memcached configuration with it pointing to one or more nodes in the cluster? Example: https://github.com/enyim/EnyimMemcached/wiki/MemcachedClient-Configuration

When you're testing this, are you testing on a machine that has access to the Elasticache cluster via Cache Security Group?

My suspicion is that if it also doesn't work using the built-in configuration for Enyim Memcached, then you probably have a security group issue.

[kirtisanghi]

When I used Couchbase it was working with default memcached configuration. But I have to use it with Amazon ElastiCache. I observed Enyim.Caching.dll version is different in both the cases. For default memcached do we need to specify nodes in configuration file?

Thanks a lot

[davidsulpy]

Ok, so if it was working with Couchbase then perhaps there is another issue. But, I am able to successfully use Enyim's default configuration for memcached, and also use ElastiCacheClusterCofig and see success (at least initially) with both. My issue is that when using ElastiCacheClusterConfig, after some amount of time, I seem to loose all connection to cache nodes.

I haven't had a chance to turn on debug logs yet because Enyim's logging hooks are out of date.

For you though, try confirming that it works with Enyim's default configuration as specified in that link I gave. The basic difference is that you're specifying each node individually instead of using the auto-discovery magic which this library aims to provide. In Enyim's docs, you can see that nodes can be specified either by app.config or an object implementation of IMemcachedClientConfiguration.

[kirtisanghi]

``` ``` Do I need to specify other properties as well? Is it ok to give server and socket.

[davidsulpy]

Yep, as long as you specify like this:

<servers>
    <!-- make sure you use the same ordering of nodes in every configuration you have -->
    <add address="ip address" port="port number" />
    <add address="ip address" port="port number" />
</servers>

inside the enyim.com/memcached block, it should be right.

[normj]

@davidsulpy I just released version 1.0.0.1 of the client which fixes an issue with the nodes being considered dead nodes and never being reattempted. Can you let me know if that fixes your problem.

[kirtisanghi]

Ok I will use.Thanks

[kirtisanghi]

Hi David,

Is there any way to find the no of items in cache through c# code. I want to test that after flushAll my cache is cleared?

Thanks in Advance

[davidsulpy]

@kirtisanghi https://github.com/enyim/EnyimMemcached/blob/master/MemcachedTest/MemcachedClientTest.cs#L358

If you look at that like (350) you can see how the Enyim team is testing the flush. Additionally there is an unanswered SO question here that may help you: http://stackoverflow.com/questions/18408956/enyim-memcached-client-getting-cluster-cached-item-count

For your tests, I'd recommend doing something like the what the Enyim team is doing.

[kirtisanghi]

Hi

I am using the new released version but I am facing problem. It stored only 6 keys not its not writing to the cache. Please help me.

[kirtisanghi]

It stored only 6 keys now its not writing to the cache.

Is there any limit?

Thanks in advance.

[Eyalesos]

@normj @davidsulpy I have the same issue. I use version 1.0.0.4 Occasionally, my connections to any cache node just stop, the node is considered dead, and the data is not being stored/retrieved during this stop. (for about 2 mins)

Can you please assist, did you find any solution?

Thanks!

[SubramanianERS]

@davidsulpy was your issue resolved? Am having the same issue. With more debugging, I was able to find out that autodiscovery does not work as expected. My key is stored in node A, but to retrieve the value for the node, it is trying to get from node B. And obviously, it returns nothing. It does not occur very frequently, but this issue needs to be resolved in my application. Anybody else facing this issue?

Thanks