Redis Cluster on AWS

[tech-sam]

Hi Karel ,

I had configured as below

redis_host = ${?REDIS_HOST}
redis_port = ${?REDIS_PORT}
cache_source = ${?CACHE_SOURCE}
cache_database = ${?CACHE_DATABASE}

play.cache.redis {

  default-cache:  "test-cache"
  bind-default: false
 source:     ${cache_source}
  instances {
    test-cache {
      host:       ${redis_host}
      port:       ${redis_port}
      database:   ${cache_database}
      cluster: [
        {
                host:       ${redis_host}
        port:       ${redis_port}
        database:   ${cache_database}
        }
      ]

    }
  }
}

so cache source property for development environment is standalone and redis is installed locally and it is working like a charm and once it is moved to production source property changed to cluster and host is changed to remote address like redis2.6vek1s.clustercfg.aps1.cache.amazonaws.com then I am getting error like

2018-01-31 09:37:35.594 [null] [core-akka.actor.default-dispatcher-5] ERROR play.api.cache.redis - Command SETEX for key 'classTag::demotasklk:user-cache:356a192b7913b04c54574d18c28d46e6395428ab' failed. [play.api.cache.redis.DetailedReports $anonfun$doLog$3 79] java.lang.RuntimeException: server not found: no server available
    at redis.RedisCluster.$anonfun$send$2(RedisCluster.scala:161)
    at scala.Option.getOrElse(Option.scala:121)
    at redis.RedisCluster.send(RedisCluster.scala:161)
    at redis.commands.Strings.setex(Strings.scala:88)
    at redis.commands.Strings.setex$(Strings.scala:87)
    at redis.RedisCluster.setex(RedisCluster.scala:20)
    at play.api.cache.redis.connector.RedisConnectorImpl.setTemporally(RedisConnectorImpl.scala:73)
    at play.api.cache.redis.connector.RedisConnectorImpl.$anonfun$set$1(RedisConnectorImpl.scala:60)
    at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:302)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

and I have one more doubt in reference.conf it is mention like at line # 95 Note: When cluster is set, the 'host', 'port', and 'database' properties does not apply. just need some clarity on this any help is appreciated !!!

[brucelau]

I had the same problem!!!

[KarelCemus]

Hi guys,

and I have one more doubt in reference.conf it is mention like at line # 95 Note: When cluster is set, the 'host', 'port', and 'database' properties does not apply. just need some clarity on this

Your test-cache has 4 inner attributes:

• host - applies only with source = standalone
• port - applies only with source = standalone
• database - applies only with source = standalone
• cluster - applies only with source = cluster

Hope this clarifies it.

Next, cluster doesn't support attribute database so you can safely remove it. But that is just a side note.

And regarding the issue, I successfully reproduced it, I'll investigate what's going on here and keep you posted.

[KarelCemus]

This seems not to be issue of play-redis. It's the issue of the internal connector rediscala. There is already opened issue

[KarelCemus]

Can you confirm that this issue applies to you? We can then try to figure out some workaround or support directly by play-redis

[tech-sam]

Just verifying this opened issue will update you in a bit and regarding the configuration I have done so inside the test-cache , host , port and database I am going to use for dev environment so cache_sourcewill update as a standalone and cluster will use for production so just I need to update cache_source = clusteronly, so I thought in dev environment it will ignore the cluster config . Thanks for guiding !!!

[KarelCemus]

Your config is fine, everything is ignored properly. Only the play.cache.redis.instances.test-cache.cluster.database is NEVER used because it is not a valid option. Redis cluster enforces database 0 thus there is not config needed. That's all. Otherwise, the config is alright.

[tech-sam]

@KarelCemus , its the same issue which I am facing , verified the redis installation which is done at Amazon ElastiCache similar to issue , So now we have to find out some workaround !!

[KarelCemus]

Alright, before I'd consider any direct support by play-redis, we need to verify that the solution actually works. The solution is proposed in the linked issue. So I suggest you will implement the workaround using current version of play-redis and if you manage to get it working, then I'll integrate it into the library. Would it work for you?

Workaround:

1. instead of source: cluster, you have to use your own source to be able to inject the solution properly resolving the url. So use source: custom, where custom is any name you want but the reserved names.

2. Implement RedisInstance, more specifically RedisCluster, returning properly resolved nodes as is suggested in the issue.

3. Register that implemented class in your module and qualify it with the name of the source, e.g.,:

bind[ RedisInstance ].qualifiedWith( "custom" ).to( classOf[AwsRedisCluster] )

This should do the thing. Please confirm that. If you manage, post here a solution so I could integrate it into the library.

[KarelCemus]

@ImportSumit Any update on this issue?

[tech-sam]

Hi Karel, My apologies for the delayed response.; I was held up with some other issues .I’d hoped to get back to you sooner.

[francjohny]

Would be great if this can be merged soon if it works.

[KarelCemus]

@francjohny This needs to be verified by someone. I am not sure it works, right now, this is just a proposition needed to be verified. Are you willing to test it? If it would work, I'll implement it into the core

[francjohny]

@KarelCemus I would love to help. I am not sure if I am doing it right though. Redis works like a charm in standalone mode, however I am trying to connect it with Redis Cluster over at AWS. I am able to successfully connect it via the console following this guide. I would love to do the same in the application. This is my Redis configuration:

play.cache.redis {
   instances {
    play {
      cluster:  [
        {
          host:        test.ktm4tj.ng.0001.use2.cache.amazonaws.com
          port:        6379
        }
      ]
    }
   }
  source: cluster
  timeout: 1s
  prefix: null
  dispatcher: akka.actor.default-dispatcher
  invocation: lazy
  recovery: log-and-default
  bind-default: true
  default-cache: play
}

On running the application, I get the following error. The connection seems to fail. I would like to know how to integrate this with Elasticache Redis.

play.api.UnexpectedException: Unexpected exception[TimeoutException: Futures timed out after [10 seconds]]
    at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:186)
    at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
    at play.core.server.AkkaHttpServer.modelConversion(AkkaHttpServer.scala:184)
    at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:190)
    at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:107)
    at akka.stream.impl.fusing.MapAsync$$anon$24.onPush(Ops.scala:1191)
    at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:512)
    at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:475)
    at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:371)
    at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:584)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
    at scala.concurrent.Await$.$anonfun$result$1(package.scala:215)
    at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
    at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)
    at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
    at scala.concurrent.Await$.result(package.scala:142)
    at redis.RedisCluster.<init>(RedisCluster.scala:58)
    at play.api.cache.redis.connector.RedisCommandsCluster.<init>(RedisCommands.scala:94)
    at play.api.cache.redis.connector.RedisCommandsProvider.get$lzycompute(RedisCommands.scala:23)

I've enabled and disabled the following modules:

play.modules {
  enabled += "play.api.cache.redis.RedisCacheModule"
  disabled += "play.api.cache.ehcache.EhCacheModule"
}

Could you guide me through this? Btw, thanks for this valuable contribution.

[francjohny]

My bad. I believe I have to run my application on EC2 and access the Redis cluster from there. I will update you in case i face the similar issue mentioned above.

[francjohny]

So, I was able to connect to my Redis cluster via ssh tunnelling. I am trying to run my application with this redis config

play.cache.redis {
   instances {
    play {
      cluster:  [
        {
          host:        localhost
          port:        6379
        }
      ]
    }
   }
  source: cluster
  timeout: 1s
  prefix: null
  dispatcher: akka.actor.default-dispatcher
  invocation: lazy
  recovery: log-and-default
  bind-default: true
  default-cache: play
}

and ended up getting this error:

[error] a.a.OneForOneStrategy - ByteString(45, 69, 82, 82, 32, 84, 104, 105, 115, 32, 105, 110, 115, 116, 97, 110, 99, 101, 32, 104, 97, 115, 32, 99, 108, 117, 115, 116, 101, 114, 32, 115, 117, 112, 112, 111, 114, 116, 32, 100, 105, 115, 97, 98, 108, 101, 100, 13, 10) (of class akka.util.ByteString$ByteString1C)
scala.MatchError: ByteString(45, 69, 82, 82, 32, 84, 104, 105, 115, 32, 105, 110, 115, 116, 97, 110, 99, 101, 32, 104, 97, 115, 32, 99, 108, 117, 115, 116, 101, 114, 32, 115, 117, 112, 112, 111, 114, 116, 32, 100, 105, 115, 97, 98, 108, 101, 100, 13, 10) (of class akka.util.ByteString$ByteString1C)
    at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:254)
    at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:252)
    at redis.protocol.RedisProtocolReply$$anonfun$5.applyOrElse(RedisProtocolReply.scala:164)
    at redis.protocol.RedisProtocolReply$$anonfun$5.applyOrElse(RedisProtocolReply.scala:164)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at redis.api.clusters.ClusterSlots$$anonfun$1.applyOrElse(Clusters.scala:69)
    at redis.api.clusters.ClusterSlots$$anonfun$1.applyOrElse(Clusters.scala:60)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at redis.Operation.decodeRedisReplyThenComplete(Operation.scala:9)
    at redis.actors.RedisReplyDecoder.decodeRedisReply(RedisReplyDecoder.scala:64)
[error] application - 

@KarelCemus I'm not sure if this is the same error. Could you please confirm?

[KarelCemus]

@francjohny is your redis instance configured as a cluster? usually, in cluster mode, you have multiple nodes, not just one running at default port for a standalone instance.

Redis works like a charm in standalone mode, however I am trying to connect it with Redis Cluster over at AWS.

Yes, this bug applies to cluster mode on AWS or when the domain name is used instead of the IP address.

Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]

I am not sure if this is the same error, the original error above is a bit different. However, your error suggests that the connection to redis failed and is not established, so yes, it might be related if and only if you are running it in cluster mode on AWS.

If so, could you try this workaround I proposed? It should help in this case.

[francjohny]

@KarelCemus Sorry for the long wait, was a little busy. As you thought so, the redis instance was not configured as a cluster. I am however trying a local setup with 3 redis instances configured as:

instances {
    play {
      cluster:  [
        {
          host:        localhost
          port:        6379
        },
        {
          host:        localhost
          port:        6380
        },
        {
          host:        localhost
          port:        6381
        }
      ]
    }
   }

  source: cluster
}

Happened to fall into this issue. Would be great if it could be merged if its verified. Thanks.

[KarelCemus]

What exactly do you mean? There are multiple things on the table

1. old version of Akka
2. NPE in OneForOneStrategy

I'm not sure the first is really issue, it seems more like a warning but yes, that PR resolves it. And regarding the other one, that is a race condition, I created a PR into rediscala project but waiting for the next release.

[francjohny]

@KarelCemus To be precise, I ran into this issue.

[error] a.a.OneForOneStrategy - null
java.lang.NullPointerException: null
    at redis.RedisCluster.$anonfun$onConnectStatus$1(RedisCluster.scala:46)
    at redis.RedisCluster.$anonfun$onConnectStatus$1$adapted(RedisCluster.scala:41)
    at redis.actors.RedisWorkerIO.onConnected(RedisWorkerIO.scala:70)
    at redis.actors.RedisWorkerIO$$anonfun$connecting$1.applyOrElse(RedisWorkerIO.scala:57)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
    at akka.actor.Actor.aroundReceive(Actor.scala:517)
    at akka.actor.Actor.aroundReceive$(Actor.scala:515)
    at redis.actors.RedisWorkerIO.aroundReceive(RedisWorkerIO.scala:15)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
    at akka.actor.ActorCell.invoke(ActorCell.scala:496)

[error] a.a.OneForOneStrategy - ByteString(45, 69, 82, 82, 32, 84, 104, 105, 115, 32, 105, 110, 115, 116, 97, 110, 99, 101, 32, 104, 97, 115, 32, 99, 108, 117, 115, 116, 101, 114, 32, 115, 117, 112, 112, 111, 114, 116, 32, 100, 105, 115, 97, 98, 108, 101, 100, 13, 10) (of class akka.util.ByteString$ByteString1C)
scala.MatchError: ByteString(45, 69, 82, 82, 32, 84, 104, 105, 115, 32, 105, 110, 115, 116, 97, 110, 99, 101, 32, 104, 97, 115, 32, 99, 108, 117, 115, 116, 101, 114, 32, 115, 117, 112, 112, 111, 114, 116, 32, 100, 105, 115, 97, 98, 108, 101, 100, 13, 10) (of class akka.util.ByteString$ByteString1C)
    at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:254)
    at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:252)
    at redis.protocol.RedisProtocolReply$$anonfun$5.applyOrElse(RedisProtocolReply.scala:164)
    at redis.protocol.RedisProtocolReply$$anonfun$5.applyOrElse(RedisProtocolReply.scala:164)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at redis.api.clusters.ClusterSlots$$anonfun$1.applyOrElse(Clusters.scala:69)
    at redis.api.clusters.ClusterSlots$$anonfun$1.applyOrElse(Clusters.scala:60)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
    at redis.Operation.decodeRedisReplyThenComplete(Operation.scala:9)
    at redis.actors.RedisReplyDecoder.decodeRedisReply(RedisReplyDecoder.scala:64)

[KarelCemus]

The first one is already resolved and waits for the new release of rediscala.

The second one seems like you have wrong configuration, either using cluster as standlone or vice versa.

[francjohny]

@KarelCemus Thanks for the quick reply. I will try again with the updated fix and let you know.

[francjohny]

@KarelCemus Thanks! The first fix worked! 👍

[KarelCemus]

@francjohny Actually, I'm quite confused about the status of this issue. We are discussing here at least 3 different issues.

1. Issue of running play-redis on AWS - there is pending workaround and needs to be tested. Any progress on this?
2. the NPE within rediscala, it is already resolved but is waiting of the next release
3. your probably wrong set up of the redis database.

Which of these do you mean? Is there any progress on the first issue?

[KarelCemus]

@francjohny @ImportSumit @brucelau What's the status of this issue? Has anyone tried the workaround?

This discussion went off and needs to get back on the track to resolve and close this issue.

[KarelCemus]

This issue is inactive for long time and the discussion went off the road, so I'm closing the issue.

• Akka was updated to 2.5.6 version so it should resolve some issues mentioned here
• Rediscala connector was updated to fixed the NPE in OneForOneStrategy

Both fixes appear in 2.1.0 of play-redis

There is no progress in the original issue with running the cluster on AWS since the beginning. There is no feedback on the proposed workaround, so I am closing it without resolving it. If the issue will appear again in future, feel free to reopen this ticket after testing the proposed workaround. If the workaround works, I can implement a direct support inside the play-redis to avoid the manual workaround.

[shja88]

FYI, I tried implementing the workaround but either I did not implemented it as it should have been done or its just not working. anyway, thanks for the help

[KarelCemus]

@shja88 Can you attach your code so I could review it? It doesn't feel right that it is not working.

[tech-sam]

Hi Karel , hope you are doing good !! , Sorry for the very long delay , this time will try to fix this issue asap as I have prioritize this issue only , I was going through your workaround but need some more info as I am using java implementation of play , it is good if you can put some sample code of config with custom source and implementation of redis instance

[KarelCemus]

Try this, but it is Scala code, you have to convert it into Java yourself. However, I haven't tested it, so there might be a bug.

package play.api.cache.redis

import javax.inject.Inject

import play.api.Configuration
import play.api.cache.redis.configuration._

class AwsRedisCluster @Inject()( configuration: Configuration ) extends RedisCluster with RedisDelegatingSettings {

  def name = "custom"

  def settings = RedisSettings.load( 
    configuration.underlying, 
    "play.cache.redis.instances.custom" 
  )

  def nodes = List(
    RedisHost( "localhost", 6378 )
  )
}

Several notes:

1. Don't forget to register it with Guice

bind[ RedisInstance ].qualifiedWith( "custom" ).to( classOf[AwsRedisCluster] )

2. properly implement the nodes

3. you can put the instance configuration into the configuration file, the settings are loaded from the HOCON, see the play.cache.redis.instances.custom path.

[tech-sam]

I am try to implement this initially with Scala only if it didn't work will try with java , So now after creating a new module like below

class RedisCacheModule extends AbstractModule { override def configure() = { bind[RedisInstance].qualifiedWith("custom").to(classOf[AwsRedisCluster]) } }

getting compile time exception like

ambiguous reference to overloaded definition, both method bind in class AbstractModule of type (x$1: Class[play.api.cache.redis.configuration.RedisInstance])com.google.inject.binder.AnnotatedBindingBuilder[play.api.cache.redis.configuration.RedisInstance] and method bind in class AbstractModule of type (x$1: com.google.inject.TypeLiteral[play.api.cache.redis.configuration.RedisInstance])com.google.inject.binder.AnnotatedBindingBuilder[play.api.cache.redis.configuration.RedisInstance] match expected type ?

and next thing is like my config file is like

play.cache.redis {
  source:    my-cache
  default-cache:  "my-cache
  bind-default: false
  instances {
    my-cache {
      host:       ${redis_host}
      port:       ${redis_port}

    }
  }
}

source is the custom name as my-default and I am using my own implementation of SyncCacheApi also which is working fine below is the sample for that also which is working fine

public class MyCacheModule extends AbstractModule {

    List<String> namedCaches = new ArrayList<String>(Arrays.asList("options-cache", "user-cache", "session-cache"));

    @Override
    protected void configure() {
        Provider<SyncCacheApi> syncCacheApi = getProvider(Key.get(SyncCacheApi.class, Names.named("my-cache")));
        bind(SyncCacheApi.class).toInstance(new MySyncCacheApi(syncCacheApi,"my-cache"));
        namedCaches.forEach(cache -> {
            bind(SyncCacheApi.class).annotatedWith(new NamedCacheImpl(cache))
                    .toInstance(new GtDefaultSyncCacheApi(syncCacheApi, cache));
        });
        Provider<CacheApi> cacheProvider = getProvider(Key.get(CacheApi.class, Names.named("my-cache")));

    }
}

this is the code from AwsRedisCluster class please verify this also

  def settings = RedisSettings.load( 
    configuration.underlying,
    "play.cache.redis.instances.my-cache" 
  )

Just little bit worry about custom implementation will cause any issue

[KarelCemus]

Several issues here:

1. Change bind[RedisInstance] to bind(classOf[RedisInstance])

2. Change .qualifiedWith("custom") to ].qualifiedWith("my-cache") because that is the name you have in the source property in the config file.

3. I'm not sure how about the missing quote in this default-cache: "my-cache

4. host: ${redis_host} and port: ${redis_port} are not actually considered because you use your source. However, you can configure here the properties such as a policy, timeouts etc. See the RedisSettings class to get insight

5. I am using my own implementation of SyncCacheApi Alright but I'd rather go with the default implementations at first to reduce the chance of making some error.

6. Just little bit worry about custom implementation will cause any issue it shouldn't but as I write above, I wouldn't do it now to ease the testing.

[tech-sam]

thanks for prompt reply !! looks like qualifiedWith not a part of com.google.inject.binder.AnnotatedBindingBuilder

it should work like this also

bind(classOf[RedisInstance]).annotatedWith(Names.named("my-cache")).to(classOf[AwsRedisCluster])

but getting exception like

[play.api.http.DefaultHttpErrorHandler logServerError 205] play.api.UnexpectedException: Unexpected exception[Missing: No configuration setting found for key 'play.cache.redis.instances.my-cache.source']
        at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:186)
        at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
        at play.core.server.AkkaHttpServer.modelConversion(AkkaHttpServer.scala:184)
        at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:190)
        at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$3(AkkaHttpServer.scala:107)
        at play.core.server.AkkaHttpServer$$Lambda$1750/787709889.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsync$$anon$23.onPush(Ops.scala:1172)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)
        at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:462)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:368)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:457)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:546)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:725)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:740)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:650)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
        at akka.actor.ActorCell.invoke(ActorCell.scala:496)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'play.cache.redis.instances.my-cache.source'
        at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
        at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
        at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
        at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
        at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
        at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
        at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
        at com.typesafe.config.impl.SimpleConfig.getValue(SimpleConfig.java:203)
        at com.typesafe.config.impl.SimpleConfig.getValue(SimpleConfig.java:37)
        at play.api.cache.redis.configuration.RedisInstanceProvider$.load(RedisInstanceProvider.scala:64)
        at play.api.cache.redis.configuration.RedisInstanceManagerImpl.instanceOfOption(RedisInstanceManager.scala:79)
        at play.api.cache.redis.configuration.RedisInstanceManager.$anonfun$foreach$1(RedisInstanceManager.scala:38)
        at play.api.cache.redis.configuration.RedisInstanceManager$$Lambda$1947/980340958.apply(Unknown Source)
        at scala.collection.TraversableViewLike$FlatMapped.$anonfun$foreach$3$adapted(TraversableViewLike.scala:178)
        at scala.collection.TraversableViewLike$FlatMapped$$Lambda$1948/268157514.apply(Unknown Source)
        at scala.collection.Iterator.foreach(Iterator.scala:929)
        at scala.collection.Iterator.foreach$(Iterator.scala:929)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
        at scala.collection.IterableLike.foreach(IterableLike.scala:71)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
        at scala.collection.IterableLike$$anon$1.foreach(IterableLike.scala:310)
        at scala.collection.TraversableViewLike$FlatMapped.foreach(TraversableViewLike.scala:177)
        at scala.collection.TraversableViewLike$FlatMapped.foreach$(TraversableViewLike.scala:176)
        at scala.collection.IterableViewLike$$anon$5.foreach(IterableViewLike.scala:119)
        at play.api.cache.redis.configuration.RedisInstanceManager.foreach(RedisInstanceManager.scala:38)
        at play.api.cache.redis.configuration.RedisInstanceManager.foreach$(RedisInstanceManager.scala:38)
        at play.api.cache.redis.configuration.RedisInstanceManagerImpl.foreach(RedisInstanceManager.scala:65)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
        at play.api.cache.redis.configuration.RedisInstanceManagerImpl.flatMap(RedisInstanceManager.scala:65)
        at play.api.cache.redis.RedisCacheModule.bindings(RedisCacheModule.scala:26)
        at play.api.inject.guice.GuiceableModuleConversions.guice(GuiceInjectorBuilder.scala:339)
        at play.api.inject.guice.GuiceableModuleConversions.guice$(GuiceInjectorBuilder.scala:338)
        at play.api.inject.guice.GuiceableModule$.guice(GuiceInjectorBuilder.scala:273)
        at play.api.inject.guice.GuiceableModuleConversions$$anon$3.$anonfun$guiced$2(GuiceInjectorBuilder.scala:318)
        at play.api.inject.guice.GuiceableModuleConversions$$anon$3$$Lambda$1926/1656962105.apply(Unknown Source)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$Lambda$1407/140540140.apply(Unknown Source)
        at scala.collection.immutable.List.foreach(List.scala:389)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.immutable.List.map(List.scala:295)
        at play.api.inject.guice.GuiceableModuleConversions$$anon$3.guiced(GuiceInjectorBuilder.scala:318)
        at play.api.inject.guice.GuiceableModule$.$anonfun$guiced$1(GuiceInjectorBuilder.scala:295)
        at play.api.inject.guice.GuiceableModule$$$Lambda$1925/2071341740.apply(Unknown Source)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$Lambda$1440/1261155383.apply(Unknown Source)
        at scala.collection.immutable.List.foreach(List.scala:389)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
        at scala.collection.immutable.List.flatMap(List.scala:352)
        at play.api.inject.guice.GuiceableModule$.guiced(GuiceInjectorBuilder.scala:295)
        at play.api.inject.guice.GuiceBuilder.createModule(GuiceInjectorBuilder.scala:170)
        at play.api.inject.guice.GuiceApplicationBuilder.applicationModule(GuiceApplicationBuilder.scala:109)
        at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:185)
        at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:137)
        at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
        at play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
        at play.core.server.DevServerStart$$anon$1$$Lambda$1888/72066614.apply(Unknown Source)
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
        at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:171)

[tech-sam]

looks like I have missed source property inmy-cache which I am using custom for now , actual error is like

No implementation for play.api.cache.redis.configuration.RedisInstance annotated with @play.cache.NamedCache(value=my-cache) was bound.
  while locating play.api.cache.redis.configuration.RedisInstance annotated with @play.cache.NamedCache(value=my-cache)
  at play.api.cache.redis.GuiceProvider$.bindings(RedisCacheModule.scala:74):
Binding(interface play.api.cache.redis.impl.RedisCaches qualified with QualifierInstance(@play.cache.NamedCache(value=my-cache)) to ProviderTarget(play.api.cache.redis.GuiceRedisCacheProvider@25b77bc8)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.api.cache.redis.impl.RedisCaches annotated with @play.cache.NamedCache(value=my-cache)
  at play.api.cache.redis.GuiceProvider$QualifiedBindingKey.toBindings(RedisCacheModule.scala:63):
Binding(interface play.cache.SyncCacheApi qualified with QualifierInstance(@javax.inject.Named(value=my-cache)) to ProviderTarget(play.api.cache.redis.DeprecatedNamedCacheInstanceProvider@2416c5e6)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.cache.SyncCacheApi annotated with @com.google.inject.name.Named(value=gt-default)

1 error
        at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
        at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:56)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.pac4j.core.exception.TechnicalException: com.google.inject.ProvisionException: Unable to provision, see the following errors:

[KarelCemus]

It seems more like my bad and poor documentation. I revised the source code and obviously, the value of the source property must be custom. I am sorry for misguiding you.

Alright, you are registering this class

@Named("my-cache") class MyInstance

but it requires

@NamedCache("my-cache") class MyInstance

So you need to adjust the module.

bind(classOf[RedisInstance]).annotatedWith(new NamedCacheImpl("my-cache")).to(classOf[AwsRedisCluster])

I hope this works.

[tech-sam]

Sorry Karel , I didn't get where to change @NamedCache("my-cache") class MyInstance my-cache class implementation is like

@Singleton
public class MySyncCacheApi implements SyncCacheApi {

    private Provider<MySyncCacheApi> cacheApiProvider;
    private SyncCacheApi cacheApi;

    private String prefix;

    public MySyncCacheApi(Provider<MySyncCacheApi> cacheApiProvider, String prefix) {
        this.prefix = prefix;
        this.cacheApiProvider = cacheApiProvider;
    }

    private SyncCacheApi getCacheApi() {
        if (cacheApi == null) {
            cacheApi = cacheApiProvider.get();
        }
        return cacheApi;
    }

    }

and how I have register with Guice is like

List<String> namedCaches = new ArrayList<String>(Arrays.asList("options-cache", "user-cache", "session-cache"));
protected void configure() {

        Provider<SyncCacheApi> syncCacheApi = getProvider(Key.get(SyncCacheApi.class, Names.named("my-cache")));
        bind(SyncCacheApi.class).toInstance(new MySyncCacheApi(syncCacheApi,"my-cache"));
        namedCaches.forEach(cache -> {
            bind(SyncCacheApi.class).annotatedWith(new NamedCacheImpl(cache))
                    .toInstance(new MyDefaultSyncCacheApi(syncCacheApi, cache));
        });
        Provider<CacheApi> cacheProvider = getProvider(Key.get(CacheApi.class, Names.named("my-cache")));
    }

so only I am injecting above named cache like@NamedCache("options-cache") SyncCacheApi cache an this was working fine and its already there in production env. now we need to solve this No implementation for play.api.cache.redis.configuration.RedisInstance annotated with @play.cache.NamedCache(value=my-cache) was bound.

[KarelCemus]

Which version of play-redis you have?

[tech-sam]

day before yesterday only I have upgraded to 2.1.0

[KarelCemus]

Alright, then I suggests you to use 2.1.2, there are only minor bug fixes.

However, see the Migration guide, especially the part about the named cache annotation to understand what I meant above. The changelog has the more detailed description.

bind(SyncCacheApi.class).annotatedWith(new NamedCacheImpl(cache))

This is the correct approach with the NamedCacheImpl. You have to do this also with binding the RedisInstance because @NamedCache is expected qualifier.

This explains the error you encounter:

No implementation for play.api.cache.redis.configuration.RedisInstance annotated with @play.cache.NamedCache(value=my-cache) was bound.

It expects to have @NamedCache("my-cache") class MyCache extends RedisInstance but you do not bind it.

Is it clearer now?

[tech-sam]

May be I misunderstood also !! SyncCacheApi is already annotated with named cache which is below

Provider<SyncCacheApi> syncCacheApi = getProvider(Key.get(SyncCacheApi.class, Names.named("my-cache")));
        bind(SyncCacheApi.class).toInstance(new MySyncCacheApi(syncCacheApi, "my-cache"));

and for It expects to have @NamedCache("my-cache") class MyCache extends RedisInstance but you do not bind it. MyCache is nothing but AwsRedisCluster class which extends RedisCluster

class AwsRedisCluster @Inject()( configuration: Configuration ) extends RedisCluster with RedisDelegatingSettings {
    def name = "my-cache"

  def settings = RedisSettings.load( 
    configuration.underlying,
    "play.cache.redis.instances.my-cache" 
  )

  def nodes = List(
    RedisHost( "localhost", 6379 )
  )
}

and this I am registering with guice with namedCache only like

class RedisCacheModule extends AbstractModule {
  override def configure() = {
    bind(classOf[RedisInstance]).annotatedWith(new NamedCacheImpl("my-cache")).to(classOf[AwsRedisCluster])
  }
}

and If I injecting SyncCacheApi anywhere in other classes I am already using @NamedCache
@NamedCache("my-cache")SyncCacheApi

below is my config

play.cache.redis {
  source:          my-cache
  default-cache:  "my-cache"
  bind-default: false
  instances {
    my-cache {
      source:     custom      
    }
  }
}

so what is wrong here , I have upgraded to 2.1.2 also , please verify Thank you for your effort

[KarelCemus]

It seems correct. It doesn't work? What's the exception?

[tech-sam]

No implementation for play.api.cache.redis.configuration.RedisInstance annotated with @play.cache.NamedCache(value=my-cache) was bound.
  while locating play.api.cache.redis.configuration.RedisInstance annotated with @play.cache.NamedCache(value=my-cache)
  at play.api.cache.redis.GuiceProvider$.bindings(RedisCacheModule.scala:74):
Binding(interface play.api.cache.redis.impl.RedisCaches qualified with QualifierInstance(@play.cache.NamedCache(value=my-cache)) to ProviderTarget(play.api.cache.redis.GuiceRedisCacheProvider@191b297d)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.api.cache.redis.impl.RedisCaches annotated with @play.cache.NamedCache(value=my-cache)
  at play.api.cache.redis.GuiceProvider$QualifiedBindingKey.toBindings(RedisCacheModule.scala:62):
Binding(interface play.cache.SyncCacheApi qualified with QualifierInstance(@play.cache.NamedCache(value=my-cache)) to ProviderTarget(play.api.cache.redis.NamedCacheInstanceProvider@3ae8d57d)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)

[KarelCemus]

This is very odd. Are you sure, that this line is actually invoked? Could you debug it, e.g., through a logging statement?

bind(classOf[RedisInstance]).annotatedWith(new NamedCacheImpl("my-cache")).to(classOf[AwsRedisCluster])

Are you sure, that your RedisCacheModule is enabled? Because the issue exactly matches this binding configuration.

[tech-sam]

RedisCacheModule is enable for sure

enabled += "play.api.cache.redis.RedisCacheModule"
disabled += "play.api.cache.ehcache.EhCacheModule"

but one thing I have missed after going through logs I found I am injecting SyncCacheApi like

@Inject
    public TestService(@NamedCache("my-cache")SyncCacheApi cacheApi) {
        super(cacheApi);    
    }

and error is like

Binding(interface play.api.cache.redis.impl.RedisCaches qualified with QualifierInstance(@play.cache.NamedCache(value=my-cache)) to ProviderTarget(play.api.cache.redis.GuiceRedisCacheProvider@191b297d)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.api.cache.redis.impl.RedisCaches annotated with @play.cache.NamedCache(value=my-cache)
  at play.api.cache.redis.GuiceProvider$QualifiedBindingKey.toBindings(RedisCacheModule.scala:62):
Binding(interface play.cache.SyncCacheApi qualified with QualifierInstance(@play.cache.NamedCache(value=my-cache)) to ProviderTarget(play.api.cache.redis.NamedCacheInstanceProvider@3ae8d57d)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.cache.SyncCacheApi annotated with @play.cache.NamedCache(value=my-cache)
    for the 1st parameter of com.common.service.TestService.<init>(TestService.java:60)
  while locating com.common.service.TestService

[tech-sam]

I am putting logs to debug bind(classOf[RedisInstance]).annotatedWith(new NamedCacheImpl("my-cache")).to(classOf[AwsRedisCluster])

[tech-sam]

looks like got the issue I have not enable the my RedisCacheModule itself , really sorry for this silly mistakes need to change the RedisCacheModule name

[KarelCemus]

I have not enable the my RedisCacheModule itself

That's what I meant. I'm glad we've found it.

[tech-sam]

looks like God is not in good mood today

Error in custom provider, java.util.NoSuchElementException: None.get
  at play.api.cache.redis.GuiceProvider$QualifiedBindingKey.toBindings(RedisCacheModule.scala:62):
Binding(interface play.cache.SyncCacheApi qualified with QualifierInstance(@play.cache.NamedCache(value=my-cache)) to ProviderTarget(play.api.cache.redis.NamedCacheInstanceProvider@362c389)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.cache.SyncCacheApi annotated with @play.cache.NamedCache(value=my-cache)
    for the 1st parameter of com.common.service.TestService.<init>(TestService.java:45)
  while locating com.common.service.TestService

error is for every class wherever I am injecting @NamedCache("my-cache")SyncCacheApi cacheApi

[KarelCemus]

I'll take a look at it later today, I'll write my own example and let you know

[tech-sam]

TestService Injection error I have resolved , and I am feeling that we are very close to fix also , some how play.api.cache.redis.RedisCacheModule is not able to inject my cache provider

and I am having solid doubt also

bind(classOf[RedisInstance]).annotatedWith(new NamedCacheImpl("my-cache")).to(classOf[AwsRedisCluster])

RedisInstance should not be annotated With my-cache as it is a Redis Server Instance not a Cache Instance

1) Error in custom provider, java.util.NoSuchElementException: None.get
  at play.api.cache.redis.GuiceProvider$QualifiedBindingKey.toBindings(RedisCacheModule.scala:63):
Binding(interface play.cache.SyncCacheApi qualified with QualifierInstance(@javax.inject.Named(value=my-cache)) to ProviderTarget(play.api.cache.redis.DeprecatedNamedCacheInstanceProvider@1c489af0)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
  while locating play.cache.SyncCacheApi annotated with @com.google.inject.name.Named(value=my-cache)

1 error
        at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
        at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:56)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.pac4j.core.exception.TechnicalException: com.google.inject.ProvisionException: Unable to provision, see the following errors:

[tech-sam]

Hi Karel , Any Update ??