search

KarelCemus / play-redis

Play framework 2 cache plugin as an adapter to redis-server
Mozilla Public License 2.0
164 stars 44 forks source link

Issue where "server not found" but is there OR extra "classTag::" in Key? #243

Closed arigoldx closed 3 years ago

arigoldx commented 3 years ago

I think this might just be a case of me not setting things up correctly and therefore I want to say at the outset that once this is cleared up I'm more than happy to open a PR adding to the documentation.

My setup

play-redis 2.6.1 Play 2.8.1 Scala 2.13.2

Issue

When my play application loads I see

[info] p.a.cache.redis - Redis cluster cache actor started. It is connected to  localhost:7002
[info] r.a.RedisClientActor - Connect to /127.0.0.1:7002
[info] r.a.RedisClientActor - Connected to localhost/127.0.0.1:7002
[info] p.a.c.r.c.RedisCommandsCluster$$anon$2 - refreshClusterSlots: Map(ClusterSlot(10923,16383,ClusterNode(127.0.0.1,7002,6a99c1396c144fdfa3f70966f3094b8f5b8a925c),List(ClusterNode(127.0.0.1,7004,4b91647a852298dadf957121786cc05819e0c8aa))) -> RedisConnection(Actor[akka://application/user/RedisClientPool-$b#-1366800206],scala.concurrent.stm.ccstm.CCSTMRefs$BooleanRef@674ecd2b))

which makes me think that all is well.

But then when running the application, I see a few errors, the first of which is:

[error] p.a.cache.redis - Command GET for key 'classTag::eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.ff7q3dXyyvZ7_7s2SJcGByhjrIZVWhjySYNvXipkYC95YHnr-bb1xzL_Hjs2-5BPf2l62XTNviyOvwivNsiVAXcewiN1FFzz7chQZKqz6DJAmApfU-djvgh1t5njRSa0YhPidBeLLxXCzfCUfcxZlHhhtRwUwzQuOAGDb4M_9kE._epRtJX8x0rUFaB9.tc5gaIhtXB4wqR9mk-jeezr5-nyNwj3qlObYe_RL0UzzA4HWWHn2aB0zh44Vk5TUdM736anZBeeES4MocyHToBmvm3qFuv5L6XvEro87eyApJzEXE10_dO_V2yflv166WDqpzD3z9fJHidPgiD1uWPcDj00r_202g3nA21h0s3Ebyxaw_Ds7alY66wzVCwUibiyTqViDAj--NMCnd3x485m2ECNOMerD3sCo27EigALRNnn6L8BwWcYVT1uMT76UYoxN8Loi7XWD-LG43kuT.0vQFTbhbPyrmbKZvEbAPCA' failed.
java.lang.RuntimeException: server not found: no server available
    at redis.RedisCluster.$anonfun$send$2(RedisCluster.scala:159)
    at scala.Option.getOrElse(Option.scala:201)
    at redis.RedisCluster.send(RedisCluster.scala:159)
    at play.api.cache.redis.connector.RedisCommandsCluster$$anon$2.play$api$cache$redis$connector$RedisRequestTimeout$$super$send(RedisCommands.scala:106)
    at play.api.cache.redis.connector.RedisRequestTimeout.continue$2(RequestTimeout.scala:74)
    at play.api.cache.redis.connector.RedisRequestTimeout.$anonfun$send$4(RequestTimeout.scala:76)
    at scala.Option.fold(Option.scala:263)
    at play.api.cache.redis.connector.RedisRequestTimeout.send(RequestTimeout.scala:76)
    at play.api.cache.redis.connector.RedisRequestTimeout.send$(RequestTimeout.scala:72)
    at play.api.cache.redis.connector.RedisCommandsCluster$$anon$2.send(RedisCommands.scala:106)

I'm running a local redis cluster as per the redis cluster tutorial and via the cluster nodes command in redis-cli the cluster appears up and running. Furthermore, when I connect to the redis cluster listed above (or if i sea

127.0.0.1:7002> keys *
1) "eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.ff7q3dXyyvZ7_7s2SJcGByhjrIZVWhjySYNvXipkYC95YHnr-bb1xzL_Hjs2-5BPf2l62XTNviyOvwivNsiVAXcewiN1FFzz7chQZKqz6DJAmApfU-djvgh1t5njRSa0YhPidBeLLxXCzfCUfcxZlHhhtRwUwzQuOAGDb4M_9kE._epRtJX8x0rUFaB9.tc5gaIhtXB4wqR9mk-jeezr5-nyNwj3qlObYe_RL0UzzA4HWWHn2aB0zh44Vk5TUdM736anZBeeES4MocyHToBmvm3qFuv5L6XvEro87eyApJzEXE10_dO_V2yflv166WDqpzD3z9fJHidPgiD1uWPcDj00r_202g3nA21h0s3Ebyxaw_Ds7alY66wzVCwUibiyTqViDAj--NMCnd3x485m2ECNOMerD3sCo27EigALRNnn6L8BwWcYVT1uMT76UYoxN8Loi7XWD-LG43kuT.0vQFTbhbPyrmbKZvEbAPCA"

which is almost exactly the key that is wanted in the "server not found" error above.

Lastly, I've got the right version of play-redis in my build.sbt (2.6.1) and here's my config:

play.cache.redis {
  bind-default = true

  instances {
    role-cache {
      source = cluster
      cluster = [
        { host = localhost, port = 7001 }
      ]
    }
    authenticated-user-cache {
      source = cluster
      cluster = [
        { host = localhost, port = 7002 }
      ]
    }
    play {
      source = cluster
      cluster = [
        { host = localhost, port = 7000 }
      ]
    }
  }
}

I noticed that the key in redis starts with eyJlbm... but the error shows that it's looking for a key that starts classTag::eyJlbm... so maybe (probably) the issue is the extra classTag?

Any ideas? It almost seems like there's a disconnect between when the SET and GET, a disconnect that adds classTag:: 🤔.

(Thanks so much!)

arigoldx commented 3 years ago

Given that the issue might be related to the extra classTag I thought that I should mention that since we're serializing our own class, I'm using jackson serialization as per the akka docs that are linked to. Here's my config:

akka {
  actor {
    serializers {
      jackson-cbor = "akka.serialization.jackson.JacksonCborSerializer"
    }

    serialization-bindings {
      "services.masterdata.User" = jackson-cbor
    }
  }
}
KarelCemus commented 3 years ago

This seems that your redis configuration is invalid. The setup you shared shows just one node but the cluster mode. The minimal number of nodes in a cluster mode is three, which is also mentioned in the tutorial you linked:

Note that the minimal cluster that works as expected requires to contain at least three master nodes. For your first tests it is strongly suggested to start a six nodes cluster with three masters and three slaves.

So it may be the primary issue that the connection is not established since the redis setup is invalid.

Secondary, you don't mention which play-redis API you use. Most advanced APIs accept a class tag as a parameter but Play's APIs don't. To overcome this limitation of Play's API, the play-redis stores the class tag along the actual value to be able to recover it. However, this requires consistent access to the value - get and set via the same API, either Play's or the advanced API by play-redis. If you set the value via one API and access it via the other, it may not work since the classtag might be missing.

Start with fixing your redis setup.

Note: the connection is established lazily if I remember correctly so that is why it fails when you first time use it.

arigoldx commented 3 years ago

Hmm.. I set up redis just like the tutorial and I'm pretty sure there are 3 master nodes:

~  $ redis-cli -c -p 7002
127.0.0.1:7002> cluster nodes
239efc0edeb6f887ee8955645e0427a67e343f45 127.0.0.1:7003@17003 slave 1b2277cfd9410524bf5384181f76cebc2508563f 0 1613662292020 2 connected
1b2277cfd9410524bf5384181f76cebc2508563f 127.0.0.1:7001@17001 master - 0 1613662291000 2 connected 5461-10922
4b91647a852298dadf957121786cc05819e0c8aa 127.0.0.1:7004@17004 slave 6a99c1396c144fdfa3f70966f3094b8f5b8a925c 0 1613662291503 3 connected
f2cb7a175f43f42a915245a4967221ca7781e15b 127.0.0.1:7000@17000 master - 0 1613662290881 1 connected 0-5460
6a99c1396c144fdfa3f70966f3094b8f5b8a925c 127.0.0.1:7002@17002 myself,master - 0 1613662289000 3 connected 10923-16383
10d768e3f832bdcf43341bbf62253f4027b0d9ec 127.0.0.1:7005@17005 slave f2cb7a175f43f42a915245a4967221ca7781e15b 0 1613662290985 1 connected

Good point on not posting my use of the play-redis API -- I think I was concerned that I was quoting too much config 😅. In any event I've got this in both java files where we use the cache:

import play.cache.redis.AsyncCacheApi;
import play.cache.NamedCache;

and then

public class MasterdataService {
  @Inject
  @NamedCache("authenticated-user-cache")
  public AsyncCacheApi userCache;

Seems like something is working 'cause data is making it into redis 😖.

Any idea about what that extra classTag:: is doing after the [error] p.a.cache.redis - Command GET for key?

Thanks!

KarelCemus commented 3 years ago

If you have 3 nodes you must list them all in the play-redis config, you list just one

arigoldx commented 3 years ago

Well that sure seems like it help a bunch. Thanks! :)

[info] r.a.RedisClientActor - Connect to /127.0.0.1:7000
[info] r.a.RedisClientActor - Connect to /127.0.0.1:7001
[info] r.a.RedisClientActor - Connect to /127.0.0.1:7002
[info] r.a.RedisClientActor - Connected to localhost/127.0.0.1:7001
[info] r.a.RedisClientActor - Connected to localhost/127.0.0.1:7002
[info] r.a.RedisClientActor - Connected to localhost/127.0.0.1:7000

In fact, I'm not seeing any "server not found" errors now which probably closes this issue. I'll open a PR adding the list-all-nodes info above to play-redis's cluster example because right now the example shows only one node.

Now my issue is regarding serialization which seems out of scope for play-redis... hopefully I'll get jackson working 😅.

arigoldx commented 3 years ago

So I was able to get serialization working. :tada: Fwiw, I had trouble when using jackson-cbor so I switched to jackson-json and, lo and behold, I had no more errors.

That was Friday.

Today I tried again -- with zero change to the setup -- and am getting the same error as I did earlier:

[error] p.a.cache.redis - Command GET for key 'classTag::eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.WVNcMzr3yyQpp7wLCNc82E_59p6HqtCB3kCvMd4-ZMAxiIJKJuSrmTdzDtPPiD76vaRl1VmqMzfk6uQxp_1hIF9zfLwbzKJrazhMlb_8JsMN7kJnM5_GH7D8UiR_euPFyPkf0JLB9Y6DaSs75etqZuPowFHw_pxv5VbuZz6Xi_8.r4fTZfB_I3dkhnq4.Z0fybeKZYG8mLszdC8cVlCqgFr5QSMrpLXjfO08KeUvjlSEmxYu8KbvGriTZAjHWSi0S_Rrq6laEPCmz5qyCDDPDUZJwX5W7m0ot6onKLdcdTWaykMFUSHrKnI7kHAlYHwjrML8WGJeYhvr7cyyOSyHTiHcvuC-WmJLYWUI9GLvluRy7RD5e7zIUewp8b1bfGKemQtLzoYKqEJoHWFtbMp9wVCg0L5479Emm_7WYFzguPiDBjQcIKH9Da7CR_Lbjw-jM6V5pblCl7Cp7xgyd.3IgzjxjsaC6A9AvI8_xAwQ' failed.
java.lang.RuntimeException: server not found: no server available
    at redis.RedisCluster.$anonfun$send$2(RedisCluster.scala:159)
    at scala.Option.getOrElse(Option.scala:201)
    at redis.RedisCluster.send(RedisCluster.scala:159)
    at play.api.cache.redis.connector.RedisCommandsCluster$$anon$2.play$api$cache$redis$connector$RedisRequestTimeout$$super$send(RedisCommands.scala:106)
    at play.api.cache.redis.connector.RedisRequestTimeout.continue$2(RequestTimeout.scala:74)
    at play.api.cache.redis.connector.RedisRequestTimeout.$anonfun$send$4(RequestTimeout.scala:76)
    at scala.Option.fold(Option.scala:263)
    at play.api.cache.redis.connector.RedisRequestTimeout.send(RequestTimeout.scala:76)
    at play.api.cache.redis.connector.RedisRequestTimeout.send$(RequestTimeout.scala:72)
    at play.api.cache.redis.connector.RedisCommandsCluster$$anon$2.send(RedisCommands.scala:106)

as per my previous comment RedisClientActor is connected to the three primary cluster nodes. And as per my comment from before that, the cluster shows that data made it in:

~  $ redis-cli --cluster call localhost:7000 KEYS "*"
>>> Calling KEYS *
localhost:7000:
127.0.0.1:7005:
127.0.0.1:7004: eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.JyYH3AhJIS2WNtIUrS3K7PJtjXGwX9PpKZYnpbXy7xZUEgkNctCB7ZuSVIYgsQ5WZ7km7_As2mpwU_sGqYkMmbeJFtKEFltk51H8wxm5F0Y5X-W1wWZRMfw3aoQ28PL9ROD8osbBIO4NaKqn0ETR-Oyf5wsYOWjilc9Y1rXP3dE.tCL77UXYlu7OR3H-.55PKECGXi_tQiJuSGxsu9daj_SwHVhIOQP1swI5RBSp-QRH-QxvNtQSUeM_PIH7lwOzXDML4DxGxhxlUTIqSK7q4pBCd7KPon0wdYkhB9CTSEHLJy1aP4VaWgtPpm4PFUdRC6jXWsIeh0KlblLptEfnKvS6LefgS_DbEOweaL-mD39NBb1n1VpCvrQDdkN0JlFGMK3b8xbpE9lttFJqnL16kZTFUNAlLxHpHbsEW7wUNm5oUVwHOviTFw9OpCKnZfoTKlKndgtRNNia2LEqY.Bhl1hltTF8yZvp7TjzI9rw
eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.WVNcMzr3yyQpp7wLCNc82E_59p6HqtCB3kCvMd4-ZMAxiIJKJuSrmTdzDtPPiD76vaRl1VmqMzfk6uQxp_1hIF9zfLwbzKJrazhMlb_8JsMN7kJnM5_GH7D8UiR_euPFyPkf0JLB9Y6DaSs75etqZuPowFHw_pxv5VbuZz6Xi_8.r4fTZfB_I3dkhnq4.Z0fybeKZYG8mLszdC8cVlCqgFr5QSMrpLXjfO08KeUvjlSEmxYu8KbvGriTZAjHWSi0S_Rrq6laEPCmz5qyCDDPDUZJwX5W7m0ot6onKLdcdTWaykMFUSHrKnI7kHAlYHwjrML8WGJeYhvr7cyyOSyHTiHcvuC-WmJLYWUI9GLvluRy7RD5e7zIUewp8b1bfGKemQtLzoYKqEJoHWFtbMp9wVCg0L5479Emm_7WYFzguPiDBjQcIKH9Da7CR_Lbjw-jM6V5pblCl7Cp7xgyd.3IgzjxjsaC6A9AvI8_xAwQ
classTag::GET /users:e00ef304-0121-43b4-9af7-38ddb0a42e04
127.0.0.1:7002: classTag::GET /users:e00ef304-0121-43b4-9af7-38ddb0a42e04
eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.WVNcMzr3yyQpp7wLCNc82E_59p6HqtCB3kCvMd4-ZMAxiIJKJuSrmTdzDtPPiD76vaRl1VmqMzfk6uQxp_1hIF9zfLwbzKJrazhMlb_8JsMN7kJnM5_GH7D8UiR_euPFyPkf0JLB9Y6DaSs75etqZuPowFHw_pxv5VbuZz6Xi_8.r4fTZfB_I3dkhnq4.Z0fybeKZYG8mLszdC8cVlCqgFr5QSMrpLXjfO08KeUvjlSEmxYu8KbvGriTZAjHWSi0S_Rrq6laEPCmz5qyCDDPDUZJwX5W7m0ot6onKLdcdTWaykMFUSHrKnI7kHAlYHwjrML8WGJeYhvr7cyyOSyHTiHcvuC-WmJLYWUI9GLvluRy7RD5e7zIUewp8b1bfGKemQtLzoYKqEJoHWFtbMp9wVCg0L5479Emm_7WYFzguPiDBjQcIKH9Da7CR_Lbjw-jM6V5pblCl7Cp7xgyd.3IgzjxjsaC6A9AvI8_xAwQ
eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.JyYH3AhJIS2WNtIUrS3K7PJtjXGwX9PpKZYnpbXy7xZUEgkNctCB7ZuSVIYgsQ5WZ7km7_As2mpwU_sGqYkMmbeJFtKEFltk51H8wxm5F0Y5X-W1wWZRMfw3aoQ28PL9ROD8osbBIO4NaKqn0ETR-Oyf5wsYOWjilc9Y1rXP3dE.tCL77UXYlu7OR3H-.55PKECGXi_tQiJuSGxsu9daj_SwHVhIOQP1swI5RBSp-QRH-QxvNtQSUeM_PIH7lwOzXDML4DxGxhxlUTIqSK7q4pBCd7KPon0wdYkhB9CTSEHLJy1aP4VaWgtPpm4PFUdRC6jXWsIeh0KlblLptEfnKvS6LefgS_DbEOweaL-mD39NBb1n1VpCvrQDdkN0JlFGMK3b8xbpE9lttFJqnL16kZTFUNAlLxHpHbsEW7wUNm5oUVwHOviTFw9OpCKnZfoTKlKndgtRNNia2LEqY.Bhl1hltTF8yZvp7TjzI9rw
127.0.0.1:7001:
127.0.0.1:7003:

Looks like we just can't get data out. Its as if the server was available for the PUT but the connection was lost before the GET. For a while I thought that maybe the issue with key because the error is for a key that starts with classTag::eyJlbm but redis only shows a key starting with eyJlbm (no "classTag::"). That said the error says "server not found" not "key not found". :thinking:

I'm going to restart my redis cluster as well as restart sbt. And see what that does.

arigoldx commented 3 years ago

That did it.

I didn't only restart the cluster nodes & rebuild the cluster. I also cleaned out the appendonly.aof and nodes. conf files so that the cluster started super fresh ✨.

Interesting that the key that is the difference between the current, correct-and-working set of keys and the one from before when things weren't working is the one from the error message, the one that starts "classTag::eyJlbm"... :tada:!

~  $ redis-cli --cluster call localhost:7000 KEYS "*"
>>> Calling KEYS *
localhost:7000:
127.0.0.1:7004: eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.O8VEww0iwW0Fg-U4b7BCYME2LqDYvdD0kfe_ThhoV6QzKuniVsdufOVtnxHZziKVMbsulJQZKPtTetzT1QFZ6KCU05FMZsr6d0OsNDIKxzwol5zlIfaHS0OIfWWTIYfVPwLPWe6J0kYdWwv8enoKDE5P_swGKFXzSIUCT5M3rGY.GXmgLMEUDDThHazN.cm5yFZB8keUYGL-M_lt6TxxlA_yN4YbgZL1AziA__S-PEONm-HbRfwDQd4KzyGrD0Mdg7Dm4yVqE73l0ly21LvIOtqiZUD1JMTaNoXAYJzDtqZePbATzlZQW2IFytEqoDn5luEPJtgD9gs3D3rrBwyfThpBSXfBhrcUe14hSZBZY7rHi5mPIWSzME049JRBJJwgrXz_THmtuRnSHsOTRZjaualmoKQ3N0Ujns97r5vQgQd-LOyUld6YsfmpUVeEBiQ3O5vKKxCoV1Pn3KQVB.1mgYvY7hZkl7dspjOpt2IQ
classTag::eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.O8VEww0iwW0Fg-U4b7BCYME2LqDYvdD0kfe_ThhoV6QzKuniVsdufOVtnxHZziKVMbsulJQZKPtTetzT1QFZ6KCU05FMZsr6d0OsNDIKxzwol5zlIfaHS0OIfWWTIYfVPwLPWe6J0kYdWwv8enoKDE5P_swGKFXzSIUCT5M3rGY.GXmgLMEUDDThHazN.cm5yFZB8keUYGL-M_lt6TxxlA_yN4YbgZL1AziA__S-PEONm-HbRfwDQd4KzyGrD0Mdg7Dm4yVqE73l0ly21LvIOtqiZUD1JMTaNoXAYJzDtqZePbATzlZQW2IFytEqoDn5luEPJtgD9gs3D3rrBwyfThpBSXfBhrcUe14hSZBZY7rHi5mPIWSzME049JRBJJwgrXz_THmtuRnSHsOTRZjaualmoKQ3N0Ujns97r5vQgQd-LOyUld6YsfmpUVeEBiQ3O5vKKxCoV1Pn3KQVB.1mgYvY7hZkl7dspjOpt2IQ
classTag::GET /users:e00ef304-0121-43b4-9af7-38ddb0a42e04
127.0.0.1:7001: GET /users/0a6a0267-29aa-43ac-8e15-b14f2278ff6d:e00ef304-0121-43b4-9af7-38ddb0a42e04
GET /users:e00ef304-0121-43b4-9af7-38ddb0a42e04
classTag::GET /users/0a6a0267-29aa-43ac-8e15-b14f2278ff6d:e00ef304-0121-43b4-9af7-38ddb0a42e04
127.0.0.1:7005:
127.0.0.1:7002: classTag::GET /users:e00ef304-0121-43b4-9af7-38ddb0a42e04
classTag::eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.O8VEww0iwW0Fg-U4b7BCYME2LqDYvdD0kfe_ThhoV6QzKuniVsdufOVtnxHZziKVMbsulJQZKPtTetzT1QFZ6KCU05FMZsr6d0OsNDIKxzwol5zlIfaHS0OIfWWTIYfVPwLPWe6J0kYdWwv8enoKDE5P_swGKFXzSIUCT5M3rGY.GXmgLMEUDDThHazN.cm5yFZB8keUYGL-M_lt6TxxlA_yN4YbgZL1AziA__S-PEONm-HbRfwDQd4KzyGrD0Mdg7Dm4yVqE73l0ly21LvIOtqiZUD1JMTaNoXAYJzDtqZePbATzlZQW2IFytEqoDn5luEPJtgD9gs3D3rrBwyfThpBSXfBhrcUe14hSZBZY7rHi5mPIWSzME049JRBJJwgrXz_THmtuRnSHsOTRZjaualmoKQ3N0Ujns97r5vQgQd-LOyUld6YsfmpUVeEBiQ3O5vKKxCoV1Pn3KQVB.1mgYvY7hZkl7dspjOpt2IQ
eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.O8VEww0iwW0Fg-U4b7BCYME2LqDYvdD0kfe_ThhoV6QzKuniVsdufOVtnxHZziKVMbsulJQZKPtTetzT1QFZ6KCU05FMZsr6d0OsNDIKxzwol5zlIfaHS0OIfWWTIYfVPwLPWe6J0kYdWwv8enoKDE5P_swGKFXzSIUCT5M3rGY.GXmgLMEUDDThHazN.cm5yFZB8keUYGL-M_lt6TxxlA_yN4YbgZL1AziA__S-PEONm-HbRfwDQd4KzyGrD0Mdg7Dm4yVqE73l0ly21LvIOtqiZUD1JMTaNoXAYJzDtqZePbATzlZQW2IFytEqoDn5luEPJtgD9gs3D3rrBwyfThpBSXfBhrcUe14hSZBZY7rHi5mPIWSzME049JRBJJwgrXz_THmtuRnSHsOTRZjaualmoKQ3N0Ujns97r5vQgQd-LOyUld6YsfmpUVeEBiQ3O5vKKxCoV1Pn3KQVB.1mgYvY7hZkl7dspjOpt2IQ
127.0.0.1:7003: GET /users:e00ef304-0121-43b4-9af7-38ddb0a42e04
GET /users/0a6a0267-29aa-43ac-8e15-b14f2278ff6d:e00ef304-0121-43b4-9af7-38ddb0a42e04
classTag::GET /users/0a6a0267-29aa-43ac-8e15-b14f2278ff6d:e00ef304-0121-43b4-9af7-38ddb0a42e04

That makes me think that maybe the earlier error was wrong? That while it was surfacing as "server not found" maybe the underlying cause was something else? Like maybe since the other keys were persisted in that error, there was an issue persisting that specific key?

Hmm.. I wonder how to debug. Gonna think on it 💭.

arigoldx commented 3 years ago

Fwiw, I left this overnight and retried and had the saaame issue:

[error] p.a.cache.redis - Command GET for key 'classTag::eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.WVNcMzr3yyQpp7wLCNc82E_59p6HqtCB3kCvMd4-ZMAxiIJKJuSrmTdzDtPPiD76vaRl1VmqMzfk6uQxp_1hIF9zfLwbzKJrazhMlb_8JsMN7kJnM5_GH7D8UiR_euPFyPkf0JLB9Y6DaSs75etqZuPowFHw_pxv5VbuZz6Xi_8.r4fTZfB_I3dkhnq4.Z0fybeKZYG8mLszdC8cVlCqgFr5QSMrpLXjfO08KeUvjlSEmxYu8KbvGriTZAjHWSi0S_Rrq6laEPCmz5qyCDDPDUZJwX5W7m0ot6onKLdcdTWaykMFUSHrKnI7kHAlYHwjrML8WGJeYhvr7cyyOSyHTiHcvuC-WmJLYWUI9GLvluRy7RD5e7zIUewp8b1bfGKemQtLzoYKqEJoHWFtbMp9wVCg0L5479Emm_7WYFzguPiDBjQcIKH9Da7CR_Lbjw-jM6V5pblCl7Cp7xgyd.3IgzjxjsaC6A9AvI8_xAwQ' failed.
java.lang.RuntimeException: server not found: no server available

Did the full redis-cluster restart (as described above) and things went back to working. HMM. :thinking:

KarelCemus commented 3 years ago

Difficult to help here since you made lots for customizations here. It seems like there might be two possible issues. First, the connection might be unstable, second, the classtag is not set or expires too soon. To debug the second, try this:

  1. wipe redis db
  2. put some value
  3. check there exists both the key and the key with a classtag prefix
  4. check their TTL

btw you logs show that you attached just node 7000 7001 and 7002 but your redis logs show much more. What's your current play-redis config?

KarelCemus commented 3 years ago

Closing the issue since you said it is already solved. For future reference, it was probably caused by an invalid config - mix of standalone and cluster config and instance.

If the issue persists, feel free to reopen and post additional details