search

devsisters / shardcake

Sharding and location transparency for Scala
https://devsisters.github.io/shardcake/
Apache License 2.0
389 stars 30 forks source link

Problem with using multiple pods #52

Closed burakakca closed 1 year ago

burakakca commented 1 year ago

Simple Memory Example

GuildApp

object GuildApp extends ZIOAppDefault {
  val config: ZLayer[Any, SecurityException, Config] =
    ZLayer(
      System
        .env("port")
        .map(_.flatMap(_.toIntOption).fold(Config.default)(port => Config.default.copy(shardingPort = port)))//8001
    )

  val program =
    for {
      _     <- Sharding.registerEntity(Guild, behavior)
      _     <- Sharding.registerScoped
      guild <- Sharding.messenger(Guild)
      _     <- guild.send("guild1")(Join("user1", _)).debug
      _     <- guild.send("guild1")(Join("user2", _)).debug
      _     <- guild.send("guild1")(Join("user3", _)).debug
      _     <- guild.send("guild1")(Join("user4", _)).debug
      _     <- guild.send("guild1")(Join("user5", _)).debug
      _     <- guild.send("guild1")(Join("user6", _)).debug
      _     <- ZIO.never
    } yield ()

  def run: Task[Unit] =
    ZIO
      .scoped(program)
      .provide(
        config,
        ZLayer.succeed(GrpcConfig.default),
        Serialization.javaSerialization,
        Storage.memory,
        ShardManagerClient.liveWithSttp,
        GrpcPods.live,
        Sharding.live,
        GrpcShardingService.live
      )
}

GuildAppPod2

 //... same code 8002
  val program =
    for {
      _     <- Sharding.registerEntity(Guild, behavior)
      _     <- Sharding.registerScoped
      guild <- Sharding.messenger(Guild)
      _     <- guild.send("guild1")(Join("a", _)).debug
      _     <- guild.send("guild1")(Join("b", _)).debug
      _     <- guild.send("guild1")(Join("c", _)).debug
      _     <- guild.send("guild1")(Join("d", _)).debug
      _     <- guild.send("guild1")(Join("e", _)).debug
      _     <- guild.send("guild1")(Join("f", _)).debug
      _     <- ZIO.never
    } yield ()

//... same code

simple.ShardManagerApp

timestamp=2022-12-21T09:03:14.168151Z level=INFO thread=#zio-fiber-7 message="Shard Manager loaded" location=com.devsisters.shardcake.ShardManager.live file=ShardManager.scala line=208
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
timestamp=2022-12-21T09:03:14.749419Z level=INFO thread=#zio-fiber-6 message="Shard Manager server started on port 8080." location=com.devsisters.shardcake.Server.run file=Server.scala line=29
timestamp=2022-12-21T09:03:29.723609Z level=INFO thread=#zio-fiber-50 message="Registering Pod(localhost:8001,1.0.0)" location=com.devsisters.shardcake.ShardManager.register file=ShardManager.scala line=33
timestamp=2022-12-21T09:03:30.188122Z level=INFO thread=#zio-fiber-20 message="ShardsAssigned(localhost:8001,HashSet(69, 138, 101, 249, 234, 88, 170, 115, 5, 269, 202, 217, 276, 120, 247, 10, 56, 142, 153, 174, 185, 42, 24, 288, 37, 25, 257, 52, 14, 184, 110, 125, 196, 157, 189, 20, 46, 93, 284, 152, 228, 289, 57, 78, 261, 29, 216, 164, 179, 106, 238, 121, 84, 211, 253, 147, 280, 61, 221, 293, 132, 1, 265, 74, 206, 89, 133, 116, 243, 292, 248, 270, 220, 102, 233, 6, 60, 117, 85, 201, 260, 160, 192, 165, 33, 28, 38, 297, 70, 275, 21, 137, 92, 229, 252, 197, 65, 97, 285, 224, 156, 9, 188, 53, 169, 141, 77, 193, 212, 96, 109, 256, 124, 225, 173, 13, 129, 41, 134, 73, 2, 266, 205, 128, 237, 105, 244, 298, 166, 148, 264, 45, 161, 17, 149, 32, 34, 279, 64, 180, 296, 176, 191, 22, 44, 286, 291, 59, 118, 281, 204, 259, 27, 71, 12, 54, 144, 49, 236, 181, 86, 159, 187, 172, 113, 219, 274, 81, 230, 76, 7, 245, 39, 98, 271, 208, 103, 140, 213, 91, 66, 155, 198, 108, 240, 251, 130, 278, 223, 135, 299, 267, 167, 35, 226, 3, 241, 80, 162, 255, 209, 112, 123, 194, 145, 48, 63, 295, 18, 282, 150, 95, 263, 50, 67, 199, 16, 127, 31, 177, 182, 154, 11, 72, 175, 143, 43, 99, 87, 203, 218, 104, 250, 231, 40, 26, 258, 158, 186, 171, 139, 23, 55, 114, 8, 75, 207, 272, 82, 290, 119, 58, 235, 246, 214, 287, 151, 300, 36, 146, 30, 51, 190, 273, 168, 262, 183, 19, 210, 107, 268, 79, 195, 94, 283, 239, 242, 4, 294, 126, 136, 15, 68, 62, 178, 277, 131, 47, 163, 200, 122, 83, 215, 222, 232, 100, 90, 111, 254, 227))" location=com.devsisters.shardcake.ShardManager.live file=ShardManager.scala line=207
timestamp=2022-12-21T09:03:43.032295Z level=INFO thread=#zio-fiber-70 message="Registering Pod(localhost:8002,1.0.0)" location=com.devsisters.shardcake.ShardManager.register file=ShardManager.scala line=33
timestamp=2022-12-21T09:03:53.242502Z level=INFO thread=#zio-fiber-75 message="Unregistering localhost:8002" location=com.devsisters.shardcake.ShardManager.unregister file=ShardManager.scala line=56

GuildApp Pod1

timestamp=2022-12-21T09:03:29.249530Z level=INFO thread=#zio-fiber-29 message="Registered entity guild" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
Success(Set(user1pod1))
Success(Set(user1pod1, user2pod1))
Success(Set(user1pod1, user2pod1, user3pod1))
Success(Set(user1pod1, user2pod1, user3pod1, user4pod1))
Success(HashSet(user5pod1, user3pod1, user2pod1, user4pod1, user1pod1))
Success(HashSet(user5pod1, user3pod1, user2pod1, user4pod1, user6pod1, user1pod1))

GuildAppPod2

timestamp=2022-12-21T09:03:43.008882Z level=INFO thread=#zio-fiber-29 message="Registered entity guild" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
<FAIL> Fail(com.devsisters.shardcake.errors.SendTimeoutException: Timeout sending message to guild guild1 - Join(a,Replier(5d0f1408-9361-40b9-8a6e-844c0b903a19)),Stack trace for thread "zio-fiber-6":
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252)
    at example.simple.GuildAppPod2.program(GuildAppPod2.scala:22)
    at example.simple.GuildAppPod2.run(GuildAppPod2.scala:33)
    at example.simple.GuildAppPod2.run(GuildAppPod2.scala:34))
timestamp=2022-12-21T09:03:53.289079Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-6" com.devsisters.shardcake.errors.SendTimeoutException: com.devsisters.shardcake.errors.SendTimeoutException: Timeout sending message to guild guild1 - Join(a,Replier(5d0f1408-9361-40b9-8a6e-844c0b903a19))
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252)
    at example.simple.GuildAppPod2.program(GuildAppPod2.scala:22)
    at example.simple.GuildAppPod2.run(GuildAppPod2.scala:33)
    at example.simple.GuildAppPod2.run(GuildAppPod2.scala:34)"

Process finished with exit code 1

Expected Result

I think it should work with two pods on the memory state

Complex redis examples

GuildBehavior

  object Guild extends EntityType[GuildMessage]("guild")
  object Lonca extends EntityType[GuildMessage]("lonca")

GuildApp Pod1

import com.devsisters.shardcake._
import com.devsisters.shardcake.interfaces.Serialization
import dev.profunktor.redis4cats.RedisCommands
import example.complex.GuildBehavior.GuildMessage.{ Join, Terminate }
import example.complex.GuildBehavior._
import zio._

object GuildApp extends ZIOAppDefault {
  val config: ZLayer[Any, SecurityException, Config] =
    ZLayer(
      System
        .env("port")
        .map(_.flatMap(_.toIntOption).fold(Config.default)(port => Config.default.copy(shardingPort = port)))//2000
    )

  val program: ZIO[Sharding with Scope with Serialization with RedisCommands[Task, String, String], Throwable, Unit] =
    for {
      _     <- Sharding.registerEntity(Guild, behavior, p => Some(Terminate(p)))
      _     <- Sharding.registerScoped
      guild <- Sharding.messenger(Guild)
      user1 <- Random.nextUUID.map(_.toString)
      user2 <- Random.nextUUID.map(_.toString)
      user3 <- Random.nextUUID.map(_.toString)
      _     <- guild.send("guild1")(Join(user1, _)).debug
      _     <- guild.send("guild1")(Join(user2, _)).debug
      _     <- guild.send("guild1")(Join(user3, _)).debug
      _     <- ZIO.never
    } yield ()

  def run: Task[Unit] =
    ZIO
      .scoped(program)
      .provide(
        config,
        ZLayer.succeed(GrpcConfig.default),
        ZLayer.succeed(RedisConfig.default),
        redis,
        StorageRedis.live,
        KryoSerialization.live,
        ShardManagerClient.liveWithSttp,
        GrpcPods.live,
        Sharding.live,
        GrpcShardingService.live
      )
}

GuildAppTwo Pod2

//... same code  port 3000
  val program: ZIO[Sharding with Scope with Serialization with RedisCommands[Task, String, String], Throwable, Unit] =
    for {
      _     <- Sharding.registerEntity(Lonca, behavior, p => Some(Terminate(p)))
      _     <- Sharding.registerScoped
      lonca <- Sharding.messenger(Lonca)
      user1 <- Random.nextUUID.map(_.toString)
      user2 <- Random.nextUUID.map(_.toString)
      user3 <- Random.nextUUID.map(_.toString)
      _     <- lonca.send("lonca1")(Join(user1, _)).debug
      _     <- lonca.send("lonca2")(Join(user2, _)).debug
      _     <- lonca.send("lonca3")(Join(user3, _)).debug
      _     <- ZIO.never
    } yield ()
//... same code

complex.SharManagerApp

timestamp=2022-12-21T09:33:39.419819Z level=INFO thread=#zio-fiber-7 message="Shard Manager loaded" location=com.devsisters.shardcake.ShardManager.live file=ShardManager.scala line=208
timestamp=2022-12-21T09:33:39.671944Z level=INFO thread=#zio-fiber-6 message="Shard Manager server started on port 8080." location=com.devsisters.shardcake.Server.run file=Server.scala line=29
timestamp=2022-12-21T09:35:47.164573Z level=INFO thread=#zio-fiber-197 message="Registering Pod(localhost:2000,1.0.0)" location=com.devsisters.shardcake.ShardManager.register file=ShardManager.scala line=33
timestamp=2022-12-21T09:35:47.634762Z level=INFO thread=#zio-fiber-32 message="ShardsAssigned(localhost:2000,HashSet(69, 138, 101, 249, 234, 88, 170, 115, 5, 269, 202, 217, 276, 120, 247, 10, 56, 142, 153, 174, 185, 42, 24, 288, 37, 25, 257, 52, 14, 184, 110, 125, 196, 157, 189, 20, 46, 93, 284, 152, 228, 289, 57, 78, 261, 29, 216, 164, 179, 106, 238, 121, 84, 211, 253, 147, 280, 61, 221, 293, 132, 1, 265, 74, 206, 89, 133, 116, 243, 292, 248, 270, 220, 102, 233, 6, 60, 117, 85, 201, 260, 160, 192, 165, 33, 28, 38, 297, 70, 275, 21, 137, 92, 229, 252, 197, 65, 97, 285, 224, 156, 9, 188, 53, 169, 141, 77, 193, 212, 96, 109, 256, 124, 225, 173, 13, 129, 41, 134, 73, 2, 266, 205, 128, 237, 105, 244, 298, 166, 148, 264, 45, 161, 17, 149, 32, 34, 279, 64, 180, 296, 176, 191, 22, 44, 286, 291, 59, 118, 281, 204, 259, 27, 71, 12, 54, 144, 49, 236, 181, 86, 159, 187, 172, 113, 219, 274, 81, 230, 76, 7, 245, 39, 98, 271, 208, 103, 140, 213, 91, 66, 155, 198, 108, 240, 251, 130, 278, 223, 135, 299, 267, 167, 35, 226, 3, 241, 80, 162, 255, 209, 112, 123, 194, 145, 48, 63, 295, 18, 282, 150, 95, 263, 50, 67, 199, 16, 127, 31, 177, 182, 154, 11, 72, 175, 143, 43, 99, 87, 203, 218, 104, 250, 231, 40, 26, 258, 158, 186, 171, 139, 23, 55, 114, 8, 75, 207, 272, 82, 290, 119, 58, 235, 246, 214, 287, 151, 300, 36, 146, 30, 51, 190, 273, 168, 262, 183, 19, 210, 107, 268, 79, 195, 94, 283, 239, 242, 4, 294, 126, 136, 15, 68, 62, 178, 277, 131, 47, 163, 200, 122, 83, 215, 222, 232, 100, 90, 111, 254, 227))" location=com.devsisters.shardcake.ShardManager.live file=ShardManager.scala line=207
timestamp=2022-12-21T09:35:48.172872Z level=INFO thread=#zio-fiber-218 message="Creating listener for channel: RedisChannel(shard_assignments)" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-21T09:36:08.975143Z level=INFO thread=#zio-fiber-242 message="Registering Pod(localhost:3000,1.0.0)" location=com.devsisters.shardcake.ShardManager.register file=ShardManager.scala line=33
timestamp=2022-12-21T09:36:09.412721Z level=INFO thread=#zio-fiber-251 message="Unregistering localhost:3000" location=com.devsisters.shardcake.ShardManager.unregister file=ShardManager.scala line=56

GuildApp Pod1

timestamp=2022-12-21T09:35:46.832875Z level=INFO thread=#zio-fiber-43 message="Registered entity guild" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
timestamp=2022-12-21T09:35:47.682447Z level=INFO thread=#zio-fiber-68 message="Started entity guild1" location=example.complex.GuildBehavior.behavior file=GuildBehavior.scala line=26
Success(Set(1d3bb131-4601-417d-a487-0f5a308b76f6))
Success(Set(1d3bb131-4601-417d-a487-0f5a308b76f6pod1, e4071203-58bd-47c4-9609-6a0b34288ceb))
Success(Set(e4071203-58bd-47c4-9609-6a0b34288cebpod1, 1d3bb131-4601-417d-a487-0f5a308b76f6pod1, 7b22920c-34a2-4197-9474-73312e151002))
timestamp=2022-12-21T09:35:47.911356Z level=INFO thread=#zio-fiber-46 message="Creating listener for channel: RedisChannel(shard_assignments)" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19

GuildAppTwo Pod2

timestamp=2022-12-21T09:36:08.960621Z level=INFO thread=#zio-fiber-43 message="Registered entity lonca" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/burak/.cache/coursier/v1/https/repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
<FAIL> Fail(io.grpc.StatusException: INTERNAL: Entity type lonca was not registered.,Stack trace for thread "zio-fiber-48":
    at com.devsisters.shardcake.GrpcPods.sendMessage(GrpcPods.scala:76)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:216)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:234)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:265)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:262)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:248)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252))
timestamp=2022-12-21T09:36:09.371320Z level=INFO thread=#zio-fiber-50 message="Creating listener for channel: RedisChannel(shard_assignments)" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-21T09:36:09.543638Z level=INFO thread=#zio-fiber-6 message="Releasing PubSub connection: redis://localhost" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-21T09:36:09.544986Z level=INFO thread=#zio-fiber-6 message="Releasing PubSub connection: redis://localhost" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-21T09:36:09.545296Z level=INFO thread=#zio-fiber-6 message="Releasing Commands connection: redis://localhost" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-21T09:36:09.546868Z level=INFO thread=#zio-fiber-6 message="Releasing Redis connection: RedisURI(redis://localhost)" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-21T09:36:09.568137Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-48" io.grpc.StatusException: io.grpc.StatusException: INTERNAL: Entity type lonca was not registered.
    at com.devsisters.shardcake.GrpcPods.sendMessage(GrpcPods.scala:76)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:216)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:234)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:265)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:262)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:248)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252)"

Process finished with exit code 1

Expected Result

I can add two entities to the same pod but when I want to add them to two different pods I get an error.

Question

In the Redis example, when I send a message to the entity, does it get the data from the cache / memory or does it pull the data from redis?

So, is the need for redis necessary for the pods to be able to map to each other, or is it also necessary for adding and accessing records?

ghostdogpr commented 1 year ago

The "simple" example doesn't work with multiple pods. Redis is indeed used not only for storing assignments, but also to propagate assignment changes to other pods. When using in-memory storage, the changes are not propagated.

burakakca commented 1 year ago

When I add an entity record from one or both pods ; Pod1 and Pod2

      _     <- guild.send("guild1")(Join(user1, _)).debug
      _     <- guild.send("guild1")(Join(user2, _)).debug
      _     <- guild.send("guild1")(Join(user3, _)).debug

Is this record added to the Redis and same-time cache or not?

Or let's say I have a Get function and this function returns a user, in this point, Is user returns from the cache or Redis.

Also can you help me that complex example section error above?

Complex redis examples

ghostdogpr commented 1 year ago

In the simple example, both the entity data and the shard assignments are stored in memory. In the complex example, both the entity data and the shard assignments are stored in Redis.

I think the error you have is because of using JDK 19? Try with JDK 11?

burakakca commented 1 year ago

I think the error you have is because of using JDK 19? Try with JDK 11?

JDK 11

ghostdogpr commented 1 year ago

Ah my bad, I didn't see the right error: Entity type lonca was not registered.

All the pods that call registerScoped need to call registerEntity for all the entity types.

burakakca commented 1 year ago

Yes, It works now.

For example, I have 1000 entities, How much do I pod need? If I have 2 pods or more, propagate is gonna be fifty-fifty to the pods?

ghostdogpr commented 1 year ago

Yes, entities will be balanced across all pods. With 2 pods, they will have the same amount of shards, which means similar amount of entities.

How many pods you need depends on many things like pod spec (CPU, memory), how much resource your behavior uses, how many messages, etc.

burakakca commented 1 year ago

Okey I understand. Thanks for your help.

burakakca commented 1 year ago

how can we develop an example that stores “entity data in memory” with multi-pod configuration. shard assignments data could be stored in redis or memory - it doesn't matter.

espacially want to store entity data in memory with multi-pod config

ghostdogpr commented 1 year ago

If you store entity data in memory, when the entity is moved to another pod its state will be reset. But for the sake of the example you can totally do that. But the shard manager needs redis storage if you want to use multiple pods.