Making multiple pods - Githubissues

burakakca commented 1 year ago

Hi,

I'm running a complex example. I didn't understand how to make multiple pods in this example.

Redis server

ghostdogpr commented 1 year ago

Just start multiple processes (note that each process needs to use a different port)

burakakca commented 1 year ago

Like this?

GuildApp

  val config: ZLayer[Any, SecurityException, Config] =
    ZLayer(
      System
        .env("port")
        .map(_.flatMap(_.toIntOption).fold(Config.default)(port => Config.default.copy(shardingPort = 5000)))
    )

GuildApp2

object MyConfig {
  val default: Config = Config(
    numberOfShards = 300,
    selfHost = "localhost",
    shardingPort = 64321,
    shardManagerUri = uri"http://localhost:8080/api/graphql",
    serverVersion = "1.0.0",
    entityMaxIdleTime = 1 minute,
    entityTerminationTimeout = 3 seconds,
    sendTimeout = 10 seconds,
    refreshAssignmentsRetryInterval = 5 seconds,
    unhealthyPodReportInterval = 5 seconds,
    simulateRemotePods = false
  )
}

  val config: ZLayer[Any, SecurityException, Config] =
    ZLayer(
      System
        .env("port")
        .map(_.flatMap(_.toIntOption).fold(MyConfig.default)(port => MyConfig.default.copy(shardingPort = 5001)))
    )

ghostdogpr commented 1 year ago

The code in GuildApp will read from environment variable if there is one named "port", so if you set the environment variable to a different value, you can run GuildApp multiple times.

burakakca commented 1 year ago

Config.default.copy(shardingPort = 5000

This part doesn't affect just I added it there some port for run app

burakakca commented 1 year ago

@ghostdogpr

GuildApp2

Or I used it like this but Error Same as below

ZLayer.succeed(MyConfig.default),

Task :app:GuildAppTwo.main()
timestamp=2022-12-11T08:32:10.296443494Z level=INFO thread=#zio-fiber-41 message="Registered entity guild1" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
<FAIL> Fail(java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field protected transient int java.util.AbstractList.modCount accessible: module java.base does not "opens java.util" to unnamed module @1572270e,Stack trace for thread "zio-fiber-47":
    at com.twitter.chill.ResourcePool.borrow(ResourcePool.java:44)
    at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:114)
    at com.devsisters.shardcake.KryoSerialization.live.$anon.encode(KryoSerialization.scala:20)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:235)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:267)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:272)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:251)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252))
timestamp=2022-12-11T08:32:10.608219314Z level=INFO thread=#zio-fiber-4 message="Releasing PubSub connection: redis://localhost" location=shardcake.gradle.simple.RedisConfig.redis.logger.$anon.info file=RedisConfig.scala line=19
timestamp=2022-12-11T08:32:10.609762683Z level=INFO thread=#zio-fiber-4 message="Releasing PubSub connection: redis://localhost" location=shardcake.gradle.simple.RedisConfig.redis.logger.$anon.info file=RedisConfig.scala line=19
timestamp=2022-12-11T08:32:10.610279628Z level=INFO thread=#zio-fiber-4 message="Releasing Commands connection: redis://localhost" location=shardcake.gradle.simple.RedisConfig.redis.logger.$anon.info file=RedisConfig.scala line=19
timestamp=2022-12-11T08:32:10.611635447Z level=INFO thread=#zio-fiber-4 message="Releasing Redis connection: RedisURI(redis://localhost)" location=shardcake.gradle.simple.RedisConfig.redis.logger.$anon.info file=RedisConfig.scala line=19
timestamp=2022-12-11T08:32:10.626327882Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-47" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field protected transient int java.util.AbstractList.modCount accessible: module java.base does not "opens java.util" to unnamed module @1572270e
    at com.twitter.chill.ResourcePool.borrow(ResourcePool.java:44)
    at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:114)
    at com.devsisters.shardcake.KryoSerialization.live.$anon.encode(KryoSerialization.scala:20)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:235)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:267)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:272)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:251)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252)"

> Task :app:GuildAppTwo.main() FAILED

burakakca commented 1 year ago

When I was want to run a second GuildApp I got that error.

ghostdogpr commented 1 year ago

Hmm, weird. I just tried it, on 2.x branch and without modifying any code:

have Redis up and running
run complex/ShardManagerApp
run complex/GuildApp with port=2552 env variable
run complex/GuildApp with port=2553 env variable

And it worked as expected.

burakakca commented 1 year ago

Steps.

git clone -b series/2.x https://github.com/devsisters/shardcake.git
sbt builds the project and I got the below error; I pass that error with closing grpcProtocol codes in sbt.build and added "shardcake-protocol-grpc" package

shardcake/protocol-grpc/src/main/scala/com/devsisters/shardcake/GrpcPods.scala:6:33
object protobuf is not a member of package com.devsisters.shardcake
import com.devsisters.shardcake.protobuf.sharding._

changed redis client config with for me ""redis://localhost""
Added two run configurations for GuildApp port=2552 and port=2553
Start ShardManagerApp
Start GuildApp port=2552 -> OK
Start GuildApp port=2553 -> I got below error again

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
timestamp=2022-12-15T07:03:45.799446446Z level=INFO thread=#zio-fiber-43 message="Registered entity guild" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
<FAIL> Fail(java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field protected transient int java.util.AbstractList.modCount accessible: module java.base does not "opens java.util" to unnamed module @3e9d855d,Stack trace for thread "zio-fiber-49":
    at com.twitter.chill.ResourcePool.borrow(ResourcePool.java:44)
    at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:114)
    at com.devsisters.shardcake.KryoSerialization.live.$anon.encode(KryoSerialization.scala:20)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:213)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:265)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:262)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:248)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252))
timestamp=2022-12-15T07:03:46.119676694Z level=INFO thread=#zio-fiber-6 message="Releasing PubSub connection: redis://localhost" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-15T07:03:46.120940077Z level=INFO thread=#zio-fiber-6 message="Releasing PubSub connection: redis://localhost" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-15T07:03:46.121340384Z level=INFO thread=#zio-fiber-6 message="Releasing Commands connection: redis://localhost" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-15T07:03:46.122813094Z level=INFO thread=#zio-fiber-6 message="Releasing Redis connection: RedisURI(redis://localhost)" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
timestamp=2022-12-15T07:03:46.135017843Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-49" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field protected transient int java.util.AbstractList.modCount accessible: module java.base does not "opens java.util" to unnamed module @3e9d855d
    at com.twitter.chill.ResourcePool.borrow(ResourcePool.java:44)
    at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:114)
    at com.devsisters.shardcake.KryoSerialization.live.$anon.encode(KryoSerialization.scala:20)
    at com.devsisters.shardcake.Sharding.sendToPod(Sharding.scala:213)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:265)
    at com.devsisters.shardcake.Sharding.messenger.$anon.sendMessage.trySend(Sharding.scala:262)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:248)
    at com.devsisters.shardcake.Sharding.messenger.$anon.send(Sharding.scala:252)"

Process finished with exit code 1

ghostdogpr commented 1 year ago

Maybe JDK version? I was with Java 11.

burakakca commented 1 year ago

Yeah, It works but with few warnings. I was using Java 19

SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
timestamp=2022-12-15T07:29:00.835670Z level=INFO thread=#zio-fiber-43 message="Registered entity guild" location=com.devsisters.shardcake.Sharding.live file=Sharding.scala line=422
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/burak/.cache/coursier/v1/https/repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
timestamp=2022-12-15T07:29:01.270086Z level=INFO thread=#zio-fiber-50 message="Creating listener for channel: RedisChannel(shard_assignments)" location=example.complex.package.redis.logger.$anon.info file=package.scala line=19
Success(Set(03d9238e-456d-4189-9abc-d914c45e0a5e, 1cb4e981-ac3c-4b99-8395-076383614b33, f2c0793b-5a85-4294-9226-487fb38d872d, 40eeeb83-4a2a-48e9-8614-dd4b098f1909))
Success(HashSet(03d9238e-456d-4189-9abc-d914c45e0a5e, f2c0793b-5a85-4294-9226-487fb38d872d, 40eeeb83-4a2a-48e9-8614-dd4b098f1909, 1cb4e981-ac3c-4b99-8395-076383614b33, c273f1fb-fe34-44cc-917d-d0e6daa0a2f5))
Failure(java.lang.Exception: Guild is already full!)

ghostdogpr commented 1 year ago

Yeah the warnings are okay, I got them too. Probably Kryo is not fully compatible with Java 19 yet.

burakakca commented 1 year ago

Quick questions

I run two same applications and there are two pods. Where is my GuildMessage entity now?
Should I need to run a pod server to access the entities

ghostdogpr commented 1 year ago

I run two same applications and there are two pods. Where is my GuildMessage entity now?

The entity will be on 1 of the 2 pods (which one is decided by the shard manager). Shardcake ensures that the same entity is never in more than 1 pod at any given time.

Should I need to run a pod server to access the entities

You can communicate with entities from any pod: the same pods that are hosting entities but also any other pods.

devsisters / shardcake

Making multiple pods #46