Server requires too much memory

wwerner commented 5 years ago

After the recent updates to vlingo-wire (I suspect somewhere around https://github.com/vlingo/vlingo-wire/pull/13 and https://github.com/vlingo/vlingo-wire/compare/12f372d49439...10c9ded38288 ), the server consumes roughly 4GB of memory. Trying to start the server w/ less memory leads to failures like in the exception below. After giving the server 3.8G Xmx, startup works as expected.

This occured first in CircleCI b/c the server is killed by the system for consuming too much memory. After setting -Xmx to sth. below 3800m, the following exceptions are thrown.

java.lang.IllegalArgumentException: Actor instantiation failed because: ActorFactory failed actor creation for: Address[id=9223372036854775803, name=(none)]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:659)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:143)
    at io.vlingo.http.resource.Server.startWith(Server.java:111)
    at io.vlingo.http.resource.Server.startWith(Server.java:81)
    at io.vlingo.http.resource.Server.startWith(Server.java:60)
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68)
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87)
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96)
Caused by: java.lang.InstantiationException: ActorFactory failed actor creation for: Address[id=9223372036854775803, name=(none)]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:139)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    ... 8 more
Exception in thread "main" java.lang.NullPointerException
    at io.vlingo.actors.Stage.actorFor(Stage.java:152)
    at io.vlingo.http.resource.Server.startWith(Server.java:111)
    at io.vlingo.http.resource.Server.startWith(Server.java:81)
    at io.vlingo.http.resource.Server.startWith(Server.java:60)
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68)
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87)
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96)

VaughnVernon commented 5 years ago

@wwerner This looks exactly like the problem resolved a few days ago. Please do the following:

- rm ~/.m2/repository/io/vlingo/

$ cd {vlingo_root}/vlingo-symbio-jdbc
$ ./pgbounce.sh
$ cd ../vlingo-lattice-exchange-rabbitmq
$ ./rmqbounce.sh 
$ cd ../vlingo-platform
$ ./gradlew
$ mvn clean install

Then run the server. Does that fix the problem?

wwerner commented 5 years ago

No, I was already running against the most current master of all dependencies.

Looking at the heap, practically all memory is used in 2 ConsumerByteBufferPoolss à 1.6G

Looking at the code, this totally makes sense. In https://github.com/vlingo/vlingo-schemata/blob/master/src/main/java/io/vlingo/schemata/Bootstrap.java#L71, we start the server w/ a 16M max message size w/ 2 dispatchers & the default maxBufferPool of 100. => 3.2G

Thanks @VaughnVernon for the pointers.

kmruiz commented 5 years ago

Probably the ConsumerByteBufferPool needs to be lazy-loaded. Why are we binding the max message size with the pool? Is it just simpler?

I would say to reduce the size of the pooled ByteBuffer to 16K (for example, can be configured) and create on-demand temporal ByteBuffers only if the expected message size is bigger.

Probably this issue needs to be tackled in vlingo-http.

WDYT @aleixmorgadas , @VaughnVernon ?

VaughnVernon commented 5 years ago

We should support custom buffers for special case sizes.

vlingo / xoom-schemata

Server requires too much memory #81