search

vlingo / xoom-schemata

The VLINGO XOOM Schema Registry.
https://vlingo.io
Mozilla Public License 2.0
17 stars 9 forks source link

[Backend] After updating to current 0.9.1-RC2 snapshot, the server fails to start #79

Closed wwerner closed 5 years ago

wwerner commented 5 years ago

After pulling the changes of all dependencies (updates between 2019-10-30 and 2019-11-08), the server fails to start with the exception below.

The corresponding test io.vlingo.schemata.BootstrapTest#testThatBootstrapStartsServerCleanly also fails.

I checked whether the Definition args match the Actor's c'tor as indicated in the exception, but they do, see screenshot below.

Instead, the exception seems to be thrown during initialization of ServerActor from the Actor super c'tor when trying to create a new TestEnvironment. In io/vlingo/actors/testkit/TestEnvironment.java:17, TestWorld.Instance.get().world().stage() throws an NPE as TestWorld.Instance.get() returns null.

This happens in the unit tests as well as when running the application in dev or prod modes. In the latter cases, is it correct that no ActorFactory.threadLocalEnvironment is found?

java.lang.IllegalArgumentException: Actor instantiation failed because: ActorFactory failed actor creation for: Address[id=101, name=(none)]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:659)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:406)
    at io.vlingo.actors.Actor.childActorFor(Actor.java:181)
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannelActor.startProcessors(ServerRequestResponseChannelActor.java:161)
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannelActor.<init>(ServerRequestResponseChannelActor.java:44)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:143)
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannel.start(ServerRequestResponseChannel.java:54)
    at io.vlingo.http.resource.ServerActor.<init>(ServerActor.java:83)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:143)
    at io.vlingo.http.resource.Server.startWith(Server.java:111)
    at io.vlingo.http.resource.Server.startWith(Server.java:81)
    at io.vlingo.http.resource.Server.startWith(Server.java:60)
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68)
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87)
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96)
Caused by: java.lang.InstantiationException: ActorFactory failed actor creation for: Address[id=101, name=(none)]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:139)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    ... 31 more
java.lang.IllegalArgumentException: Actor instantiation failed because: ActorFactory failed actor creation for: Address[id=9223372036854775802, name=server-request-response-channel]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:659)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:143)
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannel.start(ServerRequestResponseChannel.java:54)
    at io.vlingo.http.resource.ServerActor.<init>(ServerActor.java:83)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:143)
    at io.vlingo.http.resource.Server.startWith(Server.java:111)
    at io.vlingo.http.resource.Server.startWith(Server.java:81)
    at io.vlingo.http.resource.Server.startWith(Server.java:60)
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68)
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87)
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96)
Caused by: java.lang.InstantiationException: ActorFactory failed actor creation for: Address[id=9223372036854775802, name=server-request-response-channel]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:139)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    ... 19 more
java.lang.IllegalArgumentException: Actor instantiation failed because: ActorFactory failed actor creation for: Address[id=9223372036854775803, name=(none)]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:659)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:143)
    at io.vlingo.http.resource.Server.startWith(Server.java:111)
    at io.vlingo.http.resource.Server.startWith(Server.java:81)
    at io.vlingo.http.resource.Server.startWith(Server.java:60)
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68)
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87)
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96)
Caused by: java.lang.InstantiationException: ActorFactory failed actor creation for: Address[id=9223372036854775803, name=(none)]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:139)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    ... 8 more
Exception in thread "main" java.lang.NullPointerException
    at io.vlingo.actors.Stage.actorFor(Stage.java:152)
    at io.vlingo.http.resource.Server.startWith(Server.java:111)
    at io.vlingo.http.resource.Server.startWith(Server.java:81)
    at io.vlingo.http.resource.Server.startWith(Server.java:60)
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68)
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87)
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96)
13:27:13.779 [pool-2-thread-5] ERROR io.vlingo.actors.Logger - ActorFactory: failed actor creation. This is sometimes cause be the constructor parameter types not matching the types in the Definition.parameters(). Often it is caused by a failure in the actor constructor. We have attempted to uncover the root cause here, but that may not be available in some cases.
The root cause may be: java.lang.NullPointerException
See stacktrace for more information. We strongly recommend reviewing your constructor for possible failures in dependencies that it creates.
java.lang.NullPointerException: null
    at io.vlingo.actors.testkit.TestEnvironment.<init>(TestEnvironment.java:17) ~[classes/:na]
    at io.vlingo.actors.Actor.<init>(Actor.java:162) ~[classes/:na]
    at io.vlingo.wire.channel.SocketChannelSelectionProcessorActor.<init>(SocketChannelSelectionProcessorActor.java:49) ~[classes/:na]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_212]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_212]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_212]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_212]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116) ~[classes/:na]
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75) ~[classes/:na]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656) ~[classes/:na]
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448) ~[classes/:na]
    at io.vlingo.actors.Stage.actorFor(Stage.java:406) ~[classes/:na]
    at io.vlingo.actors.Actor.childActorFor(Actor.java:181) ~[classes/:na]
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannelActor.startProcessors(ServerRequestResponseChannelActor.java:161) ~[classes/:na]
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannelActor.<init>(ServerRequestResponseChannelActor.java:44) ~[classes/:na]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_212]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_212]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_212]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_212]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116) ~[classes/:na]
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75) ~[classes/:na]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656) ~[classes/:na]
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448) ~[classes/:na]
    at io.vlingo.actors.Stage.actorFor(Stage.java:143) ~[classes/:na]
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannel.start(ServerRequestResponseChannel.java:54) ~[classes/:na]
    at io.vlingo.http.resource.ServerActor.<init>(ServerActor.java:83) ~[classes/:na]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_212]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_212]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_212]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_212]
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116) ~[classes/:na]
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75) ~[classes/:na]
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656) ~[classes/:na]
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448) ~[classes/:na]
    at io.vlingo.actors.Stage.actorFor(Stage.java:143) ~[classes/:na]
    at io.vlingo.http.resource.Server.startWith(Server.java:111) ~[classes/:na]
    at io.vlingo.http.resource.Server.startWith(Server.java:81) ~[classes/:na]
    at io.vlingo.http.resource.Server.startWith(Server.java:60) ~[classes/:na]
    at io.vlingo.schemata.Bootstrap.<init>(Bootstrap.java:68) ~[classes/:na]
    at io.vlingo.schemata.Bootstrap.instance(Bootstrap.java:87) ~[classes/:na]
    at io.vlingo.schemata.Bootstrap.main(Bootstrap.java:96) ~[classes/:na]

image

wwerner commented 5 years ago

Stepping through the issue several times, the cause seems to be an OOM thrown from ElasticResourcePool.initialize (when creating the third pool w/ each 100 minIdle)

java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at io.vlingo.wire.message.ByteBufferAllocator.allocate(ByteBufferAllocator.java:17)
    at io.vlingo.wire.message.BasicConsumerByteBuffer.<init>(BasicConsumerByteBuffer.java:23)
    at io.vlingo.wire.message.ConsumerByteBufferPool$PoolAwareConsumerByteBuffer.<init>(ConsumerByteBufferPool.java:72)
    at io.vlingo.wire.message.ConsumerByteBufferPool$ConsumerByteBufferFactory.create(ConsumerByteBufferPool.java:49)
    at io.vlingo.wire.message.ConsumerByteBufferPool$ConsumerByteBufferFactory.create(ConsumerByteBufferPool.java:31)
    at io.vlingo.common.pool.ElasticResourcePool.initialize(ElasticResourcePool.java:64)
    at io.vlingo.common.pool.ElasticResourcePool.<init>(ElasticResourcePool.java:58)
    at io.vlingo.common.pool.ElasticResourcePool.<init>(ElasticResourcePool.java:51)
    at io.vlingo.wire.message.ConsumerByteBufferPool.<init>(ConsumerByteBufferPool.java:28)
    at io.vlingo.wire.channel.SocketChannelSelectionProcessorActor.<init>(SocketChannelSelectionProcessorActor.java:55)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
    at io.vlingo.actors.Stage.actorFor(Stage.java:406)
    at io.vlingo.actors.Actor.childActorFor(Actor.java:181)
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannelActor.startProcessors(ServerRequestResponseChannelActor.java:161)
    at io.vlingo.wire.fdx.bidirectional.ServerRequestResponseChannelActor.<init>(ServerRequestResponseChannelActor.java:44)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at io.vlingo.actors.ActorFactory.start(ActorFactory.java:116)
    at io.vlingo.actors.ActorFactory.actorFor(ActorFactory.java:75)
    at io.vlingo.actors.Stage.createRawActor(Stage.java:656)
    at io.vlingo.actors.Stage.actorProtocolFor(Stage.java:448)
wwerner commented 5 years ago

After some debugging w/ @VaughnVernon we found the cause. Recording it here for traceability.

SocketChannelSelectionProcessorActor creates an ElasticResourcePool for each instance, which, at some point throws an OOM which is swallowed around io/vlingo/actors/ActorFactory.java:121 and thus was never visible in stack traces.

Vaughn is currently resolving it, I'll re-test and close this issue afterwards.

wwerner commented 5 years ago

Closed w/ https://github.com/vlingo/vlingo-wire/compare/12f372d49439...10c9ded38288