search

muoncore / newton

Extend Muon into the world of Spring. Implement Muon apis, especially around events to provide DDD patterns natively in Spring Boot
http://muoncore.io
GNU Lesser General Public License v3.0
12 stars 5 forks source link

Restarting photon causes any further event processing to fail on clients #41

Closed bobjana closed 7 years ago

bobjana commented 7 years ago 
2017-06-11 04:03:17.599 DEBUG 89713 --- [o-auto-1-exec-8] .w.s.m.s.DefaultHandlerExceptionResolver : Resolving exception from handler [public org.springframework.http.ResponseEntity<?> mu.cibecs.security.customer.CustomerController.create(mu.cibecs.security.customer.CustomerController$CreateCustomerRequest) throws java.util.concurrent.ExecutionException,java.lang.InterruptedException]: io.muoncore.exception.MuonException: Failed to persist domain event CustomerCreatedEvent(id=593ca4e439e0815e7f6137f7, domain=qui, name=Gibson, Gibson and Gibson):**A timeout occurred, the remote service did not send a response**
2017-06-11 04:03:17.600 ERROR 89713 --- [o-auto-1-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is io.muoncore.exception.MuonException: Failed to persist domain event CustomerCreatedEvent(id=593ca4e439e0815e7f6137f7, domain=qui, name=Gibson, Gibson and Gibson):A timeout occurred, the remote service did not send a response] with root cause

io.muoncore.exception.MuonException: Failed to persist domain event CustomerCreatedEvent(id=593ca4e439e0815e7f6137f7, domain=qui, name=Gibson, Gibson and Gibson):A timeout occurred, the remote service did not send a response
    at io.muoncore.protocol.event.client.AggregateEventClient.lambda$publishDomainEvents$0(AggregateEventClient.java:45)

Only was client app restarts it is able to process events

daviddawson commented 7 years ago

Partially addressed in https://github.com/muoncore/muon-amqp/issues/1

daviddawson commented 7 years ago

Caused by https://github.com/muoncore/muon-java/issues/60

bobjana commented 7 years ago

Causes different exceptions to occur now:

2017-06-22 13:14:50.594  INFO 17227 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_SECURITY-SERVICE/10.0.0.4:security-service:0: registering service...
2017-06-22 13:14:50.596  INFO 17227 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_SECURITY-SERVICE/10.0.0.4:security-service:0 - registration status: 204
2017-06-22 13:14:50.599  INFO 17227 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_SECURITY-SERVICE/10.0.0.4:security-service:0 - registration status: 204
2017-06-22 13:20:19.515  INFO 17227 --- [      channel-4] i.m.t.s.client.SharedSocketRoute         : Shutting down shared-route due to channel failure
2017-06-22 13:20:19.518  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : CustomerDomainUniqueService is already unlocked
2017-06-22 13:20:19.518  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : AdministratorViewStore-test/Customer is already unlocked
2017-06-22 13:20:19.519  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : AdministratorViewStore-security-service/Customer is already unlocked
2017-06-22 13:20:19.519  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : UserUsernameUniqueService is already unlocked
2017-06-22 13:20:19.519  WARN 17227 --- [      channel-4] nClusterAwareTrackingSubscriptionManager : NewtonEvent subscription has ended, will attempt to reconnect in 5000ms
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : saga-manager-security-service/Customer is already unlocked
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : UserViewStore-security-service/User is already unlocked
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : saga-manager-security-service/User is already unlocked
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : UserViewStore-test/User is already unlocked
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : UserViewStore-test-service/User is already unlocked
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : AdministratorViewStore-security-service/User is already unlocked
2017-06-22 13:20:24.526  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : CustomerViewStore-security-service/Customer is already unlocked
2017-06-22 13:20:24.526  WARN 17227 --- [      channel-4] nClusterAwareTrackingSubscriptionManager : NewtonEvent subscription has ended, will attempt to reconnect in 5000ms
2017-06-22 13:20:29.531  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : CustomerNameUniqueService is already unlocked
2017-06-22 13:20:29.531  INFO 17227 --- [      channel-4] i.m.newton.cluster.LocalOnlyLockService  : AdministratorViewStore-test/User is already unlocked
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
    at io.muoncore.codec.json.JsonOnlyCodecs.decode(JsonOnlyCodecs.java:38)
    at io.muoncore.protocol.event.client.EventClientProtocol.lambda$new$0(EventClientProtocol.java:56)
    at io.muoncore.channel.impl.TimeoutChannel$2.lambda$send$0(TimeoutChannel.java:73)
    at io.muoncore.transport.client.RingBufferLocalDispatcher.route(RingBufferLocalDispatcher.java:404)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$SingleThreadTask.run(RingBufferLocalDispatcher.java:297)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:171)
    at io.muoncore.transport.client.RingBufferLocalDispatcher$3.onEvent(RingBufferLocalDispatcher.java:168)
    at reactor.jarjar.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
daviddawson commented 7 years ago

could you specify the versions you are using of photonlite and newton?

bobjana commented 7 years ago

photon-lite: 0.0.5 newton: 0.0.10-SNAPSHOT

daviddawson commented 7 years ago

Caused by incorrect handling of channel failure in the event protocol.

Fixed in https://github.com/muoncore/stack-event/commit/c61b939a6b276b9a8da374fa71ebfdd2db668467

bobjana commented 7 years ago

Resolved