search

SignalK / signalk-server-java

DEPRECATED - see https://github.com/SignalK/signalk-java
Apache License 2.0
6 stars 9 forks source link

repeatably error and server crash #9

Closed manuprendlair closed 8 years ago

manuprendlair commented 9 years ago

Hello,

I experiment some pb with the server. I am not able to use it more than 1 hours without crash. My config:

Firstly when I start to sen nmea string to UDP, I have this error who come repeatedly:

[) thread #4 - seda://commonOut] OutputFilterProcessor DEBUG Outputting:{breadcrumbId=ID-blueocean-nav-51270-1430049210217-0-15170, OUTPUT_TYPE=OUTPUT_NMEA, websocket.connectionKey=78c75864-af2b-43e1-ae1b-1b11a873c768} [) thread #4 - seda://commonOut] OutputFilterProcessor DEBUG Outputting:Message: {"context":"vessels.self","updates":[]} [ nioEventLoopGroup-4-2] SubscriptionManager DEBUG Adding 9f4a9c12-b632-460c-8ffb-1d4dd502c782/9f4a9c12-b632-460c-8ffb-1d4dd502c782, outputType=OUTPUT_NMEA [ thread #5 - timer://heartbeat] HeartbeatProcessor ERROR Exception occurred during execution on the exchange: Exchange[Message: {"context":"vessels.self","updates":[]}] org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: {"context":"vessels.self","updates":[]}] at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1572) at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:623) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:471) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:467) at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:245) at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:379) at nz.co.fortytwo.signalk.processor.HeartbeatProcessor.process(HeartbeatProcessor.java:81) at org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:165) at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:73) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.lang.IllegalStateException: Queue full at java.util.AbstractQueue.add(AbstractQueue.java:98) at org.apache.camel.component.seda.SedaProducer.addToQueue(SedaProducer.java:230) at org.apache.camel.component.seda.SedaProducer.process(SedaProducer.java:167) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:68) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:375) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:343) at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:233) at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:343) at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184) at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124) at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:233) ... 11 more [ thread #5 - timer://heartbeat] all ERROR Exchange[ExchangePattern: InOnly, BodyType: null, Body: [Body is null]]

Then after sometime (30 min to 1h...), the server stop responding with this error:

org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: {vessels.kiaOra.navigation.position.latitude=44.053721, vessels.kiaOra.navigation.position.longitude=4.103768, vessels.kiaOra.navigation.position.source=vessels.kiaOra.sources.nmea.0183.RMC, vessels.kiaOra.navigation.position.timestamp=2015-04-27T08:21:20.294Z, vessels.kiaOra.navigation.speedOverGround.source=vessels.kiaOra.sources.nmea.0183.RMC, vessels.kiaOra.navigation.speedOverGround.timestamp=2015-04-27T08:21:20.294Z, vessels.kiaOra.navigation.speedOverGround.value=4.629959999999999, vessels.kiaOra.sources.nmea.0183.RMC.source=unknown, vessels.kiaOra.sources.nmea.0183.RMC.timestamp=2015-04-27T08:21:20.294Z, vessels.kiaOra.sources.nmea.0183.RMC.value=$GPRMC,082113.77,A,4403.21076,N,00406.22608,E,0009.0,180.0,270415,0.0,W,A_16}] at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1572) at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:308) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:144) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) at org.apache.camel.processor.FatalFallbackErrorHandler.process(FatalFallbackErrorHandler.java:49) at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:896) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:387) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:291) at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:200) at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:147) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread "Thread-14" java.lang.OutOfMemoryError: Java heap space [) thread #0 - seda://inputData] SedaConsumer WARN Error processing exchange. Exchange[Message: $WIMWV,073.7,R,013.0,N,A_22 ]. Caused by: [java.lang.OutOfMemoryError - Java heap space] java.lang.OutOfMemoryError: Java heap space Exception in thread "qtp26986565-90" java.lang.OutOfMemoryError: Java heap space [ nioEventLoopGroup-4-2] NioEventLoop WARN Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space

With the help of Roberts I change hawtio.start to false. But I have same error I described.

Emmanuel

rob42 commented 9 years ago

This is probably the message buffer stacking up messages until the memory overflows. Possibly the heartbeat message is continuing to be sent after the client goes away. I'm not seeing this in normal websocket connections, my system runs for 24+ hours without issue.

Will need to test further on TCP/UDP etc but very busy at present

rob42 commented 9 years ago

Hmm, in above error: Caused by: java.lang.IllegalStateException: Queue full Just realised that the NemaStudio is probably sending, but I dont know if it reads from the UDP port. If not then the heartbeat messages will queue up and overflow. Guess i will need to deal with the case where a client sends but doesnt read... Actually that doesnt matter in UDP as its just send/no-ack. Lost messages are normal for UDP.

rob42 commented 8 years ago

Closing as old and not repeatable for me. Pls re-open if you still suffer this problem