search

cloudant-labs / clouseau

Expose Lucene features as an erlang-like node
Apache License 2.0
58 stars 32 forks source link

Clouseau stops working #58

Open schneuwlym opened 1 year ago

schneuwlym commented 1 year ago

We observed on multiple nodes that Clouseau suddenly stopped working. The logfile contained a stacktrace and it did no longer respond to a curl request. CouchDB still responded to curl queries.

We are using version 2.21.0

Logfile

2022-09-30 12:00:09,523 clouseau.main [INFO] Clouseau running as clouseau@127.0.0.1
2022-10-06 09:02:46,363 scalang.node.ErlangHandler [INFO] channel disconnected org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2904f200 [id: 0x53447e05, /127.0.0.1:33241 :> /127.0.0.1:34077] DISCONNECTED. peer: 'couchdb@127.0.0.1
2022-10-06 09:02:46,381 scalang.node.ErlangHandler [INFO] channel disconnected org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38575969 [id: 0x12d3b732, /127.0.0.1:45821 :> /127.0.0.1:34077] DISCONNECTED. peer: null
2022-10-06 09:02:46,387 scalang.node.ServerHandshakeHandler [ERROR] Channel closed during handshake
2022-10-06 09:02:46,371 scalang.node.ErlangHandler [ERROR] error caught in erlang handler 'couchdb@127.0.0.1
java.nio.channels.ClosedChannelException
    at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:645)
    at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
    at scalang.node.PacketCounter.writeRequested(PacketCounter.scala:27)
    at org.jboss.netty.handler.execution.ExecutionHandler.handleDownstream(ExecutionHandler.java:167)
    at org.jboss.netty.channel.Channels.write(Channels.java:632)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
    at scalang.node.PacketCounter.writeRequested(PacketCounter.scala:27)
    at org.jboss.netty.channel.Channels.write(Channels.java:632)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
    at scalang.node.PacketCounter.writeRequested(PacketCounter.scala:27)
    at org.jboss.netty.channel.Channels.write(Channels.java:611)
    at org.jboss.netty.channel.Channels.write(Channels.java:578)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
    at scalang.node.FailureDetectionHandler.messageReceived(FailureDetectionHandler.scala:31)
    at scalang.node.PacketCounter.messageReceived(PacketCounter.scala:17)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
    at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:76)
    at scalang.node.PacketCounter.messageReceived(PacketCounter.scala:17)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
    at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
    at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
    at overlock.threadpool.ErrorLoggedThread.run(NamedThreadFactory.scala:40)
juanjopb commented 1 year ago

I had a similar situation yesterday, unfortunately the logs were off but the clouseau stopped working on some nodes.