tronprotocol / tips

TRON Improvement Proposals
229 stars 203 forks source link

issue: fix bug of writing zeromq in multithreads #441

Closed 317787106 closed 1 year ago

317787106 commented 2 years ago

what happened

We start tron blockchain with two sr nodes and one fullnode, I encounter following error:

05:05:57.027 ERROR [Thread-7] [DB](Manager.java:285) unknown throwable happened in process capsule loop
        java.lang.ArrayIndexOutOfBoundsException: 256
        at zmq.pipe.YQueue.backPos(YQueue.java:71)
        at zmq.pipe.YPipe.write(YPipe.java:52)
        at zmq.pipe.Pipe.write(Pipe.java:273)
        at zmq.socket.pubsub.Dist.write(Dist.java:176)
        at zmq.socket.pubsub.Dist.distribute(Dist.java:161)
        at zmq.socket.pubsub.Dist.sendToMatching(Dist.java:137)
        at zmq.socket.pubsub.XPub.xsend(XPub.java:177)
        at zmq.SocketBase.send(SocketBase.java:718)
        at org.zeromq.ZMQ$Socket.send(ZMQ.java:3197)
        at org.zeromq.ZMQ$Socket.send(ZMQ.java:3111)
        at org.tron.common.logsfilter.nativequeue.NativeMessageQueue.publishTrigger(NativeMessageQueue.java:65)
        at org.tron.common.logsfilter.EventPluginLoader.postTransactionTrigger(EventPluginLoader.java:497)
        at org.tron.common.logsfilter.capsule.TransactionLogTriggerCapsule.processTrigger(TransactionLogTriggerCapsule.java:270)
        at org.tron.core.db.Manager.lambda$new$1(Manager.java:279)
        at java.lang.Thread.run(Thread.java:748)
......
05:05:57.031 ERROR [DPosMiner] [consensus](BlockHandleImpl.java:56) Handle block Num:1310,ID:000000000000051ef7401f312704b8e92c3f5ed8dfcce560d7d7e944e09fc467 failed.
java.lang.ArrayIndexOutOfBoundsException: 256
        at zmq.pipe.YQueue.backPos(YQueue.java:71)
        at zmq.pipe.YPipe.write(YPipe.java:52)
        at zmq.pipe.Pipe.write(Pipe.java:273)
        at zmq.socket.pubsub.Dist.write(Dist.java:176)
        at zmq.socket.pubsub.Dist.distribute(Dist.java:161)
        at zmq.socket.pubsub.Dist.sendToMatching(Dist.java:137)
        at zmq.socket.pubsub.XPub.xsend(XPub.java:177)
        at zmq.SocketBase.send(SocketBase.java:718)
        at org.zeromq.ZMQ$Socket.send(ZMQ.java:3197)
        at org.zeromq.ZMQ$Socket.send(ZMQ.java:3111)
        at org.tron.common.logsfilter.nativequeue.NativeMessageQueue.publishTrigger(NativeMessageQueue.java:65)
        at org.tron.common.logsfilter.EventPluginLoader.postSolidityLogTrigger(EventPluginLoader.java:477)
        at org.tron.core.db.Manager.postSolidityLogContractTrigger(Manager.java:1693)
        at org.tron.core.db.Manager.postSolidityTrigger(Manager.java:1924)
        at org.tron.core.db.Manager.pushBlock(Manager.java:1182)
        at org.tron.core.consensus.BlockHandleImpl.produce(BlockHandleImpl.java:54)
        at org.tron.consensus.dpos.DposTask.produceBlock(DposTask.java:107)
        at org.tron.consensus.dpos.DposTask.lambda$init$0(DposTask.java:57)
        at java.lang.Thread.run(Thread.java:748)

Reason

I found use ZMQ.Socket to send data may fail in multithreads with rare rates. it's almost the same problem as that case in https://github.com/zeromq/jeromq/issues/500.

Solution

I catch the RuntimeException:

  public void publishTrigger(String data, String topic) {
    if (Objects.isNull(publisher) || Objects.isNull(context.isClosed()) || context.isClosed()) {
      return;
    }

    try {
      publisher.sendMore(topic);
      publisher.send(data);
    } catch (RuntimeException e) {
      logger.error("write data to zeromq failed, data:{}, topic:{}, error:{}", data, topic,
          e.getMessage());
    }
  }