search

hashgraph / hedera-block-node

New Block Node services
Apache License 2.0
15 stars 2 forks source link

bug: Helidon fails with ArrayIndexOutOfBoundsException under load #178

Open mattp-swirldslabs opened 1 month ago

mattp-swirldslabs commented 1 month ago

Here is the error thrown from Helidon when the Block Node is under load:

2024-09-16 09:38:45 Sep 16, 2024 3:38:45 PM com.hedera.block.server.producer.ProducerBlockItemObserver onError
2024-09-16 09:38:45 SEVERE: onError method invoked with an exception: 
2024-09-16 09:38:45 io.grpc.StatusRuntimeException: CANCELLED: client cancelled
2024-09-16 09:38:45     at io.grpc@1.65.1/io.grpc.Status.asRuntimeException(Status.java:525)
2024-09-16 09:38:45     at io.grpc.stub@1.65.1/io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291)
2024-09-16 09:38:45     at io.helidon.webserver.grpc@4.1.0/io.helidon.webserver.grpc.GrpcProtocolHandler.data(GrpcProtocolHandler.java:222)
2024-09-16 09:38:45     at io.helidon.webserver.http2@4.1.0/io.helidon.webserver.http2.Http2ServerStream.handle(Http2ServerStream.java:526)
2024-09-16 09:38:45     at io.helidon.webserver.http2@4.1.0/io.helidon.webserver.http2.Http2ServerStream.run(Http2ServerStream.java:282)
2024-09-16 09:38:45     at io.helidon.webserver.http2@4.1.0/io.helidon.webserver.http2.Http2Connection$StreamRunnable.run(Http2Connection.java:895)
2024-09-16 09:38:45     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2024-09-16 09:38:45     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2024-09-16 09:38:45     at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)
2024-09-16 09:38:45 
2024-09-16 09:38:45 Sep 16, 2024 3:38:45 PM io.helidon.webserver.grpc.GrpcProtocolHandler data
2024-09-16 09:38:45 SEVERE: Failed to process grpc request: +--------+-------------------------------------------------+----------------+
2024-09-16 09:38:45 |   index|  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |            data|
2024-09-16 09:38:45 +--------+-------------------------------------------------+----------------+
2024-09-16 09:38:45 |00000000| 00 00 00                                        |...             |
2024-09-16 09:38:45 +--------+-------------------------------------------------+----------------+
2024-09-16 09:38:45 
2024-09-16 09:38:45 java.lang.ArrayIndexOutOfBoundsException: This buffer has 3 bytes, requested to read at 3
2024-09-16 09:38:45     at io.helidon.common.buffers@4.1.0/io.helidon.common.buffers.ReadOnlyArrayData.read(ReadOnlyArrayData.java:64)
2024-09-16 09:38:45     at io.helidon.common.buffers@4.1.0/io.helidon.common.buffers.BufferData.readUnsignedInt32(BufferData.java:328)
2024-09-16 09:38:45     at io.helidon.webserver.grpc@4.1.0/io.helidon.webserver.grpc.GrpcProtocolHandler.data(GrpcProtocolHandler.java:191)
2024-09-16 09:38:45     at io.helidon.webserver.http2@4.1.0/io.helidon.webserver.http2.Http2ServerStream.handle(Http2ServerStream.java:526)
2024-09-16 09:38:45     at io.helidon.webserver.http2@4.1.0/io.helidon.webserver.http2.Http2ServerStream.run(Http2ServerStream.java:282)
2024-09-16 09:38:45     at io.helidon.webserver.http2@4.1.0/io.helidon.webserver.http2.Http2Connection$StreamRunnable.run(Http2Connection.java:895)
2024-09-16 09:38:45     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2024-09-16 09:38:45     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2024-09-16 09:38:45     at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)

Details:

a-saksena commented 3 weeks ago

Piggyback on Helidon PBJ plugin as per @rbair23 and @mattp-swirldslabs 's conversation.

mattp-swirldslabs commented 1 week ago

Tracking exploratory data here: https://www.notion.so/swirldslabs/Block-Node-Network-Stack-Troubleshooting-11c7c9ab2591800ca277ed42ecad2b54?pvs=4