Error: 2 UNKNOWN: Stream removed

usnmas commented 4 years ago

Description

For the load test using node.js, client node sends 350 query tx/sec simultaneously to Hyperledger Fabric 1.4.4 nodes based on 4 peers (containers) with level DB.
Up to 200 concurrent tx's, there's no error at all.
250 and 300 tx's showed one or two of the same error (only the 1st line) during 50 repeated tests respectively.

Reference

Supposed to be related to grpc problems
grpc issue
similar issue
peer.js

Logs

/home/guest/code/node % ./queryDataAutoAT.sh & [1] 3019269 /home/guest/code/node % 2020-04-07T06:14:50.574Z - error: [Channel.js]: Error: 2 UNKNOWN: Stream removed 2020-04-07T06:16:09.771Z - error: [Channel.js]: Error: 2 UNKNOWN: Stream removed 2020-04-07T06:16:04.036Z - error: [Channel.js]: Error: 2 UNKNOWN: Stream removed 2020-04-07T06:17:12.809Z - error: [Channel.js]: Error: 14 UNAVAILABLE: failed to connect to all addresses 2020-04-07T06:18:08.033Z - error: [Channel.js]: Error: 14 UNAVAILABLE: failed to connect to all addresses 2020-04-07T06:25:01.403Z - error: [Channel.js]: Error: 2 UNKNOWN: Stream removed 2020-04-07T06:24:47.084Z - error: [Channel.js]: Error: 14 UNAVAILABLE: failed to connect to all addresses 2020-04-07T06:25:27.044Z - error: [Peer.js]: sendDiscovery - timed out after:300000 2020-04-07T06:25:27.102Z - error: [Channel.js]: Error: REQUEST_TIMEOUT 2020-04-07T06:26:21.184Z - error: [Channel.js]: Error: 14 UNAVAILABLE: failed to connect to all addresses 2020-04-07T06:30:26.629Z - error: [Network]: _initializeInternalChannel: Unable to initialize channel. Attempted to contact 4 Peers. Last error was Error: 2 UNKNOWN: Stream removed at Object.exports.createStatusError (/local/scratch/code/node/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:1209:28) at InterceptingListener._callNext (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:847:24) { code: 2, metadata: [Metadata], details: 'Stream removed', peer: [Object] } Failed to evaluate transaction: Error: Unable to initialize channel. Attempted to contact 4 Peers. Last error was Error: 2 UNKNOWN: Stream removed at Object.exports.createStatusError (/local/scratch/code/node/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:1209:28) at InterceptingListener._callNext (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:847:24) { code: 2, metadata: [Metadata], details: 'Stream removed', peer: [Object] } 2020-04-07T06:30:45.669Z - error: [Peer.js]: sendDiscovery - timed out after:300000 2020-04-07T06:31:00.838Z - error: [Channel.js]: Error: REQUEST_TIMEOUT 2020-04-07T06:31:09.319Z - error: [Channel.js]: Error: 14 UNAVAILABLE: failed to connect to all addresses 2020-04-07T06:31:22.620Z - error: [Peer.js]: sendDiscovery - timed out after:300000 2020-04-07T06:31:28.080Z - error: [Channel.js]: Error: 2 UNKNOWN: Stream removed 2020-04-07T06:31:55.002Z - error: [Peer.js]: sendDiscovery - timed out after:300000 2020-04-07T06:31:55.104Z - error: [Channel.js]: Error: REQUEST_TIMEOUT 2020-04-07T06:52:25.709Z - error: [Network]: _initializeInternalChannel: Unable to initialize channel. Attempted to contact 4 Peers. Last error was Error: REQUEST_TIMEOUT at Timeout._onTimeout (/local/scratch/code/node/node_modules/fabric-client/lib/Peer.js:200:19) at listOnTimeout (internal/timers.js:549:17) at processTimers (internal/timers.js:492:7) 2020-04-07T06:53:22.572Z - error: [Peer.js]: sendProposal - timed out after:300000 E0407 18:53:22.826838802 3020047 chttp2_transport.cc:2838] ipv4:130.195.10.172:8051: Keepalive watchdog fired. Closing transport. 2020-04-07T06:52:12.264Z - error: [Peer.js]: sendDiscovery - timed out after:300000 2020-04-07T06:53:43.304Z - error: [Peer.js]: sendDiscovery - timed out after:300000 Failed to evaluate transaction: Error: Unable to initialize channel. Attempted to contact 4 Peers. Last error was Error: REQUEST_TIMEOUT at Timeout._onTimeout (/local/scratch/code/node/node_modules/fabric-client/lib/Peer.js:200:19) at listOnTimeout (internal/timers.js:549:17) at processTimers (internal/timers.js:492:7) 2020-04-07T06:54:02.329Z - error: [Peer.js]: sendDiscovery - timed out after:300000 2020-04-07T06:54:30.817Z - error: [Channel.js]: Error: REQUEST_TIMEOUT 2020-04-07T06:55:22.354Z - error: [Channel.js]: Error: REQUEST_TIMEOUT 2020-04-07T06:55:37.629Z - error: [Network]: _initializeInternalChannel: Unable to initialize channel. Attempted to contact 4 Peers. Last error was Error: 14 UNAVAILABLE: failed to connect to all addresses at Object.exports.createStatusError (/local/scratch/code/node/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:1209:28) at InterceptingListener._callNext (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/local/scratch/code/node/node_modules/grpc/src/client_interceptors.js:847:24) { code: 14, metadata: [Metadata], details: 'failed to connect to all addresses', peer: [Object] }

usnmas commented 4 years ago

Stack Overflow : Increasing the maximum number of TCP/IP connections in Linux net.ipv4.ip_local_port_range / net.ipv4.tcp_fin_timeout

usnmas commented 4 years ago

gRPC blog : gRPC and Deadlines DEADLINE_EXCEEDED

usnmas commented 4 years ago

gRPC GO github : How to set client default timeout #2130 WithUnaryInterceptor / WithStreamInterceptor - func WithTimeout

usnmas commented 4 years ago

gRPC DOC - gRPC Concepts : RPC life cycle \ Unary / Streaming \ Deadlines / Timeouts \ RPC Termination / Cancelling an RPC

axcasella commented 3 years ago

@usnmas came across this post when I was researching resolutions for the same issue. Did you resolve it? If so, can you share some insights?

usnmas / blockchain