search

hashgraph / hedera-sdk-rust

The Hedera™ Rust SDK
Apache License 2.0
42 stars 14 forks source link

Mainnet node connections are breaking - make gRPC endpoint connection configurable #833

Open iamjpotts opened 2 months ago

iamjpotts commented 2 months ago

Problem

We are seeing a significant number of TCP and HTTP connection related errors with the gRPC endpoints, such as:

GrpcStatus(Status { code: Unavailable, message: "error trying to connect: tcp connect error: Broken pipe (os error 32)", source: Some(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })))) })

GrpcStatus(Status { code: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", source: Some(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))) })

GrpcStatus(Status { code: Unavailable, message: "error trying to connect: tcp connect error: deadline has elapsed", source: Some(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Custom { kind: TimedOut, error: Elapsed(()) })))) })

GrpcStatus(Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 503 Service Unavailable", metadata: MetadataMap { headers: {"content-length": "107", "cache-control": "no-cache", "content-type": "text/html"} }, source: None })

In most cases, backing off and retrying after a short delay of 1-2 seconds results in a successful request.

Some, but not all, could potentially be mitigated or avoided by adjustments to timeouts and keep alives, or in the alternative, adjustments to those attributes may empirically show they do not resolve the issue and stability efforts should be put elsewhere.

Solution

In the NodeConnection channel setup, several values are hard coded:

https://github.com/hashgraph/hedera-sdk-rust/blob/40cc835628347772ca1cc71b9d860e2a4c3b1214/src/client/network/mod.rs#L594-L597

We would like to make them configurable at the time a new Client is created.

Alternatives

iamjpotts commented 2 months ago

This issue is related to but not closed by: