search

tikv / grpc-rs

The gRPC library for Rust built on C Core library and futures
Apache License 2.0
1.81k stars 253 forks source link

TiKV runs into Handshake failed with fatal error SSL_ERROR_SSL. #619

Closed tonyxuqqi closed 1 year ago

tonyxuqqi commented 1 year ago

Describe the bug

The grpc-rs version is 0.10.4.

A clear and concise description of what the bug is.

TiKV sometimes runs into this error:

2023-06-16 21:49:26 | [2023/06/16 21:49:26.773 +00:00] [ERROR] [] ["Handshake failed with fatal error SSL_ERROR_SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number."]

The error is from coprocessor endpoint. When it happens, it fails the coprocessor's new connection. And also it would trigger the leader election. It appears that some existing connections can still work as it still can have some QPS during this time. When this happens, there's no network package drop or 100% CPU/memory usage. The load is quite normal.

To Reproduce Steps to reproduce the behavior: No exact repro step.

Expected behavior A clear and concise description of what you expected to happen.

System information

Additional context Add any other context about the problem here.

BusyJay commented 1 year ago

Endpoint failure should not trigger elections, it may due to other reasons. The error itself is probably caused by mixing http and https requests.

tonyxuqqi commented 1 year ago

Endpoint failure should not trigger elections, it may due to other reasons. The error itself is probably caused by mixing http and https requests.

Endpoint failure makes heartbeat request/response cannot be sent and thus triggers the election timeout.