this is (probably )not a bug. This is more of a report in case anyone experience the same problem and does not search for too long.
We use http2 heavily via tonic. Recently, we experience a mysterious bug. Quickwit would become unstable after running for a few hours on a rather large environment (35 nodes, 3k indexes).
In this environment it is not abnormal to several hundreds long running grpc streams.
Before reaching the instability, a few error message like this would show up:
internal error: h2 protocol error: http2 error: stream error sent by user: unexpected internal error encountered
It would show up indiscrimately on different grpc calls.
I assume we end up getting an internal error code instead of a stream closed code on some streams because they are not tracked in h2 anymore... We do not yet have a proper understanding of why this translated on a drop in our indexing throughput.
this is (probably )not a bug. This is more of a report in case anyone experience the same problem and does not search for too long.
We use http2 heavily via tonic. Recently, we experience a mysterious bug. Quickwit would become unstable after running for a few hours on a rather large environment (35 nodes, 3k indexes).
In this environment it is not abnormal to several hundreds long running grpc streams.
Before reaching the instability, a few error message like this would show up:
It would show up indiscrimately on different grpc calls.
They would then show up periodically.
Reverting to hyper@0.14.28 solved the problem (probably due to https://github.com/hyperium/hyper/commit/dedcb674f35eaec765a42b550caabe6f694d86d1).
I assume we end up getting an internal error code instead of a stream closed code on some streams because they are not tracked in h2 anymore... We do not yet have a proper understanding of why this translated on a drop in our indexing throughput.