http2 error: stream error received: unspecific protocol error detected

[wangjia184]

I have a reverse proxy basing on hyper and tokio. There are two backends, actually they are the same application but using different transport -- DotNet Core provides two transport choices. Kestrel and Http.sys(windows kernel driver, which only supports secure http/2, http.sys does not support plain-text http/2)

From the diagram.

• C# client can directly call gRPC on both backends directly (Blue lines)
• C# client can call gRPC API over plain-text HTTP/2 protocol through my Rust reverse proxy (green line).
• C# client cannot call gRPC API via my Rust reverse proxy when backend is secure HTTP/2 protocol (yellow and red line). The TLS handshake succeeds, but after that, it immediately failed with http2 error: stream error received: unspecific protocol error detected

Log below from rustls

[2023-11-03T10:15:50Z DEBUG rustls::anchors] add_parsable_certificates processed 1 valid and 0 invalid certs
[2023-11-03T10:15:50Z DEBUG rustls::client::hs] No cached session for DnsName("api.xxx-txn.local")
[2023-11-03T10:15:50Z DEBUG rustls::client::hs] Not resuming any session
[2023-11-03T10:15:50Z TRACE rustls::client::hs] Sending ClientHello Message {
        version: TLSv1_0,
        payload: Handshake {
            parsed: HandshakeMessagePayload {
                typ: ClientHello,
                payload: ClientHello(
                    ClientHelloPayload {
                        client_version: TLSv1_2,
                        random: 3801fa99382e1ebaefd30c6d2d3f591f60d1831a78e3f498477a9bec67c5ff30,
                        session_id: 77eadeeae0c79b53906bd66dec7934c6716ae7dd965e53767f4b3fc2c2e0d2a7,
                        cipher_suites: [
                            TLS13_AES_256_GCM_SHA384,
                            TLS13_AES_128_GCM_SHA256,
                            TLS13_CHACHA20_POLY1305_SHA256,
                            TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                            TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                            TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
                            TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
                            TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
                            TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
                            TLS_EMPTY_RENEGOTIATION_INFO_SCSV,
                        ],
                        compression_methods: [
                            Null,
                        ],
                        extensions: [
                            SupportedVersions(
                                [
                                    TLSv1_3,
                                    TLSv1_2,
                                ],
                            ),
                            ECPointFormats(
                                [
                                    Uncompressed,
                                ],
                            ),
                            NamedGroups(
                                [
                                    X25519,
                                    secp256r1,
                                    secp384r1,
                                ],
                            ),
                            SignatureAlgorithms(
                                [
                                    ECDSA_NISTP384_SHA384,
                                    ECDSA_NISTP256_SHA256,
                                    ED25519,
                                    RSA_PSS_SHA512,
                                    RSA_PSS_SHA384,
                                    RSA_PSS_SHA256,
                                    RSA_PKCS1_SHA512,
                                    RSA_PKCS1_SHA384,
                                    RSA_PKCS1_SHA256,
                                ],
                            ),
                            ExtendedMasterSecretRequest,
                            CertificateStatusRequest(
                                OCSP(
                                    OCSPCertificateStatusRequest {
                                        responder_ids: [],
                                        extensions: ,
                                    },
                                ),
                            ),
                            ServerName(
                                [
                                    ServerName {
                                        typ: HostName,
                                        payload: HostName(
                                            DnsName(
                                                "api.xxx-txn.local",
                                            ),
                                        ),
                                    },
                                ],
                            ),
                            SignedCertificateTimestampRequest,
                            KeyShare(
                                [
                                    KeyShareEntry {
                                        group: X25519,
                                        payload: 406e1467701aeaa581a89b1374bd90147e322cc09a7920103f2c6c1b7e68416f,
                                    },
                                ],
                            ),
                            PresharedKeyModes(
                                [
                                    PSK_DHE_KE,
                                ],
                            ),
                            Protocols(
                                [
                                    ProtocolName(
                                        6832,
                                    ),
                                ],
                            ),
                            SessionTicket(
                                Request,
                            ),
                        ],
                    },
                ),
            },
            encoded: 010000f8030....,
        },
    }
[2023-11-03T10:15:50Z TRACE rustls::client::hs] We got ServerHello ServerHelloPayload {
        legacy_version: TLSv1_2,
        random: ba4eef0d04460d1e41f874290f5e2b655d9920a08a49849bab52ab54cad528cd,
        session_id: 77eadeeae0c79b53906bd66dec7934c6716ae7dd965e53767f4b3fc2c2e0d2a7,
        cipher_suite: TLS13_AES_256_GCM_SHA384,
        compression_method: Null,
        extensions: [
            SupportedVersions(
                TLSv1_3,
            ),
            KeyShare(
                KeyShareEntry {
                    group: X25519,
                    payload: 62c03c22705562fcea57b800d063889cdb733ca45a0c27daa6ea2caf0b306b25,
                },
            ),
        ],
    }
[2023-11-03T10:15:50Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2023-11-03T10:15:50Z DEBUG rustls::client::tls13] Not resuming
[2023-11-03T10:15:50Z TRACE rustls::client::client_conn] EarlyData rejected
[2023-11-03T10:15:50Z TRACE rustls::conn] Dropping CCS
[2023-11-03T10:15:50Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]
[2023-11-03T10:15:50Z DEBUG rustls::client::hs] ALPN protocol is Some(b"h2")
[2023-11-03T10:15:50Z TRACE rustls::client::tls13] Server cert is [Certificate(b"0\x82\x03\xc10\x82\x02\xa9\xa0.....")]

Source code of handling upstreaming.

pub struct UpstreamConnector {
    target: SocketAddr,
    tls : bool,
}

impl Service<Uri> for UpstreamConnector {
    type Response = MaybeTlsStream;
    type Error = std::io::Error;

    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;

    fn poll_ready(&mut self, _: &mut core::task::Context<'_>) -> Poll<Result<(), Self::Error>> {
        Poll::Ready(Ok(()))
    }

    fn call(&mut self, _: Uri) -> Self::Future {
        let addr = self.target.clone();
        let tls = self.tls;
        Box::pin(async move {
            let stream = TcpStream::connect(addr).await?;

            if tls {

                let config = TLS_CLIENT_CONFIG.get_or_init(|| {
                    let file = std::fs::File::open(".\\CAPrivate.pem").expect("Failed to open CA certificates");
                    let mut reader = std::io::BufReader::new(file);
                    let certs = rustls_pemfile::certs(&mut reader).expect("Unable to load custom CA certificates");

                    let mut root_cert_store = RootCertStore::empty();
                    root_cert_store.add_parsable_certificates(&certs);

                    let mut config = ClientConfig::builder()
                        .with_safe_defaults()
                        .with_root_certificates(root_cert_store)
                        .with_no_client_auth();
                    config.dangerous().set_certificate_verifier(Arc::new(NoCertVerification{}));

                    config.alpn_protocols.push(b"h2".to_vec());
                    Arc::new(config)
                }).clone();

                let connector = TlsConnector::from(config);

                let dnsname = ServerName::try_from("api.xxx-txn.local").unwrap();
                let tls_stream = connector.connect(dnsname, stream).await.map_err(|e| {
                    error!("Failed to complete TLS handshake with {}", addr);
                    e
                })?;

                info!("TLS handshake succeeded");
                // succeeds to complete handshake

                return Ok(MaybeTlsStream::Rustls(tls_stream));
            } else {
                return Ok(MaybeTlsStream::Plain(stream));  // succeeds using plain-text HTTP/2
            }
        })
    }
}

// https://docs.rs/rustls/latest/rustls/client/trait.ServerCertVerifier.html
struct NoCertVerification;
impl ServerCertVerifier for NoCertVerification {
    // Required method
    fn verify_server_cert(
        &self,
        _end_entity: &Certificate,
        _intermediates: &[Certificate],
        _server_name: &ServerName,
        _scts: &mut dyn Iterator<Item = &[u8]>,
        _ocsp_response: &[u8],
        _now: SystemTime
    ) -> Result<ServerCertVerified, rustls::Error> {

        Ok(ServerCertVerified::assertion())
    }
}

// A wrapper for either TcpStream or TlsStream
pub enum MaybeTlsStream {
    Plain(TcpStream),
    Rustls(TlsStream<TcpStream>),
}

impl AsyncRead for MaybeTlsStream {

    fn poll_read(
        self: Pin<&mut Self>,
        cx: &mut core::task::Context<'_>,
        buf: &mut ReadBuf<'_>,
    ) -> Poll<tokio::io::Result<()>> {
        match self.get_mut() {
            MaybeTlsStream::Plain(tcp_stream) => {
                Pin::new(tcp_stream).poll_read(cx, buf)
            },
            MaybeTlsStream::Rustls(tls_stream) => {
                Pin::new(tls_stream).poll_read(cx, buf)
            }
        }

    }
}

impl AsyncWrite for MaybeTlsStream {
    fn poll_write(
        self: Pin<&mut Self>,
        cx: &mut core::task::Context<'_>,
        buf: &[u8],
    ) -> Poll<tokio::io::Result<usize>> {
        match self.get_mut() {
            MaybeTlsStream::Plain(tcp_stream) => {
                Pin::new(tcp_stream).poll_write(cx, buf)
            },
            MaybeTlsStream::Rustls(tls_stream) => {
                Pin::new(tls_stream).poll_write(cx, buf)
            }
        }
    }

    fn poll_write_vectored(
        self: Pin<&mut Self>,
        cx: &mut core::task::Context<'_>,
        bufs: &[IoSlice<'_>],
    ) -> Poll<tokio::io::Result<usize>> {
        match self.get_mut() {
            MaybeTlsStream::Plain(tcp_stream) => {
                Pin::new(tcp_stream).poll_write_vectored(cx, bufs)
            },
            MaybeTlsStream::Rustls(tls_stream) => {
                Pin::new(tls_stream).poll_write_vectored(cx, bufs)
            }
        }
    }

    fn poll_flush(self: Pin<&mut Self>, cx: &mut core::task::Context<'_>) -> Poll<tokio::io::Result<()>> {
        match self.get_mut() {
            MaybeTlsStream::Plain(tcp_stream) => {
                Pin::new(tcp_stream).poll_flush(cx)
            },
            MaybeTlsStream::Rustls(tls_stream) => {
                Pin::new(tls_stream).poll_flush(cx)
            }
        }
    }

    fn poll_shutdown(self: Pin<&mut Self>, cx: &mut core::task::Context<'_>) -> Poll<tokio::io::Result<()>> {
        match self.get_mut() {
            MaybeTlsStream::Plain(tcp_stream) => {
                Pin::new(tcp_stream).poll_shutdown(cx)
            },
            MaybeTlsStream::Rustls(tls_stream) => {
                Pin::new(tls_stream).poll_shutdown(cx)
            }
        }
    }
}

impl hyper::client::connect::Connection for MaybeTlsStream {
    fn connected(&self) -> Connected {
        match self {
            MaybeTlsStream::Plain(tcp_stream) => {
                tcp_stream.connected()
            },
            MaybeTlsStream::Rustls(tls_stream) => {
                let (tcp_stream, _ )  = tls_stream.get_ref();
                tcp_stream.connected()
            }
        }
    }
}

hyper::Client::request() method failed with http2 error: stream error received: unspecific protocol error detected.

The TLS shandshake succeeds but then it failed with this error immdetaitely. Checked on backend side. And there is no log from http.sys for this failed request.

[seanmonstar]

Can you enable logs for h2? That should show the frames sent and frames received. (h2 uses tracing, you might need to enable tracing's log feature to get interop).

[wangjia184]

Thanks @seanmonstar ,here is trace.log

[2023-11-04T00:26:15Z TRACE h2::proto::streams::streams] drop_stream_ref; stream=Stream { id: StreamId(1), state: State { inner: Closed(Error(Reset(StreamId(1), PROTOCOL_ERROR, Remote))) }, is_counted: false, ref_count: 1, next_pending_send: None, is_pending_send: false, send_flow: FlowControl { window_size: Window(1048264), available: Window(0) }, requested_send_capacity: 0, buffered_send_data: 0, send_task: None, pending_send: Deque { indices: None }, next_pending_send_capacity: None, is_pending_send_capacity: false, send_capacity_inc: true, next_open: None, is_pending_open: false, is_pending_push: false, next_pending_accept: None, is_pending_accept: false, recv_flow: FlowControl { window_size: Window(2097152), available: Window(2097152) }, in_flight_recv_data: 0, next_window_update: None, is_pending_window_update: false, reset_at: None, next_reset_expire: None, pending_recv: Deque { indices: None }, is_recv: true, recv_task: None, pending_push_promises: Queue { indices: None, _p: PhantomData<h2::proto::streams::stream::NextAccept> }, content_length: Omitted }
[2023-11-04T00:26:15Z TRACE h2::proto::streams::counts] transition_after; stream=StreamId(1); state=State { inner: Closed(Error(Reset(StreamId(1), PROTOCOL_ERROR, Remote))) }; is_closed=true; pending_send_empty=true; buffered_send_data=0; num_recv=0; num_send=0
[2023-11-04T00:26:15Z DEBUG hyper::proto::h2::client] client response error: stream error received: unspecific protocol error detected
[2023-11-04T00:26:15Z TRACE tracing::span::active] -> Connection;
[2023-11-04T00:26:15Z WARN  ubs_edge::http_server] POST 127.0.0.1/UbsService/PredictBonus - http2 error: stream error received: unspecific protocol error detected
[2023-11-04T00:26:15Z TRACE tracing::span] poll;
[2023-11-04T00:26:15Z WARN  hyper::proto::h2::server] http2 service errored: error from user's Service: http2 error: stream error received: unspecific protocol error detected
[2023-11-04T00:26:15Z TRACE tracing::span::active] -> poll;
[2023-11-04T00:26:15Z TRACE h2::proto::streams::send] send_reset(..., reason=PROTOCOL_ERROR, initiator=User, stream=StreamId(1), ..., is_reset=false; is_closed=false; pending_send.is_empty=true; state=State { inner: HalfClosedRemote(AwaitingHeaders) }

[seanmonstar]

It looks fine until the server claims there's a protocol error with the request. What headers do you send?

[wangjia184]

I see where the problem is, the scheme changed hence it needs be changed from http to https. Unlike HTTP/1, Uri scheme is part of the header. thanks @seanmonstar