GRPC/http2

[paulobressan]

What is the problem your feature solves, or the need it fulfills?

I'm trying to create a proxy for GRPC, but I don't know how. So I created an HTTP proxy to use http2 and followed the examples and I'm receiving an error of invalid HTTP version. Would someone be able to help me with an example? Maybe It needs a different configuration...

Describe the solution you'd like

Maybe a new example and docs will solve the problem.

Additional context

Log from Pingora proxy ERROR pingora_proxy: Fail to proxy: Downstream InvalidHTTPHeader context: buf: "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n\0\0\u{12}\u{4}\0\0\0\0\0\0\u{2}\0\0\0\0\0\u{4}\0 \0\0\0\u{5}\0\0@\0\0\0\u{4}\u{8}\0\0\0\0\0\0O\0\u{1}" cause: invalid HTTP version

Log from GPRC client using tonic Error: Status { code: Unknown, message: "h2 protocol error: http2 error: connection error detected: frame with invalid size", source: Some(tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: GoAway(b"", FRAME_SIZE_ERROR, Library) }))) }

[vicanso]

Is the grpc service https or h2c ? H2C is not supported, #66

[paulobressan]

Is the grpc service https or h2c?

Hi @vicanso, thank you for your answer. I'm trying to create a grpc service https. But can you explain about h2c and what is the difference between them?

[vicanso]

HTTP/2 over TCP: https://httpwg.org/specs/rfc7540.html#rfc.section.3.1

[paulobressan]

H2C is not supported, #66

@vicanso Does it mean Pingora doesn't support grpc?

[vicanso]

You can new grpc server with tls cert. My golang program:

    creds, err := credentials.NewServerTLSFromFile("~/tmp/me.dev+3.pem", "~/tmp//me.dev+3-key.pem")
    if err != nil {
        return err
    }

    s := grpc.NewServer(grpc.Creds(creds))

[paulobressan]

Do you have an example using Pingora?

[vicanso]

I think gRPC proxying requires your service to support tls.

[vicanso]

tls: https://github.com/hyperium/tonic/blob/master/examples/src/tls/server.rs#L39

h2c: https://github.com/hyperium/tonic/blob/master/examples/src/h2c/server.rs#L42

[paulobressan]

@vicanso but in those cases, I need to use the proto files in the proxy as well and I wouldn't like it... Do you know a way?

[vicanso]

Do you want a grpc gateway like: https://github.com/grpc-ecosystem/grpc-gateway ? I thinks pingora doesn't support it.

[jpvolt]

even with tls, the httpproxy fails to proxy grpc, the request goes upstream but never comes back totally to the client.

upstream peer is: "127.0.0.1:50055"
upstream_response: ResponseHeader { base: Parts { status: 200, version: HTTP/2.0, headers: {"server": "nginx/1.18.0 (Ubuntu)", "date": "Thu, 11 Apr 2024 20:29:10 GMT", "content-type": "application/grpc"} }, header_name_map: None }
2024-04-11T20:24:17.479479Z  WARN pingora_core::connectors: Unexpected data read in idle connection

on the client:

Received RST_STREAM with code 0

maybe something to do with trailers? im using basically the default example of proxy with tls, and my grpc server is behind an nignx to provide tls.

[jpvolt]

I found out that you can force http/2 with:

        let mut p = HttpPeer::new(upstream, false, "");

        p.get_mut_peer_options().unwrap().alpn = ALPN::H2;
        let peer = Box::new(p);

this makes the upstream request work, but still does not proxy it correctly.

[paulobressan]

@jpvolt I'm trying, but I configured pingora and now downstream works, but now I'm trying to understand how I can proxy to the service request. Loot my code.

use async_trait::async_trait;
use dotenv::dotenv;
use pingora::protocols::ALPN;
use pingora::server::{configuration::Opt, Server};
use pingora::Result;
use pingora::{
    proxy::{ProxyHttp, Session},
    upstreams::peer::HttpPeer,
};
use tracing::Level;

pub struct GrpcProxy;

fn main() {
    dotenv().ok();

    tracing_subscriber::fmt().with_max_level(Level::INFO).init();

    let opt = Opt::default();
    let mut server = Server::new(Some(opt)).unwrap();
    server.bootstrap();

    let mut grpc_proxy = pingora::proxy::http_proxy_service(&server.configuration, GrpcProxy);

    let mut tls_settings =
        pingora::listeners::TlsSettings::intermediate("cert/localhost.crt", "cert/localhost.key")
            .unwrap();

    tls_settings.enable_h2();

    grpc_proxy.add_tls_with_settings("0.0.0.0:8000", None, tls_settings);
    // grpc_proxy.add_tcp("0.0.0.0:8000");
    server.add_service(grpc_proxy);

    server.run_forever();
}

#[async_trait]
impl ProxyHttp for GrpcProxy {
    type CTX = ();
    fn new_ctx(&self) -> Self::CTX {}

    async fn upstream_peer(
        &self,
        _session: &mut Session,
        _ctx: &mut Self::CTX,
    ) -> Result<Box<HttpPeer>> {
        let mut peer = Box::new(HttpPeer::new("0.0.0.0:50051", false, String::default()));
        peer.options.alpn = ALPN::H2;
        Ok(peer)
    }
}

The client command to test

grpcurl -insecure -d '{"message": "Hello gRPC"}' -import-path . -proto ./proto/echo.proto localhost:8000 echo.Echo/Echo

The client error

ERROR:
  Code: Internal
  Message: server closed the stream without sending trailers

For the mock service, I'm using an example from tonic.

I didn't receive any error on the server.

[vicanso]

I can get the trailers {"grpc-status": "0", "grpc-message": ""} in response_trailer_filter, but I don't know how to encode it.

/// When a trailer is received.
    async fn response_trailer_filter(
        &self,
        _session: &mut Session,
        _upstream_trailers: &mut header::HeaderMap,
        _ctx: &mut Self::CTX,
    ) -> Result<Option<Bytes>>
    where
        Self::CTX: Send + Sync,
    {
        Ok(None)
    }

[jpvolt]

@vicanso yeah, I have the same problem here. I dont know http2 spec enough to encode, and the header map does not seem to have any trait that lets us return Bytes, assuming that the bytes on the results are the actually trailer bytes.

[andrewhavck]

This is a bug, we're not propagating upstream h2 trailers downstream, will have a patch for this soon.

[spencerbart]

Just ran into this error. Happy to help out if any is needed.

[jpvolt]

@andrewhavck , do you want any help? I'm available to contribute to this patch.

[andrewhavck]

We merged the fix internally last week it should sync this week.

[jpvolt]

thanks!

[johnhurt]

The fix @andrewhavck mentioned is available on the main branch. We are not planning on publishing a release with that for at least another week. If using this feature of the main git branch is a problem, let us know here.

[paulobressan]

@andrewhavck @johnhurt

Thank you very much, It's working!

is it possible to track each "message" in a stream connection? Because I need to metrify each event. I can only get the first event connection in the logging function but the event in the stream no.

Screencast from 2024-04-29 18-49-17.webm

[paulobressan]

Hi @andrewhavck, sorry for bothering you, do you know if it's possible to capture each event in a GRPC stream connection?

[andrewhavck]

@paulobressan we don't support this right now, feel free to open another issue. Closing this one as the original problem is now fixed.