grpc support

[elderapo]

Are there any plans/eta on supporting GRPC through cloudflared/argo tunnel?

[abelinkinbio]

You should be able to do this today with: https://developers.cloudflare.com/cloudflare-one/tutorials/warp-to-tunnel

[elderapo]

Doesn't seem like it. According to this article:

Limitations
The following products have limited capabilities with gRPC requests:

Argo Tunnel currently does not support gRPC.
...

[abelinkinbio]

@elderapo good catch. This is outdated - we'll get it updated.

[nikr-canva]

good catch. This is outdated - we'll get it updated.

Just bumping to mention that these docs are still not updated - I was about to give up on using an argo tunnel for gRPC until I found this issue.

[EngHabu]

👍🏽 this is still an issue. docs are not updated and I'm unable to use gRPC with Tunnels...

[abelinkinbio]

Thank you for bumping this thread. We're still working on getting docs updated to reflect that gRPC is supported through private network Tunnels: https://developers.cloudflare.com/cloudflare-one/tutorials/warp-to-tunnel

And that it is not supported through public hostnames Tunnels. This will likely be the case for some time as gRPC support is not on the immediate roadmap.

[Sherlouk]

Going to +1 on gRPC support for public tunnels.

This technology is increasing in popularity especially for messaging and analytical systems such as OpenTelemetry which many may wish to host on-premise to be in control of the data.

[Laur1nMartins]

Also a +1 from my side. Would be really useful for our current use case.

[abelinkinbio]

As a quick update, we're doing some discovery work right now to better understand the level of effort associated with supporting gRPC through our public hostname flow as well. We'll continue to share updates on support as we learn more. Thank you all for the additional upvotes!

[joliveirinha]

We made an analysis of what was missing and the effort necessary to support gRPC.

The reason why gRPC is not supported is because we are lacking on the support of proxying Trailers. We have made a test where we added this support to http2 transport between cloudflared and the edge, and it is working. However, supporting it over QUIC is more difficult because the protocol we use to proxy HTTP request over QUIC is not HTTP3, so we don't have the framing necessary to support trailers.

For this reason, there isn't an easy way to add that support. We plan on doing this probably on 2023 Q1. Until then, it will be possible to try gRPC on top of http once we make a new edge release and also cloudflared, but this is still not officially supported until we implement it for QUIC.

[EngHabu]

Thank you for looking into this @joliveirinha, @abelinkinbio! Do you mind updating us here when the new edge release is out and the unofficial support is there for us to try?

[abelinkinbio]

@EngHabu no problem at all. We'll be sure to notify this thread when the next release is available for testing.

[joliveirinha]

@EngHabu please feel free to test this now. You need to use the latest cloudflared version and also use a tunnel in local configuration mode. The reason for this is that the UI doesn't have the http2Origin configuration key that you need to add your ingress rule.

Also, note that you need to go to the cloudflared dashboard and enable gRPC support for your zone first.

[Laur1nMartins]

That sounds awesome!

Currently trying to make it work. I am still very new to all of this. I made sure that gRPC is enabled in the Network tab of my page. Then I created a cloudflared instance with a local configuration that connects to some subdomain of that network. What exactly do I need to put in the config.yml file that lets me accept http2 requests? I already made it work using nginx only but my efforts trying to make it work via cloudflared are unsuccessful so far.

I used this to create the cloudflared instance: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/#4-create-a-configuration-file So lets assume I have the app config mentioned there what do I need to add to proxy to an application running on localhost:50051?

[smyrgl]

Just verified that this works. You need to make sure you are using a tunnel that you create from the CLI rather than the ZeroTrust dashboard and it needs to have a config rule like this:

ingress:
    - hostname: my.example.com
      service: https://1.2.3.4
      originRequest:
        http2Origin: true

That's it. I confirmed this is working properly.

[traefik-74456bc5f6-5sc8j traefik]  10.1.235.0 - - [24/Sep/2022:05:18:31 +0000] "GET / HTTP/2.0" 200 724 "-" "-" 559 1ms

[polloloco]

When I decided to give cloudflared a try, I just assumed that grpc would be working out of the box - I was wrong obviously. After many different attempts I found this issue and tried to replicate what @smyrgl posted. It didn't want to work at all with http, so I had to modify my app to do tls with a selfsigned cert, only then cloudflared would negotiate a http2 connection. But no matter what I did, grpc wouldn't work at all. Did any of you get that working? For me every grpc request to the server would get stuck and eventually time out, without any useful error message on cloudflared (with --loglevel debug) I also verified that I had grpc support enabled for my zone, it still wouldn't work. Shouldn't that work? @joliveirinha For my tests I was using the latest version of cloudflared

cloudflared version 2022.9.1 (built 2022-09-22-0810 UTC)

Edit: I just noticed that a few minutes before I posted this, a new version (2022.10.0) was released, tried with that too and it didn't help.

Something I noticed is that the cloudflared log still shows loglines like this when there is a grpc request:

POST https://grpc.example.com/Service/SomeRpcFunction HTTP/1.1

Shouldn't that be HTTP/2?

When the grpc request gets the timeout, this is the error:

error="Incoming request ended abruptly: context canceled"

[joliveirinha]

@polloloco are you running cloudflared with --protocol http2?

As it was reported in this thread, for now we only made the change when tunnel is connected to the edge using http2 instead of quic.

[polloloco]

@joliveirinha No I wasn't - I didn't know that option is there, I didn't see it

That seems to be working for simple requests, thank you very much :) But in my application I have a serverside stream, as soon as I try to send that request the whole connection freezes and nothing goes through anymore, is that a known issue?

[joliveirinha]

It should work.

We have tested this with the official interop tests (https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md) and they were working correctly, which includes streaming requests.

Can you post the configuration you are using?

Also, @smyrgl can you confirm this worked for your gprc origin? does it have streaming endpoints?

[polloloco]

This is the config I'm using right now:

tunnel: 38160726-355f-40c6-b436-76f315e6060a
credentials-file: /root/.cloudflared/38160726-355f-40c6-b436-76f315e6060a.json
ingress:
  - hostname: grpc.example.com
    service: https://localhost:50051
    originRequest:
      noTLSVerify: true
      http2Origin: true
  - service: http_status:404

And I'm running the tunnel like this:

cloudflared tunnel --config config.yml --loglevel debug --protocol http2 run

I found this environment variable in the code TUNNEL_ORIGIN_ENABLE_HTTP2 and this commandline option --http2-origin, are those the same as the originRequest.http2Origin from the config? I tried using them but it didn't change anything for me, if I use them but don't have http2Origin: true in the config, every grpc requests fails with this error

Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: net/http: HTTP/1.x transport connection broken: malformed HTTP response

[Laur1nMartins]

I made it work using this config:

ingress:
        - hostname: <hostname>
          service: https://localhost:50051
          originRequest:
                  noTLSVerify: true
                  http2Origin: true
        - service: http_status:404

This is saved to the config.yml file.

cloudflared tunnel run

with no additional flags.

Judging by the error message you get and the problems I encountered myself maybe there is an error with your TLS setup in the app that you test with? Last time I checked not using any TLS Cert at all was not supported.

[polloloco]

The config looks identical to what I have, just the commandline is slightly different...

Last time I checked not using any TLS Cert at all was not supported.

Yes thats what I observed as well, at first I tried not using any tls, that didn't work at all. Then I switched to a selfsigned certificate and that allowed it to work, but only if I have --protocol http2 in the commandline, so I'm wondering how you got it working without that? @Laur1nMartins

And can you do streaming rpcs as well? "Normal" grpc methods work fine for me, but as soon as a stream is involved, something gets stuck and that request times out

[Nicarim]

~I was able to solve the problems with gRPC streaming getting stuck:~

1. I am using cloudflare/cloudflared:2022.10.2-amd64 as the tunnel endpoint
2. I had to add --http2-origin when running tunnel
3. I also enabled gRPC support in the domain that was involved with handling these requests (under network settings). Not sure if that is also required.

I didn't have to configure http2Origin for that hostname in tunnel's configuration

~This solved the problem for me of ArgoCD CLI getting stuck when it was trying to stream events, it now responds immediately.~

EDIT: Disregard, this didn't solve the problem for good, it still randomly gets stuck without a reason. EDIT2: I tried adding http2Origin to the ingress config as well, but it still doesn't work. It just randomly gets gRPC requests stuck for a while and then works...

[felipereyel]

Can anyone from the team sum up the info? I find that the documentation is still stating that gRPC is not supported.

I tried some of the things mentioned here and no luck.

[robinbraemer]

https://connect.build/ web or connect protocol might work, haven’t tested it yet. simple http and websockets always work.

[MaxtronGaming]

Are there any updates on gRPC support? @joliveirinha

I am now completely confused, if gRPC via a Argo/Cloudflare tunnel is possible. The docs state that it is not possible and this thread is a mixed bag.

Edit: It is working, but the docs are still confusing and the 502 Bad Gateway Error when using a browser to access the endpoint is not helping either

[wimnat]

Here are my steps I took to get this working, pieced together from this issue.

The documentation says that grpc is not supported over Cloudflare Tunnels. As @joliveirinha points out, this is because grpc will not work when using the quic transport protocol. @joliveirinha mentioned official support might be coming in Q1 2023 which has now passed. If you have any further timeline update @joliveirinha, could you let us know?

In the mean time, you can get grpc to work by using http2. What's important to understand, and what tripped me up, is that you need to configure this in two places:

1. Between the edge and cloudflared
2. Between cloudflared and the origin server

For number 1 that means using --protocol http2 when starting the tunnel. I specify it before tunnel. I don't know if order makes a difference. So it looks like this:

--protocol http2 tunnel --no-autoupdate run --token my-long-token

When you start your tunnel you should see the http2 protocol displayed in the logs:

For number 2 you need to configure it alongside your destination endpoint. @joliveirinha mentions you need to do this client side because there is no option via the web portal. This is no longer correct - I do see an option in the portal now...

You do need to have configured your grpc endpoint to be using a certificate. If it is self-signed then ensure you enable the option "No TLS Verify".

If you're doing it on the CLI side then it looks like:

ingress:
  - hostname: <hostname>
    service: https://localhost:50051
    originRequest:
      noTLSVerify: true
      http2Origin: true

Once I had this all in place I was able to reach my grpc endpoint via the Cloudflare Tunnel using grpcurl.

This was all tested using cloudflared version 2023.7.0

I tested using a tunnel created via CLI and via portal and both methods work.

One final thing, if you get back an obscure error about a missing content-type header e.g.

rpc error: code = Unknown desc = failed to query for service descriptor "some.service": unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type

... it might be because you have an Access Policy set on your tunnel and here CF is trying to redirect you to your auth page.

There's no way to follow this redirect but if you can get your CF auth token then you can pass it as a header with your grpc request and it should work. E.g.

grpcurl -H "Cookie: CF_Authorization=my-long-token" ...

I hope someone finds this useful.

[JavierSegoviaCordoba]

@wimnat Hey! Where do you see that GUI? On the create tunnel Wizard I can't see any http2.

[wimnat]

@wimnat Hey! Where do you see that GUI? On the create tunnel Wizard I can't see any http2.

It's buried in the public hostname config for the tunnel.

Log in to the portal then go to Zero Trust > Access > Tunnels

Now you should have a list of all your tunnels. Find your tunnel and click the 3 dots on the right and choose configure. If you get a box asking you to migrate then this means your tunnel was configured via CLI so you can't change anything in the portal. Use the http2Origin: true setting locally instead. Otherwise you should be on the configuration page with 3 tabs. Choose the 'Public Hostname' tab. You'll see a list of configured public hostnames. Click the 3 dots on the right and choose edit. Now click 'Additional application settings' and then expand 'TLS'. This will give you the options you need.

[joliveirinha]

@wimnat Thank you for resuming the current state of this thread. I think that will really be helpful for others in the future.

Regarding the actual work to support this over QUIC. Yes, it is true that the initial plan was to do this in 2023Q1, unfortunately due to prioritization of other features this was something that we couldn't address. This is still a feature that personally I would like to have it done, but right now I am not in the position to commit with a date.

The good news is that the support over http2 transport unblocks users from actually having this gRPC capability. the bad news, is that if you want to use that Tunnel to also do UDP proxying in the WARP-2-Tunnel mode, you can't. You will need to setup another tunnel that would use QUIC. But that is the only limitation.

Again, thanks for resuming this. We enjoy seeing our community so engaged.

[seiftnesse]

Here are my steps I took to get this working, pieced together from this issue.

The documentation says that grpc is not supported over Cloudflare Tunnels. As @joliveirinha points out, this is because grpc will not work when using the quic transport protocol. @joliveirinha mentioned official support might be coming in Q1 2023 which has now passed. If you have any further timeline update @joliveirinha, could you let us know?

In the mean time, you can get grpc to work by using http2. What's important to understand, and what tripped me up, is that you need to configure this in two places:

1. Between the edge and cloudflared
2. Between cloudflared and the origin server

For number 1 that means using --protocol http2 when starting the tunnel. I specify it before tunnel. I don't know if order makes a difference. So it looks like this:

--protocol http2 tunnel --no-autoupdate run --token my-long-token

When you start your tunnel you should see the http2 protocol displayed in the logs:

For number 2 you need to configure it alongside your destination endpoint. @joliveirinha mentions you need to do this client side because there is no option via the web portal. This is no longer correct - I do see an option in the portal now...

You do need to have configured your grpc endpoint to be using a certificate. If it is self-signed then ensure you enable the option "No TLS Verify".

If you're doing it on the CLI side then it looks like:

ingress:
  - hostname: <hostname>
    service: https://localhost:50051
    originRequest:
      noTLSVerify: true
      http2Origin: true

Once I had this all in place I was able to reach my grpc endpoint via the Cloudflare Tunnel using grpcurl.

This was all tested using cloudflared version 2023.7.0

I tested using a tunnel created via CLI and via portal and both methods work.

One final thing, if you get back an obscure error about a missing content-type header e.g.

rpc error: code = Unknown desc = failed to query for service descriptor "some.service": unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type

... it might be because you have an Access Policy set on your tunnel and here CF is trying to redirect you to your auth page.

There's no way to follow this redirect but if you can get your CF auth token then you can pass it as a header with your grpc request and it should work. E.g.

grpcurl -H "Cookie: CF_Authorization=my-long-token" ...

I hope someone finds this useful.

I've been trying to make argo tunnel for a month, but the client can't connect to the server, I've tried with http & https and with certificate and without, all to no avail Error: 14: failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2606:4700:3030::6815:34b5%5D:443: Network is unreachable

run config: cloudflared tunnel --config config.yml --loglevel debug --protocol http2 run

config.yml:

tunnel: 3bea29f41-*****************************
credentials-file: /home/basic/.cloudflared/*****************************.json
ingress:
  - hostname: test.*********.org
    service: https://localhost:50051/
    originRequest:
      noTLSVerify: true
      http2Origin: true
  - service: http_status:404

If I remove- service: http_status:404, I get error 2023-07-30T17:08:34Z ERR Couldn't start tunnel error="https://localhost:50051/ is an invalid address, ingress rules don't support proxying to a different path on the origin service. The path will be the same as the eyeball request path"

[JavierSegoviaCordoba]

@joliveirinha will it be in 2023? I was unable to get it working with mentioned approaches

[wimnat]

@rlybasic have you tried removing the trailing slash / from service: https://localhost:50051/?

[seiftnesse]

@wimnat I tried it again now, didn't help, if I click on the domain in the browser, it comes up with invalid logs (but that was before.)

[joliveirinha]

@JavierSegoviaCordoba just to make sure. Did you configured your Zone to support gRPC as well? This is something that needs to be done even if you don't use Cloudflare Tunnel.

https://developers.cloudflare.com/support/network/understanding-cloudflare-grpc-support/#enable-grpc

[seiftnesse]

@JavierSegoviaCordoba just to make sure. Did you configured your Zone to support gRPC as well? This is something that needs to be done even if you don't use Cloudflare Tunnel.

https://developers.cloudflare.com/support/network/understanding-cloudflare-grpc-support/#enable-grpc

@joliveirinha I did everything the same way, still doesn't work

[wimnat]

@rlybasic can you try with grpcurl and specify the --insecure flag to ignore cert problems?

When I was initially trying with my code, I got things working with grpcurl then switched to my golang client and it didn't work because of similar complaints about http2. I used Wireshark to check what was happening on the wire and saw an http1 response back from Cloudflare. The fix for me was ensuring that I had added the system cert pool before making the request (x509.SystemCertPool()). I'm guessing here, but it might help.

[seiftnesse]

@rlybasic can you try with grpcurl and specify the --insecure flag to ignore cert problems?

When I was initially trying with my code, I got things working with grpcurl then switched to my golang client and it didn't work because of similar complaints about http2. I used Wireshark to check what was happening on the wire and saw an http1 response back from Cloudflare. The fix for me was ensuring that I had added the system cert pool before making the request (x509.SystemCertPool()). I'm guessing here, but it might help.

@wimnat I tried doing it with postman, but it didn't work either.

[seiftnesse]

@rlybasic can you try with grpcurl and specify the --insecure flag to ignore cert problems? When I was initially trying with my code, I got things working with grpcurl then switched to my golang client and it didn't work because of similar complaints about http2. I used Wireshark to check what was happening on the wire and saw an http1 response back from Cloudflare. The fix for me was ensuring that I had added the system cert pool before making the request (x509.SystemCertPool()). I'm guessing here, but it might help.

@wimnat I tried doing it with postman, but it didn't work either.

Problem solved, only other question is how to connect CloudFlare certificate on client and server if localhost is disallowed

2023-08-22T17:48:31Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: x509: certificate is valid for *.***********.org, ***********.org, not localhost" cfRay=7f********268f8-FRA event=1 ingressRule=0 originService=https://localhost:47691
2023-08-22T17:48:31Z ERR failed to serve incoming request error="Failed to proxy HTTP: Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: x509: certificate is valid for *.************.org, ***********.org, not localhost"

[schopra8]

Really appreciate rlybasic's guide -- but unfortunately, i'm still struggling to get this work. @joliveirinha -- would it be possible for someone at Cloudflare to write a short guide / blog on how to get this to work, to clear up the confusion on the current state of gRPC w/ Cloudflared?

Ideally, we'd like to use Cloudflare load balancer to expose a gRPC service on top of several Cloudflare tunnels to private resources.

[wimnat]

@schopra8 I don't think there is much more that would be offered beyond this https://github.com/cloudflare/cloudflared/issues/491#issuecomment-1643233485. When you say "clear up the confusion", can you specify what exactly you're confused about? What messages do you see in the logs?

[Omoeba]

Has there been any progress on supporting grpc over quic?

[janani-cr]

Hopefully this helps in clearing the confusion surrounding gRPC support over Cloudflare Tunnels, https://developers.cloudflare.com/cloudflare-one/faq/cloudflare-tunnels-faq/#does-cloudflare-tunnel-support-grp

[wimnat]

Hopefully this helps in clearing the confusion surrounding gRPC support over Cloudflare Tunnels, https://developers.cloudflare.com/cloudflare-one/faq/cloudflare-tunnels-faq/#does-cloudflare-tunnel-support-grp

It seems unlikely. The docs state:

Does Cloudflare Tunnel support gRPC? Yes. Cloudflare Tunnel supports gRPC for services within a private network. Public hostname deployments are not supported at this time.

But this whole thread is all about getting them working (successfully) with public hostname deployments. So I think more than anything, it's probably just going to confuse users some more.

[abasu0713]

Okay! Finally after 3 days of excruciating toil - got it working. Couldn't have done it without this entire thread! So thank you to you all. I am writing a medium blog for it with all the steps in Github Gists. I will link it here once I am done. But for note to Cloudflare team - please update the docs. The docs are largely confusing. Since Cloudflare Tunnel supports both grpc and grpc-web using Envoy Proxy with h2 alpn protocol

[JavierSegoviaCordoba]

Okay! Finally after 3 days of excruciating toil - got it working. Couldn't have done it without this entire thread! So thank you to you all.

I am writing a medium blog for it with all the steps in Github Gists. I will link it here once I am done.

But for note to Cloudflare team - please update the docs. The docs are largely confusing. Since Cloudflare Tunnel supports both grpc and grpc-web using Envoy Proxy with h2 alpn protocol

I am interested in that medium post :)

[abasu0713]

Okay! Finally after 3 days of excruciating toil - got it working. Couldn't have done it without this entire thread! So thank you to you all. I am writing a medium blog for it with all the steps in Github Gists. I will link it here once I am done. But for note to Cloudflare team - please update the docs. The docs are largely confusing. Since Cloudflare Tunnel supports both grpc and grpc-web using Envoy Proxy with h2 alpn protocol

I am interested in that medium post :)

It's in the works. I am covering a lot in it. Like:

• EnvoyProxy on Arm64,
• Kubernetes Deployment with A/B testing,
• Managed certs for Envoy using cert-manager so Cloudflare's TLS termination can work and the origin server's own managed Certs,
• A MongoDB backend with community operator on ARM64 for a NoSQL Backend with different levels of access for different services within your backend service mesh.
• gRPC-JSON Transcoding to support: gRPC, gRPC-web and RestFul API all at the envoy level
• Container Native load-balancing
• And a HA storage backend using Ceph for the stateful workloads.

I thought I would be done by now. But I am still at like 30 percent. Will shoot for next weekend. Just an FYI