🐛 http2Origin option and --http2-origin

[coltstrgj]

Describe the bug @nikr-canva I added the flag mentioned here to my docker container (version 2022.6.3). In the log Settings: map[http2-origin:true loglevel:debug no-autoupdate:true token:*****] which means the flag is applied. I also see GET https://zitadel.mydomain.com/ui/console/assets/environment.json HTTP/1.1 which is weird considering I was expecting HTTP/2. I'm assuming that I set something up incorrectly but I'm not even sure where to begin.

I have checked https://mydomain.com/cdn-cgi/trace and it shows http=http/2

To Reproduce Not sure, unfortunately. All I did was launch the docker container, add a public hostname in the UI.

Expected behavior Should be HTTP/2

Environment and versions

• OS: alpine linux host, container is "cloudflare/cloudflared:latest" as of today.
• Architecture: AMD64
• Version: 2022.6.3

Logs and errors logs attached cloudflared.log

[nmldiegues]

We confirm that something is not working as intended with that flag.

@sudarshan-reddy is looking into it best-effort and will provide feedback here.

[fforootd]

Hi there I work for @ZITADEL

I can confirm that I find the same behaviour with Cloudflare Tunnel. Do whatever reason it looks like the origin calls are only made with HTTP/1.1.

While this works to some degree, we normally utilise h2c or HTTP/2 on the origin side to allow to mix "normal" http traffic with our gRPC APIs.

Also while digging through the issues in this repo I found this one here https://github.com/cloudflare/cloudflared/issues/625#issuecomment-1167236921 and think that's the cause for this.

@coltstrgj out of curiosity and to confirm my suspicion. I think you used no TLS on the ZITADEL side, right? At least that was what I did while recreating the error. Haven't had time to confirm the HTTP/2 (with TLS) yet.

[coltstrgj]

I have tried both with and without tls. Without TLS I saw HTTP/1.1. With TLS I had to set the "No TLS Verify" option but still saw HTTP/1.1

[nikr-canva]

I think there are a couple of issues being mixed up here

• origin requests not being HTTP/2 when they are expected to be
• grpc not working.

The first one sounds like a bug. For the second point, I think there are still various issues with HTTP trailers in the cloudflared implementation so that may not work, regardless of this fix. (I remember reading about this from cloudflare somewhere, but can't find it now)

[sudarshan-reddy]

That’s accurate. FWIW, the goal is to fix both 1 and 2 because grpc naturally needs an HTTP/2 connection to the origin.

[coltstrgj]

Sorry, I'm usually better at writing these. I noticed the issue because of grpc issues. Later I found this https://support.cloudflare.com/hc/en-us/articles/360050483011-Understanding-Cloudflare-gRPC-support

So I don't expect grpc to be fixed by this bug because it's an enhancement and unrelated (though it could stand to be documented in more than one spot). HTTP/2 not working is a real issue though.

---

It's been a while, I haven't had time to dig in to the code, and I'm only a novice go dev but when I looked a few weeks ago I didn't notice the value actually being used anywhere. If I recall it was set in a struct but not pulled out in any of the code that initializes a connection. I wonder if maybe a file got missed when checking in the changes. Obviously I'm no expert and you would probably have thought of that by now but it's an easy thing to check if theres a surviving stash or like me you have 30 branches to save various design changes.

[obezuk]

Hi folks,

Quick update on how to use HTTP2 between cloudflared and the origin:

If your Tunnel is managed via the Cloudflare Dashboard, the HTTP2 flag in "Additional application settings --> TLS --> HTTP2 Connection" has priority over CLI --http2-origin flag set in cloudflared. Ensure HTTP2 is enabled in the Public Hostname settings if your application is defined here.

cloudflared's debugging logs continue to read HTTP/1.1. This is somewhat misleading but has to do with how we internally handle requests between Cloudflare and cloudflared. You can observe your origin server's access logs to confirm it is receiving HTTP/2.0 from cloudflared.

[bithavoc]

@obezuk is this still not working with Cloudflare Tunnels and Zitadel?

This config still gives me context timeouts and fails to redirect after auth:

    - hostname: {{ .Values.zitadel.externalDomain }}
      service: http://{{ .Values.zitadel.serviceName }}:{{ .Values.zitadel.servicePort }}
      originRequest:
        http2Origin: true

[tankerkiller125]

I managed to get Zitadel working with Cloudflare Tunnels just fine with a little bit of work.

Most notably, enabling TLS on the Zitadel container with a self-hosted SSL cert, and then setting the Cloudflare Tunnels to ignore the certificate error and force HTTP2 worked fine. But I haven't been able to test GRPC, so that might still be an issue.

[fforootd]

I managed to get Zitadel working with Cloudflare Tunnels just fine with a little bit of work.

Most notably, enabling TLS on the Zitadel container with a self-hosted SSL cert, and then setting the Cloudflare Tunnels to ignore the certificate error and force HTTP2 worked fine. But I haven't been able to test GRPC, so that might still be an issue.

You could use grpcurl to test if the connection works. If it works with TLS but not without TLS it might be down to an H2C problem (zitadel supports this, unsure about cf tunnel)

[bithavoc]

My Zitadel is working as well, it was an unrelated issue.

[tankerkiller125]

I managed to test GRPC today, and it worked perfectly. However I can only test it with TLS at the moment, I couldn't test without TLS, might have to spin up another instance to test that.

As a note though, I was able to switch to a Cloudflare Origin Certificate, which means that I don't have to ignore SSL validation on the tunnel side.

[abasu0713]

I managed to test GRPC today, and it worked perfectly. However I can only test it with TLS at the moment, I couldn't test without TLS, might have to spin up another instance to test that.

As a note though, I was able to switch to a Cloudflare Origin Certificate, which means that I don't have to ignore SSL validation on the tunnel side.

This is probably the only comment on internet on this topic that gives me some hope. I have been kinda stuck on this for the past couple days! @tankerkiller125 would you be able to give me a rough idea if my assumption is correct? Since you have gRPC with TLS working? From what I understand, for gRPC to work over HTTP2 (no matter the mechanism - tunnels or load balancers) - traffic from Cloudflare network to the Origin servers needs to have TLS, right?

Cause I have a simple Envoy Proxy (not as a sidecar) setup with Cross-CORS config (not production - of course) for a simple gRPC server that is JSON-Transcoded that is running in a Kubernetes Cluster. It's not TLS enabled. I am able to get REST Responses just fine using CURL (see below)

alphaduriendur@Arkos-MacBook-Pro portfolio % curl https://<>/medium
{
 "blogUrls": [
  "https://medium.com/@arko.basu09/cloud-agnostic-s3-buckets-using-ceph-7e072de84672",
  "https://medium.com/@arko.basu09/keycloak-the-crown-jewel-of-open-source-identity-management-156bb012dc15",
  "https://medium.com/@arko.basu09/automate-build-of-containerized-flutter-web-apps-aac1ac70ed2e",
  "https://medium.com/@arko.basu09/self-host-a-gen-ai-powered-wordpress-site-using-kubernetes-on-any-commodity-hardware-10b29c53fb2d",
  "https://medium.com/@arko.basu09/inexpensive-diy-nas-vs-code-server-on-arm64-sbcs-feda441a40e1",
  "https://medium.com/@arko.basu09/compile-linux-image-for-full-ceph-support-on-orangepis-0fa584aa1aca",
  "https://medium.com/@arko.basu09/docker-run-llama-2-models-on-an-orangepi-5b-using-llama-cpp-e024996d7c4c",
  "https://medium.com/@arko.basu09/diy-data-lake-using-orangepi-3b-and-ceph-4aaad736b352"
 ]
}
alphaduriendur@Arkos-MacBook-Pro portfolio

But the GRPC calls get force terminated.

I am good. Got it working. Thanks