Support HTTP/2

[justzeros]

Support HTTP/2 for both Common Runtime and Private Spaces platforms.

[hrdwdmrbl]

and HTTP/3?

[friism]

(Heroku PM here) Yeah, we're keen to do this. HTTP/3 is going to be a little more challenging since it's not TCP.

[swifthand]

Very excited to hear that this will eventually come! Have been wishing and waiting for HTTP/2 for a while (and didn't even know about this git repo until searching for an update on the topic).

[val-kando]

Jan 29, 2023 Any updates guys?

[sja87]

Update?

[madureira]

Any news on HTTP/2 support?

[friism]

Hey, we're still working towards the goal of supporting HTTP/2 on Heroku, but it requires quite a lot of changes. Please bear with us.

In the interim I have a clarifying question:

• Are folks looking for HTTP/2 from end-user client to Heroku, with Heroku terminating HTTP/2 and forwarding HTTP/1.1 to the dyno? (This would mean that Heroku apps could "support" HTTP/2 with no or few changes, and even if using a language with limited HTTP/2 support)
• Or are folks looking for HTTP/2 all the way to the dyno? (this would enable things like GRPC servers in Heroku apps, and building apps that make use of all the features of HTTP/2)

HTTP/2 to Heroku with HTTP/1.1 forwarded to dyno is our default scenario, but we'd love to better understand customer use cases, and whether we should prioritize differently or to make both options happen.

[emilsedgh]

In our case, our API servers are hosted on Heroku and browsers directly make calls to them. The reason we want HTTP/2 support is that browsers limit the number of requests sent to a given origin [0] which means API calls would stall, thus impacting the performance on our app.

So the default scenario you mentioned does resolve our use case.

[0] https://developer.chrome.com/docs/devtools/network/reference/?utm_source=devtools#timing-explanation

[merwok]

IMO option 1 is an invisible infrastructure improvement: heroku would speak http 2 with clients that support it, improving performance with no effort on my part – so something nice but not major. But option 2 would be a real new feature for the platform, supporting new kinds of applications, which would be a welcome thing and raise people’s hopes that heroku can gain new features and stay interesting.

[incorvia]

We are looking for option 2, so we can enable GRPC. We are exploring options to migration from Heroku due to the lack of GRPC support so something like this would be very welcome.

[stefanosalvucci]

Hello, any updates @friism ?

[bh8ur8js]

I second @incorvia

It’s long overdue.

[swifthand]

@friism We are exploring features which will require gRPC. Comparable platforms appear to offer it, and if one of those features reaches a certain priority level in our roadmap, we will begin to migrate off of Heroku in order to achieve them.

[rotsee]

Option 1 (default) alone would be a great thing in itself. I'd rather have than pretty soon, than waiting for something more ambitious

[karlbecker]

Just want to second that Option 1 itself would be a great improvement.

[jtosey]

Is it fair to say that no developer resources have yet been assigned to this on Heroku's side? The above responses suggest that this is at most a discussion item, with no schedule commitments, at least in 2023.

[elimchaysengSF]

Hey all - PM for the Heroku Runtime here that works with @friism ^

First of all - our team totally understands the priority on this one, and it's the number one networking feature that both I and our networking team want to deliver to our customers since it is so long overdue. This is a sentiment I feel very strongly about and I acknowledge the strong interest from all sides in seeing this finally come to the Heroku platform.

To give some transparency on why it hasn't moved out of the research column and to answer your question @jtosey, why it has not been specifically assigned dev resources yet is because of a current (and pretty massive) network infrastructure re-platforming project.

We decided late last year to take the time to invest in revamping our networking architecture for both our shared runtime and private spaces to set us up for future success and expedite future feature delivery. We finally had some room to refactor a lot of our networking services, and we chose to prioritize that for the majority of this year alongside other under-the-hood changes to improve security and reliability.

I can't give a specific timeline yet, but the entire networking team has been knocking out phases of the project at a consistent pace. We're looking to have something customers will be able to see by the end of the year at the latest, and we have Option 1 of the HTTP/2 implementation that's been mentioned above as top of mind as we hit more milestones going forward. In short, it's the first thing we want to prioritize as soon as we get to a feature development point in the project.

Happy to answer any more questions you might have and I'll update here as soon as I have more details to share about the HTTP/2 feature development.

[yangcheng]

hi @elimchaysengSF I want to add a use case for http2, that is server send events SSE mozzila doc called out a big limitation

Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6) .

many LLM APIs are using SSE to provide a streaming experience, like OpenAI and Anthropic this makes support for HTTP2 even more urgent as many are try to add similar LLM APIs into product

[cedricschwyter]

We are looking for option 2, so we can enable GRPC. We are exploring options to migration from Heroku due to the lack of GRPC support so something like this would be very welcome.

absolutely, this is becoming a real deal breaker - we need that gRPC support :)

[brettgoulder]

@friism @elimchaysengSF I need http/2 for its increased open connection support. We cannot really use EventSource/SSE without it, due to browser connection limits. http 1.1 supports a maximum of 6 connections per browser, while http/2 supports 100.

see:

• https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#sect1
• https://textslashplain.com/2019/12/04/the-pitfalls-of-eventsource-over-http-1-1/

[friism]

Progress: the beta for the new Common Runtime router is out now. It doesn't have HTTP/2 just yet but we're working on it: https://devcenter.heroku.com/articles/heroku-runtime-router-2-0

[jtosey]

BTW is it written in Go? Bard is convinced that it is.









Ha ha ha ...

On Sep 28, 2023, at 1:56 PM, Michael Friis @.***> wrote:

Progress: the beta for the new Common Runtime router is out now. It doesn't have HTTP/2 just yet but we're working on it: https://devcenter.heroku.com/articles/heroku-runtime-router-2-0

— Reply to this email directly, view it on GitHub https://github.com/heroku/roadmap/issues/34#issuecomment-1739992565, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACQTBSHEP56RMQF7EHE6KLX4XP6VANCNFSM57LSXX5Q. You are receiving this because you were mentioned.

[jesusalatorre]

Any updates on this?

[brettgoulder]

@friism happy to see progress. It's a good thing your brillant ex-Runtime PM released SNI TLS/certificate management to make the migration to http/2 easy.

[mdahlke]

@elimchaysengSF Any updates on this?

[elimchaysengSF]

Apologies for the radio silence - we’ve been heads down working to get this going for both the Common Runtime (with the new Router) and Private Spaces. It’s still our runtime networking team’s number one customer-facing priority, but we encountered some platform work this past quarter that moved the launch date into the new year. We are very close to bringing the first phase of functionality (HTTP/2 terminated at the router) to the public beta of our router and as an app feature for Private Spaces. I will post as many updates as I can share as we get closer to the full launch, but keep an eye on the changelog.

Due to feedback here and other customer comms, we’ve been investigating bringing in h2c between the dyno and router to ideally enable gRPC use cases prior to a full HTTP/2 end-to-end implementation. This would mean your app would need to be able to handle requests in HTTP/2 cleartext (h2c) format because TLS is still terminated at our router. If anyone has feedback on the approach, I’d love to learn more here.

[Bismarck-GM]

Is there any news about HTTP/2 support?

[qstiegler]

Please keep us updated about the progress for HTTP/2. Will it be released soon?

[elimchaysengSF]

Excited to FINALLY move this into the Coming Soon column, please watch our blog and changelog in the coming weeks for information on how to enable the first implementation (first bullet) mentioned above for handling HTTP/2 on Heroku.

• Are folks looking for HTTP/2 from end-user client to Heroku, with Heroku terminating HTTP/2 and forwarding HTTP/1.1 to the dyno? (This would mean that Heroku apps could "support" HTTP/2 with no or few changes, and even if using a language with limited HTTP/2 support)

[Chris-Beckett]

Excited to FINALLY move this into the Coming Soon column, please watch our blog and changelog in the coming weeks for information on how to enable the first implementation (first bullet) mentioned above for handling HTTP/2 on Heroku.

• Are folks looking for HTTP/2 from end-user client to Heroku, with Heroku terminating HTTP/2 and forwarding HTTP/1.1 to the dyno? (This would mean that Heroku apps could "support" HTTP/2 with no or few changes, and even if using a language with limited HTTP/2 support)

Do we have a planned date of when this "Coming soon" is?

[capeterson]

@Chris-Beckett - that's going to depend on which variant of what Ethan posted that you're looking for. HTTP2 to the dyno** so you can do something like validate mTLS client certs in your app is further out. If you're looking for this urgently for something other than TLS client cert validation in your app logic we'd love to hear more about the use case.

The beta for the Heroku router supporting HTTP/2 termination and passing HTTP/1.1 to the dyno is the part that's going to be announced in the next two weeks, barring any major surprises. That will require enabling a labs flag for your app to enable during the beta period and have a constraint that it only works with custom domains (not the *.herokuapp.com URL), but otherwise should let you gain the majority of HTTP/2 benefits without needing to make any changes to your app.

This is a pretty major change to our routing layer so once you get hands on we'd love to hear your impressions.

[Chris-Beckett]

@capeterson - Our user case is more about the recommendation from pagespeed tools. Use HTTP/2 or higher. "Use a newer version of the HTTP protocol to make multiple concurrent requests on the same connections and to better prioritize resources."

[capeterson]

Perfect, you should be covered then. Hang on just a little bit longer and we'll update when the beta and blog post with more details are live!

[brettgoulder]

@capeterson can you add the feature flag to our app? (ex-Heroku Runtime PM here, happy to give you tons of feedback.)

[cseelus]

In the interim I have a clarifying question:

• Are folks looking for HTTP/2 from end-user client to Heroku, with Heroku terminating HTTP/2 and forwarding HTTP/1.1 to the dyno? (This would mean that Heroku apps could "support" HTTP/2 with no or few changes, and even if using a language with limited HTTP/2 support)
• Or are folks looking for HTTP/2 all the way to the dyno? (this would enable things like GRPC servers in Heroku apps, and building apps that make use of all the features of HTTP/2)

Definitely variant 2. A lot of good reasons were already stated here. Additionally, it's really a must have for Rails apps these days, see Thruster by 37Signals.

[searls]

Excited to FINALLY move this into the Coming Soon column, please watch our blog and changelog in the coming weeks for information on how to enable the first implementation (first bullet) mentioned above for handling HTTP/2 on Heroku.

• Are folks looking for HTTP/2 from end-user client to Heroku, with Heroku terminating HTTP/2 and forwarding HTTP/1.1 to the dyno? (This would mean that Heroku apps could "support" HTTP/2 with no or few changes, and even if using a language with limited HTTP/2 support)

Excited to hear this! Was just about to reach out before thinking to check to see if there was a roadmap issue with recent updates. Best wishes in getting the feature out the door!

[elimchaysengSF]

Excited to announce HTTP/2 on Heroku! Check out the blog post and/or the changelog to opt-in now!! 🥳 🚀

Changelog Blog Post

[brettgoulder]

Grats! Any plans for http/2 all the way to the dyno? I want this for server push.

I realize option 1 is the best way to not break any existing apps. I contended with this problem when I was PMing the Runtime team.

[brettgoulder]

Oh nevermind, see you added the issue just now ;)

[elimchaysengSF]

I've opened a new issue to track enabling HTTP/2 end-to-end on Heroku as mentioned in the blog post above: https://github.com/heroku/roadmap/issues/288

I've ported the feedback from the threads in this ticket there, and will focus the ongoing conversation in the new ticket, and closing this one out!

[cedricschwyter]

congrats, and thank you for the valuable work!

On 5/7/24 9:35 PM, elimchaysengSF wrote:

Excited to announce HTTP/2 on Heroku! Check out the blog post and/or the changelog to opt-in now!!

Changelog https://devcenter.heroku.com/changelog-items/2872 Blog Post https://blog.heroku.com/heroku-http2-public-beta

— Reply to this email directly, view it on GitHub https://github.com/heroku/roadmap/issues/34#issuecomment-2099159805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AISLPXHNSQKTSBG7QBRYJ5TZBEUHVAVCNFSM57LSXX52U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBZHEYTKOJYGA2Q. You are receiving this because you commented.Message ID: @.***>

[stefanosalvucci]

Hello, we've just tested the new feature through

• heroku labs:enable http-routing-2-dot-0 -a on our frontend website

here's what happen

• we have enable http2 on our frontend website using a wildcard sectigo certificate
• appartently everything works fine (website is reached all over the world, http2 protocol successfully detected on logs, etc)
• we started seeing randomly rows on our logs like the following

these requests causes an code=H32 desc="TLS certificate mismatch" errors, BUT the main issue is that those request should be received by another app on heroku (the backend)

basically enabling http2 on our frontend website causes that some request to our backend (a few of them) were redirected to the frontend website.

The frontend website and the backend share the same domain, but they use 2 different subdomains. Hope this can help, happy to help.

[ypaq]

@stefanosalvucci Thank you for your report! Would you mind creating a support ticket here?

[stefanosalvucci]

sure, will open now

[stefanosalvucci]

following the reply from heroku team so everyone is updated

Hi Stefano,

This is Tilman from the Heroku Runtime Networking team. Thank you for trying out Router 2.0 and reporting this issue. In fact, you have identified a bug in our logging. The host field in your report contains the HTTP Host header of the incoming request. It should contain the server name indicator (SNI) from the client hello message of the TLS handshake. We are already working on fixing this issue and will notify you once it's rolled out. I believe this clarifies your concern that the router attempted to route the request to the wrong app.

The new router validates the match of the HTTP host header with the SNI from the TLS client hello message for every request on a TLS connection to a custom domain. The old router would do this only for the first request. The reason why you see an H32 might be that your client reuses a TLS connection for a request with a Host header intended for another app

[sfcgeorge]

I'm not an expert testing this stuff but have some potential feedback (service seems slower and more variable)

• We use CloudFlare so already have HTTP/3 from browser to CF.
• Tested enabling/disabling router 2.0 to have HTTP/2 between CF and Heroku, not expecting much difference as CF doesn't support connection multiplexing to the origin yet, only header compression I guess.
• Used a blank page for testing to minimize server variability (no DB calls etc).
• I used this simple script to test TTFB. Even doing 300 reqs with router 2.0 enabled vs disabled results were inconclusive, likely due to my home network and everything in between. https://github.com/jaygooby/ttfb.sh
• That said looking at the logs and skimming the router lines across the hundreds of requests I saw clearer patterns:
  • connect was about the same for new vs old router at 0ms to 1ms.
  • service was the same or worse with new router 😔 Best case for both was 3ms. Old router's worst case was 7ms, new router's worst case was like 65ms 😬
• bytes was indeed smaller with new router; 20 vs 983 😁

So am I understanding right that service means time spent in router, and that seems worse? Could this be Erlang vs Go differences aka GC needs tuning? Or something else entirely?

[dylanz]

@elimchaysengSF it's possibly not related to this release but we've been had HTTP/2 running for 2 weeks now without any issues with the exception of yesterday. We're being hit with what looks to be a DDoS attack at almost 70k req/s. We started receiving "Backlog too deep" errors and the requests stopped hitting our application (were we have an application based WAF that is in place to block certain traffic patterns). We turned off HTTP/2 for now in order to lessen the amount of unknowns. We've been on Heroku for over 15 years and have never run into this issue before so just wanted to check in to see if there is any possible correlation between this issue and this release. Thank you!

[elimchaysengSF]

Hey all, just x-posting for some visibility about ongoing improvements and investigations for Router 2.0 due to our large adoption increase since launching HTTP/2.

You can see all the details over in the Router 2.0 Beta Roadmap item.