Heroku Common Runtime Router 2.0 - Beta Release (Pilot)

[elimchaysengSF]

The Beta release for the new Heroku common runtime router is to give an early opportunity to customers to experiment, provide feedback, and shape the future networking product for Heroku.

This new router is specific to customers using the following Heroku Dyno types: Eco, Basic, Standard, and Performance. This feature does not apply to our enterprise offerings, specifically Private or Shield Type Dynos.

As the development of the new router continues, we will continually add features to reach parity with our current production router, and we will also begin the rollout of new networking features listed here on the public roadmap.

[amerine]

Congrats!! Huge improvements for customers ahead!! 🚀🚀

[elimchaysengSF]

Released and now available to the public! Dev Center and Changelog. Stay tuned for the upcoming engineering blog post.

[elimchaysengSF]

Blog Post for more details about this release! https://blog.heroku.com/router-2dot0-the-road-to-beta

[cwatson272]

Are there any new releases planned? I'm waiting impatiently on HTTP/2.

[csuhta]

We enabled the beta on a few higher volume apps, but all we got was a flood of H99 errors, so I turned it off quite quickly. No idea how to help diagnose anything, H99 errors are rather opaque.

[elimchaysengSF]

@csuhta - thanks for raising this, our networking team is interested in learning more if you don't mind opening a support ticket so we can see if we can get you upgraded properly. Feel free to tag me in the support ticket as well.

[holtkamp]

We enabled the beta on a few higher volume apps, but all we got was a flood of H99 errors, so I turned it off quite quickly. No idea how to help diagnose anything, H99 errors are rather opaque.

Experienced the same.

@csuhta did you contact Heroku support and was the cause of the issue identified?

[elimchaysengSF]

after @csuhta's comment we've added some additional logging on our side to investigate this deeper, but it would help to have a ticket open that we could use to track the investigation

[holtkamp]

@elimchaysengSF I opened a ticket at Heroku Support: "#1392455 - [Heroku Common Runtime Router 2.0] Apps crash when enabling router 2.0" 👍

[csuhta]

Ticket #1392622 is open with our H99 log

[csuhta]

Another bit of general feedback:

• We have a web app that has moderate-to-high worldwide attention and lots of mobile users.
• On router 1.0, our Heroku metrics had a constant stream of H27 errors, which we cannot do anything about
• On router 2.0, this has changed to steady stream of H28 errors and a smattering of H27 errors. The H28 errors are also orange for some reason on the metrics chart
• All of these errors are not useful to us, we don't really want to see them, we can't do anything about them. As far as I know they don't indicate any sort of problem with our service other than we're seeing lots of clients on poor internet

Please let us hide these errors on router 2.0, or at minimum make the H28 error grey on the Heroku metrics charts. All of this is just noise that we can't resolve or filter out.

[elimchaysengSF]

Hi everyone! @slizco and our entire networking team has been hard at work addressing feedback and looking at improvements to Router 2.0 as we move through the ongoing beta period and work to get the router in it's final GA state. I've worked with Liz to pull together some info and give an update on the items we've addressed and recent improvements we’ve made.

We’ve received lots of great feedback on the beta, especially since releasing HTTP/2 on the new router. With our customers’ help, we’ve discovered a few bugs and some performance issues in specific use-cases. We will continue addressing these, but here’s a non-exhaustive list of what we’ve accomplished thus far:

• Some customers with Preboot enabled saw H99 errors when using Router 2.0. This was due to a bug in the router’s dyno state management. This has been fixed, and reports of this issue have dropped to zero. For apps receiving traffic on persistent client connections, routing back to the legacy router was delayed upon flagging out of Router 2.0. This is common for CDN or reverse proxy set-ups that front the Heroku router. This bug has been fixed by instituting graceful server-side connection closes once Router 2.0 has been disabled on an app.

• Many customers reported H99 errors in logs, but no actual adverse application behavior. It turns out that these errors were actually H27: Client Request Interrupted. This logging bug has been fixed.

• Customers reported data missing from the fwd and host router log fields in certain situations. This has been fixed.

• Some customers have been reporting higher tail request latencies. In response, we have run performance analyses and identified a few bottlenecks in the router. We’ve already pushed out a couple code changes to address these chokepoints and have more staged or in progress.

• We’re also collecting more metrics to better quantify performance improvements. We believe Router 2.0 performs just as well as the legacy router for the vast majority of requests, but we do recognize there are certain client/server set-ups for which the p95 on request latencies is higher on the new router. We are committed to understanding and improving this experience, so that Router 2.0 performs better than the legacy router for all our customers.

Furthermore, we continue to investigate select customer issues through Support tickets. We encourage our customers to open Support tickets any time they encounter a Router 2.0 or HTTP/2 issue.

Finally, a HUGE thank you to everyone who has taken the time to try out Router 2.0. You are helping us develop a better router for the entire Heroku community, and we deeply appreciate the role you play in helping us get to GA. 💜