WebSockets don't work on Apache

[FooBarWidget]

If the response has the chunked transfer encoding, then WebSockets handshakes become corrupted. See https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/phusion-passenger/XtjxA8E1idw/Z029OysamjsJ for the report.

[FooBarWidget]

See also https://github.com/phusion/passenger-nodejs-websocket-demo/issues/2

[FooBarWidget]

The chunked encoding problem can be solved by setting Content-Length: 0 in the response, which will prevent Apache from setting Transfer-Encoding.

// Process WebSocket upgrade.
Header upgrade = lookupHeader(headerData, "Upgrade", "upgrade");
if (!upgrade.empty()) {
    RH_TRACE(client, 2, "WebSocket upgrade detected.");
    headerData.append("Content-Length: 0\r\n");
}

However, it looks like the problem is more complicated than just solving this. The current Apache module currently does not support simultaneously forwarding request data and response data, so WebSockets still doesn't work. Worse, instead of triggering an error, the connection just freezes.

Fixing this is non-trivial. I'm postponing it to a later release.

[captn3m0]

:+1: Waiting for this eagerly.

[kiranos]

+1

[kiranos]

any news on this?

[FooBarWidget]

Unfortunately not. If you need WebSockets, for the time being we recommend using Nginx, or using Passenger Standalone behind Apache + mod_proxy.

[vanboom]

See https://github.com/phusion/passenger-ruby-websocket-demo/issues/2, appears WebSockets are still not working under Apache mod_passenger in 5.0.18? (It's very possible that I am doing something wrong in my Apache .conf) Thanks for Passenger - it's incredible!

[FooBarWidget]

@vanboom That is correct, WebSockets do not work yet in the Apache version. For the time being, we recommend that you use the Nginx version, or Apache + Standalone in a reverse proxy setup, if you need WebSockets.

[ThaddeusAid]

It has been almost a year since the last update. Did the upgrade to apache 2.4.x improve this?

[CamJN]

There is a websocket proxy module now for Apache 2.4, however we have not found it to be robust enough to recommend yet.

[mk-pmb]

Just discovered passenger, sounds like a gift from heaven. :+1: In the project I want to use to explore passenger, I can set specific ports (or env var names to read the ports from) for WebSockets and EventSource, will that help? It's meant as a work-around to use with Apache's mod_proxy, but maybe passenger can intercept to sanitize and monitor stuff.

[mk-pmb]

Just for the record, some people seem to have managed to use websockets in apache with wstunnel.

[CamJN]

Yes, that's the websocket proxy module I mentioned, however it only works for some people. We'll consider recommending it when it is more robust.

[jamgregory]

@CamJN - out of interest, what issues did you have with the websocket module in Apache? I presume there's some work involved on the Passenger side to integrate (if possible) with that so ActionCable will work through Passenger directly. We're looking at a future development project that will require ActionCable and ideally we'd like Passenger to support this if possible.

[CamJN]

As one example since this module is just a websocket-aware proxy, you need proxy support in your websocket library so it won't just work for everyone.

[mk-pmb]

you need proxy support in your websocket library

What features will that consist of? Can I test all those features on a per-app level, independent of which ws lib it uses? Would it help to have a shims option where I can configure what workarounds to use until some feature is available?

One feature I might imagine as "proxy support" would be about determining the IP from which a connection arrives. A work-around might be to specify the network interface from which the proxy should send its packets, to ensure it uses an interface (and thus IP address) that my statistics can identify as "not a real visitor" and which the app doesn't assume as privileged in any way. The setting might have an optional "allow localhost" flag which determines whether to fail loudly in case the specified interface will be undistinguishable from localhost.

[CamJN]

What features will that consist of?

Generating urls for links and js that refer to the proxy's url/port not the app's.

Can I test all those features on a per-app level, independent of which ws lib it uses?

No, because they have to come from the ws lib.

Would it help to have a shims option where I can configure what workarounds to use until some feature is available?

Yes, but I don't know if such a thing exists.

[mk-pmb]

I'll share my idea since I couldn't finish my research on it: A lot of web apps have their websockets behind very few, short URL prefixes like /ws/ (http://api.example.com/ws/chat). If we can make the browser and potential proxies send the websocket request as the first request in the TCP connection (a randomly selected subdomain should suffice), a netfilter string match or l7-filter should be able to pick up on the request URL and NAT it to bypass Apache. (Please make double sure to avoid false positive matches.)

For HTTPS, we could hope that browsers that support websockets will also use Server Name Indication (SNI), so netfilter might be able to detect a hostname that an app can use exclusively for its websockets, and then redirect that port 443 connection to the app directly, or an auxiliary web server, again bypassing Apache. Update: SNI hostname transmission happens too late (usually not in the first packet) for easy methods of port redirection.

[blevine]

Anyone having any luck with this? mod_wstunnel has been around for a while now. We've had no problem getting websockets to work over an Apache reverse proxy using mod_wstunnel. However, still not sure how to get this to work using Passenger in Apache integration mode.

[heangratha]

Any one have luck to get websocket work with Apache? This is my Apache configure get from: https://www.phusionpassenger.com/library/walkthroughs/deploy/meteor/ownserver/apache/oss/xenial/deploy_app.html#edit-apache-configuration-file

But the websocket won't work.. I'm using Ubuntu 16.04, Apache 2.4.18, Phusion Passenger 5.2.0

[jamgregory]

@heangratha - the only solution we could come up with was to run a separate instance of the app in Puma, then proxy all WebSocket requests to that using mod_wstunnel.

[thomasbalsloev]

Any movement on this issue?

[CamJN]

Not right now, it is unfortunately not a priority at the moment. People are recommended to use nginx if they need websockets, as it's still a pain with Apache.

[thomasbalsloev]

@CamJN

OK, I see. Thank you for the update.

[mk-pmb]

Update on my research (it's on the back burner though): We could probably write a module like fdpass to get the socket descriptors. We'd then have to exfiltrate the connection's SSL context (ideally: compile Apache with an SSL lib that officially supports this; worst case: dump worker's process memory, then recover forensically) and send that along with the connection socket file descriptor to a server that can receive both and continue the SSL session. The metasploit codebase probably has useful parts to reuse here.

Generating urls for links and js that refer to the proxy's url/port not the app's.

I'm optimistic that URL rewriting might be an option for people with a bit of backgorund knowledge about the app – which the app's admin should have.

[siegy22]

Is this a wontfix?

[CamJN]

@siegy22 not exactly, but it's blocked on the Apache project improving their module.

[annymosse]

@vanboom That is correct, WebSockets do not work yet in the Apache version. For the time being, we recommend that you use the Nginx version, or Apache + Standalone in a reverse proxy setup, if you need WebSockets.

@FooBarWidget how can i do it (Apache + Standalone in a reverse proxy setup)?"still newer in servers configs"

[mk-pmb]

@annymosse How to install Passenger Standalone Once you have that running, it gives a web server. You'll want to ensure it is not reachable directly from the outside. To achieve this, you can use explicit interface binding (to a non-public interface) and/or firewall rules. The Apache Reverse Proxy Guide shows how to make a publicly reachable apache forward traffic for some parts of your URL space, to the guarded private Passenger server.

[beppe9000]

Does this apply to cpanel ?

[deltafactory]

@beppe9000 Yes, since cPanel's default web server is Apache.

[mjobin-mdsol]

@CamJN

not exactly, but it's blocked on the Apache project improving their module.

is there an open issue on Apache's side we can track?

[mjones-vsat]

Any updates on this?

[mk-pmb]

Sorry for the wrong info earlier. Seems like apache itself still supports prefork worker mpm, just my hosting management software doesn't.

[mjones-vsat]

Is there some Apache issue that we can refer to that will unblock this?

[omar391]

how about this: https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html

[CamJN]

mod proxy wstunnel forces you to specify a separate target for your ws traffic. Essentially you need something listening on another port than the main http port, which will handle your ws traffic. I'm fairly certain that limitation is why people want a better solution.

[strugee]

I don't believe that's true. You don't need a separate port, just a separate path. This snippet, for example, is working just fine for me in production:

# Proxy WebSocket requests to /stream
ProxyPass /stream ws://127.0.0.1:56488/stream retry=0 timeout=5

# Proxy all other requests requests to /
ProxyPass / http://127.0.0.1:56488/ retry=0 timeout=5
ProxyPassReverse / http://127.0.0.1:56488/

[mjobin-mdsol]

what ? I can use Apache with Rails ActionCable now ?

@strugee you are a life saver... I had long forgotten about this...

[sudoremo]

I don't believe that's true. You don't need a separate port, just a separate path. This snippet, for example, is working just fine for me in production:

# Proxy WebSocket requests to /stream
ProxyPass /stream ws://127.0.0.1:56488/stream retry=0 timeout=5

# Proxy all other requests requests to /
ProxyPass / http://127.0.0.1:56488/ retry=0 timeout=5
ProxyPassReverse / http://127.0.0.1:56488/

Just to clarify: The actual WebSockets-endpoint is also delivered by Passenger, and not a separate process? In the past, I had to use a separate puma process, on a separate port, to facilitate WS with Apache.

[strugee]

I can't comment on Passenger support as I haven't used it for years. I was talking strictly about generic Apache mod_proxy and mod_proxy_ws.

[Gu7z]

Almost 10 years, has there been any progress or alternative solutions explored over, besides resorting to running a parallel server and rewriting routes in Apache?

[mjobin-mdsol]

I'm sure very sad when I have to move away from the Apache module when an app starts using WebSocket.