phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
5.01k stars 547 forks source link

WebSockets don't work on Apache #1202

Open FooBarWidget opened 10 years ago

FooBarWidget commented 10 years ago

If the response has the chunked transfer encoding, then WebSockets handshakes become corrupted. See https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/phusion-passenger/XtjxA8E1idw/Z029OysamjsJ for the report.

FooBarWidget commented 10 years ago

See also https://github.com/phusion/passenger-nodejs-websocket-demo/issues/2

FooBarWidget commented 10 years ago

The chunked encoding problem can be solved by setting Content-Length: 0 in the response, which will prevent Apache from setting Transfer-Encoding.

// Process WebSocket upgrade.
Header upgrade = lookupHeader(headerData, "Upgrade", "upgrade");
if (!upgrade.empty()) {
    RH_TRACE(client, 2, "WebSocket upgrade detected.");
    headerData.append("Content-Length: 0\r\n");
}

However, it looks like the problem is more complicated than just solving this. The current Apache module currently does not support simultaneously forwarding request data and response data, so WebSockets still doesn't work. Worse, instead of triggering an error, the connection just freezes.

Fixing this is non-trivial. I'm postponing it to a later release.

captn3m0 commented 9 years ago

:+1: Waiting for this eagerly.

kiranos commented 9 years ago

+1

kiranos commented 9 years ago

any news on this?

FooBarWidget commented 9 years ago

Unfortunately not. If you need WebSockets, for the time being we recommend using Nginx, or using Passenger Standalone behind Apache + mod_proxy.

vanboom commented 9 years ago

See https://github.com/phusion/passenger-ruby-websocket-demo/issues/2, appears WebSockets are still not working under Apache mod_passenger in 5.0.18? (It's very possible that I am doing something wrong in my Apache .conf) Thanks for Passenger - it's incredible!

FooBarWidget commented 9 years ago

@vanboom That is correct, WebSockets do not work yet in the Apache version. For the time being, we recommend that you use the Nginx version, or Apache + Standalone in a reverse proxy setup, if you need WebSockets.

ThaddeusAid commented 8 years ago

It has been almost a year since the last update. Did the upgrade to apache 2.4.x improve this?

CamJN commented 8 years ago

There is a websocket proxy module now for Apache 2.4, however we have not found it to be robust enough to recommend yet.

mk-pmb commented 8 years ago

Just discovered passenger, sounds like a gift from heaven. :+1: In the project I want to use to explore passenger, I can set specific ports (or env var names to read the ports from) for WebSockets and EventSource, will that help? It's meant as a work-around to use with Apache's mod_proxy, but maybe passenger can intercept to sanitize and monitor stuff.

mk-pmb commented 8 years ago

Just for the record, some people seem to have managed to use websockets in apache with wstunnel.

CamJN commented 8 years ago

Yes, that's the websocket proxy module I mentioned, however it only works for some people. We'll consider recommending it when it is more robust.

jamgregory commented 7 years ago

@CamJN - out of interest, what issues did you have with the websocket module in Apache? I presume there's some work involved on the Passenger side to integrate (if possible) with that so ActionCable will work through Passenger directly. We're looking at a future development project that will require ActionCable and ideally we'd like Passenger to support this if possible.

CamJN commented 7 years ago

As one example since this module is just a websocket-aware proxy, you need proxy support in your websocket library so it won't just work for everyone.

mk-pmb commented 7 years ago

you need proxy support in your websocket library

What features will that consist of? Can I test all those features on a per-app level, independent of which ws lib it uses? Would it help to have a shims option where I can configure what workarounds to use until some feature is available?

One feature I might imagine as "proxy support" would be about determining the IP from which a connection arrives. A work-around might be to specify the network interface from which the proxy should send its packets, to ensure it uses an interface (and thus IP address) that my statistics can identify as "not a real visitor" and which the app doesn't assume as privileged in any way. The setting might have an optional "allow localhost" flag which determines whether to fail loudly in case the specified interface will be undistinguishable from localhost.

CamJN commented 7 years ago

What features will that consist of?

Generating urls for links and js that refer to the proxy's url/port not the app's.

Can I test all those features on a per-app level, independent of which ws lib it uses?

No, because they have to come from the ws lib.

Would it help to have a shims option where I can configure what workarounds to use until some feature is available?

Yes, but I don't know if such a thing exists.

mk-pmb commented 7 years ago

I'll share my idea since I couldn't finish my research on it: A lot of web apps have their websockets behind very few, short URL prefixes like /ws/ (http://api.example.com/ws/chat). If we can make the browser and potential proxies send the websocket request as the first request in the TCP connection (a randomly selected subdomain should suffice), a netfilter string match or l7-filter should be able to pick up on the request URL and NAT it to bypass Apache. (Please make double sure to avoid false positive matches.)

For HTTPS, we could hope that browsers that support websockets will also use Server Name Indication (SNI), so netfilter might be able to detect a hostname that an app can use exclusively for its websockets, and then redirect that port 443 connection to the app directly, or an auxiliary web server, again bypassing Apache. Update: SNI hostname transmission happens too late (usually not in the first packet) for easy methods of port redirection.

blevine commented 7 years ago

Anyone having any luck with this? mod_wstunnel has been around for a while now. We've had no problem getting websockets to work over an Apache reverse proxy using mod_wstunnel. However, still not sure how to get this to work using Passenger in Apache integration mode.

heangratha commented 6 years ago

Any one have luck to get websocket work with Apache? This is my Apache configure get from: https://www.phusionpassenger.com/library/walkthroughs/deploy/meteor/ownserver/apache/oss/xenial/deploy_app.html#edit-apache-configuration-file

But the websocket won't work.. I'm using Ubuntu 16.04, Apache 2.4.18, Phusion Passenger 5.2.0

jamgregory commented 6 years ago

@heangratha - the only solution we could come up with was to run a separate instance of the app in Puma, then proxy all WebSocket requests to that using mod_wstunnel.

thomasbalsloev commented 6 years ago

Any movement on this issue?

CamJN commented 6 years ago

Not right now, it is unfortunately not a priority at the moment. People are recommended to use nginx if they need websockets, as it's still a pain with Apache.

thomasbalsloev commented 6 years ago

@CamJN

OK, I see. Thank you for the update.

mk-pmb commented 6 years ago

Update on my research (it's on the back burner though): We could probably write a module like fdpass to get the socket descriptors. We'd then have to exfiltrate the connection's SSL context (ideally: compile Apache with an SSL lib that officially supports this; worst case: dump worker's process memory, then recover forensically) and send that along with the connection socket file descriptor to a server that can receive both and continue the SSL session. The metasploit codebase probably has useful parts to reuse here.

Generating urls for links and js that refer to the proxy's url/port not the app's.

I'm optimistic that URL rewriting might be an option for people with a bit of backgorund knowledge about the app – which the app's admin should have.

siegy22 commented 5 years ago

Is this a wontfix?

CamJN commented 5 years ago

@siegy22 not exactly, but it's blocked on the Apache project improving their module.

annymosse commented 4 years ago

@vanboom That is correct, WebSockets do not work yet in the Apache version. For the time being, we recommend that you use the Nginx version, or Apache + Standalone in a reverse proxy setup, if you need WebSockets.

@FooBarWidget how can i do it (Apache + Standalone in a reverse proxy setup)?"still newer in servers configs"

mk-pmb commented 4 years ago

@annymosse How to install Passenger Standalone Once you have that running, it gives a web server. You'll want to ensure it is not reachable directly from the outside. To achieve this, you can use explicit interface binding (to a non-public interface) and/or firewall rules. The Apache Reverse Proxy Guide shows how to make a publicly reachable apache forward traffic for some parts of your URL space, to the guarded private Passenger server.

beppe9000 commented 4 years ago

Does this apply to cpanel ?

deltafactory commented 4 years ago

@beppe9000 Yes, since cPanel's default web server is Apache.

mjobin-mdsol commented 3 years ago

@CamJN

not exactly, but it's blocked on the Apache project improving their module.

is there an open issue on Apache's side we can track?

mjones-vsat commented 2 years ago

Any updates on this?

mk-pmb commented 2 years ago

Sorry for the wrong info earlier. Seems like apache itself still supports prefork worker mpm, just my hosting management software doesn't.

mjones-vsat commented 2 years ago

Is there some Apache issue that we can refer to that will unblock this?

omar391 commented 2 years ago

how about this: https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html

CamJN commented 2 years ago

mod proxy wstunnel forces you to specify a separate target for your ws traffic. Essentially you need something listening on another port than the main http port, which will handle your ws traffic. I'm fairly certain that limitation is why people want a better solution.

strugee commented 2 years ago

I don't believe that's true. You don't need a separate port, just a separate path. This snippet, for example, is working just fine for me in production:

# Proxy WebSocket requests to /stream
ProxyPass /stream ws://127.0.0.1:56488/stream retry=0 timeout=5

# Proxy all other requests requests to /
ProxyPass / http://127.0.0.1:56488/ retry=0 timeout=5
ProxyPassReverse / http://127.0.0.1:56488/
mjobin-mdsol commented 2 years ago

what ? I can use Apache with Rails ActionCable now ?

@strugee you are a life saver... I had long forgotten about this...

sudoremo commented 2 years ago

I don't believe that's true. You don't need a separate port, just a separate path. This snippet, for example, is working just fine for me in production:

# Proxy WebSocket requests to /stream
ProxyPass /stream ws://127.0.0.1:56488/stream retry=0 timeout=5

# Proxy all other requests requests to /
ProxyPass / http://127.0.0.1:56488/ retry=0 timeout=5
ProxyPassReverse / http://127.0.0.1:56488/

Just to clarify: The actual WebSockets-endpoint is also delivered by Passenger, and not a separate process? In the past, I had to use a separate puma process, on a separate port, to facilitate WS with Apache.

strugee commented 2 years ago

I can't comment on Passenger support as I haven't used it for years. I was talking strictly about generic Apache mod_proxy and mod_proxy_ws.

Gu7z commented 8 months ago

Almost 10 years, has there been any progress or alternative solutions explored over, besides resorting to running a parallel server and rewriting routes in Apache?

mjobin-mdsol commented 8 months ago

I'm sure very sad when I have to move away from the Apache module when an app starts using WebSocket.