search

yaoweibin / nginx_tcp_proxy_module

add the feature of tcp proxy with nginx, with health check and status monitor
http://yaoweibin.github.com/nginx_tcp_proxy_module
1.65k stars 365 forks source link

HOWTO for websckets #6

Open superfeedr opened 13 years ago

superfeedr commented 13 years ago

Hey

Not really an issue, but more of a request. Would this module fit the needs of loadbalancing with a wbesocket backend? If so, how?

yaoweibin commented 13 years ago

Yes, websocket is also a tcp socket. This module can't handle the websocket, but just proxy the websocket request to the backend websocket servers.

tcp {

    upstream websocket_cluster {
        # simple round-robin
        server 127.0.0.1:3306;
        server 127.0.0.1:1234;

        check interval=3000 rise=2 fall=5 timeout=1000;
    }

    server {
        listen 8888;

        proxy_pass websocket_cluster;
    }
}
superfeedr commented 13 years ago

Interesting. But will it keep the socket open/busy the the whole time?

yaoweibin commented 13 years ago

Yes, If any peer of the connection is closed, the websocket connection is closed too.

ghost commented 13 years ago

Héllo,

Is there anyway to make both tcp and http listen to the same port ?

yaoweibin commented 13 years ago

Sorry, There is no solution at present.

ghost commented 13 years ago

you should make it more clear in the documentation that this websocket things works well.

cheers & thanks for the lib,

abki.

2011/4/7 yaoweibin < reply@reply.github.com>

Sorry, There is no solution at present.

Reply to this email directly or view it on GitHub: https://github.com/yaoweibin/nginx_tcp_proxy_module/issues/6#comment_968931

yaoweibin commented 13 years ago

Thanks for your suggestion. I have added this docs. At first, I developed this module just for general TCP proxy. And now, this module is frequently used in websocket reverse proxying.

ghost commented 13 years ago

your talking about TCP and proxy thing is something I cannot understand ;) but thanks.

eeezyy commented 13 years ago

I get the error "nginx: [emerg] unknown directive "tcp" in /etc/nginx/sites-enabled/...". Does it mean I configured it wrong or my installation of nginx didn't integrate this module properly?

yaoweibin commented 13 years ago

On 2011-7-29 0:29, eeezyy wrote:

I get the error "nginx: [emerg] unknown directive "tcp" in /etc/nginx/sites-enabled/...". Does it mean I configured it wrong or my installation of nginx didn't integrate this module properly? Have you added the patch before compile?

patch -p1< /path/to/nginx_tcp_proxy_module/tcp.patch

Thanks.

Weibin Yao

eeezyy commented 13 years ago

Yes, I followed a tutorial on a blog. But before that I installed it with apt-get in ubuntu. I removed it and compiled it from the source with this module and installed it. Maybe I also had to remove the config files before the install?! I will try it again, thanks.

yaoweibin commented 13 years ago

On 2011-7-29 9:02, eeezyy wrote:

Yes, I followed a tutorial on a blog. But before that I installed it with apt-get in ubuntu. I removed it and compiled it from the source with this module and installed it. Maybe I also had to remove the config files before the install?! I will try it again, thanks. Can you type this command, and show me the result?

nginx -V

Can you show me your config file?

Weibin Yao

eeezyy commented 13 years ago

Thanks for your advise. There is nothing mentioned of this module in the configure arguments:

nginx: nginx version: nginx/1.0.4
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-debug --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-http_xslt_module --with-ipv6 --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl --with-mail --with-mail_ssl_module --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-echo --add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upstream-fair

I removed all folders and files of nginx manually and compiled it again. Now if I try to start it with "/etc/init.d/nginx start" it does nothing. When I check with "which nginx" is also shows me nothing. So I run the "nginx -V" command directly in "/usr/local/nginx/sbin/" and it showed me that the module was configured. Also the test of the config file succeeds. It seems that the problem was, that I installed nginx with apt-get first. But now when selfcompiled it is not so comfortable (can't run "/etc/init.d/nginx start" but "/usr/local/nginx/sbin/nginx").

yaoweibin commented 13 years ago

The default Nginx installation directory in Ubuntu by apt-get may be /usr/sbin. You can see it in the file of /etc/init.d/nginx. If you compile Nginx yourself. The default installation directory is under /usr/local/.

eeezyy commented 13 years ago

Thanks, it works great! But it's sad, that the same port does not work with tcp and http at the same time. Is there any workaround?

yaoweibin commented 13 years ago

On 2011-8-5 9:56, eeezyy wrote:

Thanks, it works great! But it's sad, that the same port does not work with tcp and http at the same time. Is there any workaround? Sorry, No.

Weibin Yao

Morgon commented 12 years ago

Hello.. I'm also getting unknown directive 'tcp' / 'http' after installing this patch. I'm using 1.1.4 (as I had thought their new HTTP version passthru would allow full Socket.IO connections.. it doesn't). The output of nginx -V echos the fact that I've configured it.

Could it simply be a version incompatibility, or is there anything else I could potentially try?

yaoweibin commented 12 years ago

On 2011-9-29 20:03, Morgon wrote:

Hello.. I'm also getting unknown directive 'tcp' / 'http' after installing this patch. I'm using 1.1.4 (as I had thought their new HTTP version passthru would allow full Socket.IO connections.. it doesn't). The output of nginx -V echos the fact that I've configured it.

Could it simply be a version incompatibility, or is there anything else I could potentially try? I have not test it with Nginx-1.1.4. This version support backend http/1.1, but not the websocket.

Maybe you can use a older version. I'm having a holiday for about a week. I'll test with it after the holiday.

Thanks.

Weibin Yao

Morgon commented 12 years ago

Just wanted to mention that 1.0.6 (latest stable) doesn't recognize the http or tcp directives, either. Look forward to hearing what you find out after your vacation/holiday! :)

yaoweibin commented 12 years ago

On 2011-9-30 8:57, Morgon wrote:

Just wanted to mention that 1.0.6 (latest stable) doesn't recognize the http or tcp directives, either. Look forward to hearing what you find out after your vacation/holiday! :) Can you show me your configure file? It's fine when I tested with Nginx-1.0.8.

Thanks.

Weibin Yao

Morgon commented 12 years ago

Hmm. You know, I think you have nudged me in the direction to find my issue. When you mentioned the configs I was going to get you the main config, and what do I see... 'http'. So the 'tcp' definition has to be in the main nginx.conf? All of my config changes as of late have been to vhost includes, and that's where I was trying to use the tcp block.

So is that where I'm going wrong? If so, will the tcp{} section honor the 'include' directive, and allow me to specify routes per host? I am running multiple sites, and I may want to use tcp (i.e. socket.io) on more than one.

Thanks again!

yaoweibin commented 12 years ago

On 2011-10-6 20:13, Morgon wrote:

Hmm. You know, I think you have nudged me in the direction to find my issue. When you mentioned the configs I was going to get you the main config, and what do I see... 'http'. So the 'tcp' definition has to be in the main nginx.conf? All of my config changes as of late have been to vhost includes.

So is that where I'm going wrong? If so, will the tcp{} section honor the 'include' directive, and allow me to specify routes per host? I am running multiple sites, and I may want to use tcp (i.e. socket.io) on both.

Thanks again! Yes. You are right. The 'tcp' directive can not be used in the 'http' block. This tcp module is like the mail module.

You can use 'include' directive in the tcp block.

But at present it can't route the request by host. If you want to do this, you should open different ports. This module is just a general tcp proxy module. It does not parse the content of socket at all.

Thanks.

Weibin Yao

burakdede commented 12 years ago

I am kind of confused about this tcp reverse proxy thing, I can easily configure nginx to work with 4 tornado instance here is my nginx.conf file for it https://gist.github.com/1426598 suppose my appilcation host on mysite.com and my websocket path is mysite.com/chatsocket but I stuck in here cause I can not make tcp proxy to work cause its not clear where to put tcp config file or configuration in wiki ?

yaoweibin commented 12 years ago

On 2011-12-3 17:15, Burak Dede wrote:

I am kind of confused about this tcp reverse proxy thing, I can easily configure nginx to work with 4 tornado instance here is my nginx.conf file for it https://gist.github.com/1426598 suppose my appilcation host on mysite.com and my websocket path is mysite.com/chatsocket but I stuck in here cause I can not make tcp proxy to work cause its not clear where to put tcp config file or configuration in wiki ? You should specify another socket port with this tcp proxy module. It can't listen the same port with http.

Weibin Yao

ghost commented 11 years ago

I am using websockets with TCP module. I explained it here.

vasilyev-alexander commented 11 years ago

Hello We're developing web-chat application using tornado web server with sockjs module supporting web sockets. Our traffic is proxied with your nginx_tcp_proxy_module using this standard config:

tcp { upstream websocket_cluster { server 192.168.2.226:8051; check interval=3000 rise=2 fall=5 timeout=1000; }

    server {
            listen 192.168.2.226:8888;
            proxy_pass websocket_cluster;
    }

}

Having this tcp-packets dump our question is why your module initiates disconnet?

tcpdump -n -i en1 | grep 192.168.2.226 11:18:49.804815 IP 192.168.2.199.54449 > 192.168.2.226.8888: Flags [S], seq 1732066136, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 832551655 ecr 0,sackOK,eol], length 0 11:18:49.807037 IP 192.168.2.226.8888 > 192.168.2.199.54449: Flags [S.], seq 2739465710, ack 1732066137, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 803479898 ecr 832551655], length 0 11:18:49.807086 IP 192.168.2.199.54449 > 192.168.2.226.8888: Flags [.], ack 1, win 65535, options [nop,nop,TS val 832551658 ecr 803479898], length 0 11:18:49.807263 IP 192.168.2.199.54449 > 192.168.2.226.8888: Flags [P.], seq 1:380, ack 1, win 65535, options [nop,nop,TS val 832551658 ecr 803479898], length 379 11:18:49.810656 IP 192.168.2.226.8888 > 192.168.2.199.54449: Flags [P.], seq 1:378, ack 380, win 8326, options [nop,nop,TS val 803479899 ecr 832551658], length 377 11:18:49.810695 IP 192.168.2.199.54449 > 192.168.2.226.8888: Flags [.], ack 378, win 65535, options [nop,nop,TS val 832551661 ecr 803479899], length 0 11:18:49.812660 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [S], seq 397217509, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 832551662 ecr 0,sackOK,eol], length 0 11:18:49.814261 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [S.], seq 1568401255, ack 397217510, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 1595631246 ecr 832551662], length 0 11:18:49.814311 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 1, win 65535, options [nop,nop,TS val 832551663 ecr 1595631246], length 0 11:18:49.814987 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [P.], seq 1:311, ack 1, win 65535, options [nop,nop,TS val 832551663 ecr 1595631246], length 310 11:18:49.818209 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [P.], seq 1:130, ack 311, win 8326, options [nop,nop,TS val 1595631246 ecr 832551663], length 129 11:18:49.818306 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 130, win 65535, options [nop,nop,TS val 832551666 ecr 1595631246], length 0 11:18:49.818314 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [P.], seq 130:133, ack 311, win 8326, options [nop,nop,TS val 1595631246 ecr 832551663], length 3 11:18:49.818376 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 133, win 65535, options [nop,nop,TS val 832551666 ecr 1595631246], length 0 11:18:49.819958 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [P.], seq 133:180, ack 311, win 8326, options [nop,nop,TS val 1595631246 ecr 832551663], length 47 11:18:49.819989 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 180, win 65535, options [nop,nop,TS val 832551667 ecr 1595631246], length 0 11:19:14.821323 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [P.], seq 180:183, ack 311, win 8326, options [nop,nop,TS val 1595635732 ecr 832551667], length 3 11:19:14.821375 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 183, win 65535, options [nop,nop,TS val 832576516 ecr 1595635732], length 0 11:19:39.826181 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [P.], seq 183:186, ack 311, win 8326, options [nop,nop,TS val 1595640218 ecr 832576516], length 3 11:19:39.826245 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 186, win 65535, options [nop,nop,TS val 832601310 ecr 1595640218], length 0 11:19:49.809268 IP 192.168.2.226.8888 > 192.168.2.199.54449: Flags [F.], seq 378, ack 380, win 8326, options [nop,nop,TS val 803490665 ecr 832551661], length 0 11:19:49.809328 IP 192.168.2.199.54449 > 192.168.2.226.8888: Flags [.], ack 379, win 65535, options [nop,nop,TS val 832611240 ecr 803490665], length 0 11:19:49.815474 IP 192.168.2.199.54449 > 192.168.2.226.8888: Flags [F.], seq 380, ack 379, win 65535, options [nop,nop,TS val 832611246 ecr 803490665], length 0 11:19:49.817372 IP 192.168.2.226.8888 > 192.168.2.199.54449: Flags [.], ack 381, win 8325, options [nop,nop,TS val 803490666 ecr 832611246], length 0 11:19:49.820341 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [F.], seq 186, ack 311, win 8326, options [nop,nop,TS val 1595642013 ecr 832601310], length 0 11:19:49.820387 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [.], ack 187, win 65535, options [nop,nop,TS val 832611249 ecr 1595642013], length 0 11:19:49.820482 IP 192.168.2.199.54450 > 192.168.2.226.8888: Flags [F.], seq 311, ack 187, win 65535, options [nop,nop,TS val 832611249 ecr 1595642013], length 0 11:19:49.822264 IP 192.168.2.226.8888 > 192.168.2.199.54450: Flags [.], ack 312, win 8325, options [nop,nop,TS val 1595642013 ecr 832611249], length 0

Our backend sends "hello" packet to the client (192.168.2.199.54450) every 25 seconds, is this not enough to keep connection alive? Why does this activity do not reset your timeout counter?

Thanks With respect, Alexander

yaoweibin commented 11 years ago

There are three timeout directives. Your case may break the proxy_send_timeout. Try these directives:

  timeout 1d;
  proxy_read_timeout 1d;
  proxy_send_timeout 1d;