search

processone / ejabberd

Robust, Ubiquitous and Massively Scalable Messaging Platform (XMPP, MQTT, SIP Server)
https://www.process-one.net/en/ejabberd/
Other
6.09k stars 1.51k forks source link

mod_http_ws gives 502 gateway error when proxypassed by nginx #968

Closed belldandu closed 8 years ago

belldandu commented 8 years ago

isn't the page for it supposed to give an informative page to say its working and that only xmpp clients can use it?

  -
    port: 5280
    ip: "::"
    module: ejabberd_http
    http_bind: true
    request_handlers:
    ##   "/pub/archive": mod_http_fileserver
         "/http-ws": ejabberd_http_ws
    web_admin: true
    http_poll: true
    ## register: true
    captcha: true

https://im.ilp.moe/http-ws/

saeed-rz commented 8 years ago

the request handle by nginx! i think it wrong, can you access to web admin?? what is your host address?

ros-tel commented 8 years ago 
$ dpkg -l | grep nginx
ii  nginx                                1.8.0-1~wheezy                    amd64        high performance web server

my configuration nginx working with https://github.com/otalk/stanza.io

$ cat /etc/nginx/nginx.conf
...
http {
...
    upstream xmpp_websocket {
        server 192.168.1.40:5280;
        server 192.168.1.41:5280;
        server 192.168.1.42:5280;
    }

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

 server {
...
    location ~* /http-ws$ {
        proxy_pass http://xmpp_websocket;
        proxy_http_version 1.1;
        proxy_read_timeout 512s;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location ~* /http-ws/$ {
        proxy_pass http://xmpp_websocket;
        proxy_http_version 1.1;
        proxy_read_timeout 512s;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
...
 }
}
prefiks commented 8 years ago

Please try removing those proxy_set_headers instructions, and see it it help

ros-tel commented 8 years ago

@kamijoutouma, check your nginx version websocket

Since version 1.3.13, nginx implements special mode of operation that allows setting up a tunnel between a client and proxied server if the proxied server returned a response with the code 101 (Switching Protocols), and the client asked for a protocol switch via the “Upgrade” header in a request.

belldandu commented 8 years ago

@saeed-rz I can access admin panel. I gave you my host address in the first post https://im.ilp.moe/http-ws/ a simple switch to https://im.ilp.moe/admin/ would have shown you. also https://im.ilp.moe/http-bind/ https://im.ilp.moe/http-poll/

@ros-tel @prefiks my nginx config is a bit complicated

server {
        include /home/touma/webserver.cfg/cfg.d/listen;
        root /var/www/default;
        include /home/touma/webserver.cfg/cfg.d/index;
        server_name im.ilp.moe;
        include /home/touma/webserver.cfg/cfg.d/set;
        include /home/touma/webserver.cfg/cfg.d/style;
        include /home/touma/webserver.cfg/cfg.d/errors;
        location / {
                proxy_pass http://127.0.0.1:5280;
                proxy_intercept_errors on;
                proxy_read_timeout 120;
                proxy_send_timeout 120;
                proxy_connect_timeout 120;
        }
        include /home/touma/webserver.cfg/cfg.d/php;
        include /home/touma/webserver.cfg/cfg.d/acme-challenge;
}

ignore set, style, errors and php configs as those are for my custom error page. acme-challenge is for https acme challenges via webroot listen is for making it listening on both ipv4 and 6 addresses on port 80 and 443. and index is for index.php index.html etc

The 120 timeouts are so that pidgin and other clients that are behind a coorperate firewall dont get ping timeouts while using bosh for xmpp.

as for my nginx version 

touma@Index:~/webserver.cfg/ilp.moe.d$ sudo nginx -v
[sudo] password for touma: 
nginx version: nginx/1.6.2

@prefiks thats not my nginx config? im guessing your talking to ros-tel

My main problem here is that instead of sending 101 via http it sends absolutely nothing right now. https://im.ilp.moe/http-ws/

But all of these return something

https://im.ilp.moe/admin/ https://im.ilp.moe/http-bind/ https://im.ilp.moe/http-poll/

And they work with xmpp clients.

ros-tel commented 8 years ago

WebSocket proxying requires special configuration. Add your config location for support WS

...
        location /http-ws {
                proxy_pass http://127.0.0.1:5280;
                proxy_http_version 1.1;
                proxy_read_timeout 512s;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
        }
...
belldandu commented 8 years ago

nginx: [emerg] unknown "connection_upgrade" variable

Pretty sure thats not supposed to happen lol.

belldandu commented 8 years ago 
server {
        include /home/touma/webserver.cfg/cfg.d/listen;
        root /var/www/default;
        include /home/touma/webserver.cfg/cfg.d/index;
        server_name im.ilp.moe;
        include /home/touma/webserver.cfg/cfg.d/set;
        include /home/touma/webserver.cfg/cfg.d/style;
        include /home/touma/webserver.cfg/cfg.d/errors;
        location / {
                proxy_pass http://127.0.0.1:5280;
                proxy_intercept_errors on;
                proxy_read_timeout 120;
                proxy_send_timeout 120;
                proxy_connect_timeout 120;
        }
        location /http-ws {
                proxy_http_version 1.1;
                proxy_read_timeout 512s;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
        }
        include /home/touma/webserver.cfg/cfg.d/php;
        include /home/touma/webserver.cfg/cfg.d/acme-challenge;
}

With this config i get no more 502 bad gateways just 404 not founds.

With this config 

server {
        include /home/touma/webserver.cfg/cfg.d/listen;
        root /var/www/default;
        include /home/touma/webserver.cfg/cfg.d/index;
        server_name im.ilp.moe;
        include /home/touma/webserver.cfg/cfg.d/set;
        include /home/touma/webserver.cfg/cfg.d/style;
        include /home/touma/webserver.cfg/cfg.d/errors;
        location / {
                proxy_pass http://127.0.0.1:5280;
                proxy_intercept_errors on;
                proxy_read_timeout 120;
                proxy_send_timeout 120;
                proxy_connect_timeout 120;
        }
        location /http-ws {
                proxy_pass http://127.0.0.1:5280;
                proxy_http_version 1.1;
                proxy_read_timeout 512s;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
        }
        include /home/touma/webserver.cfg/cfg.d/php;
        include /home/touma/webserver.cfg/cfg.d/acme-challenge;
}

I get 502's again.

belldandu commented 8 years ago

Either way The ws handshake fails every time.

ros-tel commented 8 years ago

in the first console tcpdump -i eth0 -vnn port 80 -c 10000 -w /tmp/front.cap in the second console tcpdump -i lo -vnn port 5280 -c 10000 -w /tmp/back.cap analyze in wireshark

belldandu commented 8 years ago

For some reason when it gets to port 80 and 443 after being proxied by nginx the connection is changed to "keep-alive" even though its sent as upgrade from port 5280.

My nginx config clearly is set to upgrade though.

I have attached the capture files. @ros-tel kami.zip

belldandu commented 8 years ago

Is it possible that this is an Nginx bug?

ros-tel commented 8 years ago

add in configuration nginx map

http {
...
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
...
}
belldandu commented 8 years ago 
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

upstream websocket {
    server 127.0.0.1:5280;
}

server {
        include /home/touma/webserver.cfg/cfg.d/listen;
        root /var/www/default;
        include /home/touma/webserver.cfg/cfg.d/index;
        server_name im.ilp.moe;
        include /home/touma/webserver.cfg/cfg.d/set;
        include /home/touma/webserver.cfg/cfg.d/style;
        include /home/touma/webserver.cfg/cfg.d/errors;
        location / {
                proxy_pass http://websocket;
                proxy_intercept_errors on;
                proxy_read_timeout 120;
                proxy_send_timeout 120;
                proxy_connect_timeout 120;
        }
        location /http-ws/{
                proxy_pass http://websocket;
                proxy_http_version 1.1;
                proxy_read_timeout 512s;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
        }
        include /home/touma/webserver.cfg/cfg.d/php;
        include /home/touma/webserver.cfg/cfg.d/acme-challenge;
}

Done already still getting 502 bad gateway errors.

belldandu commented 8 years ago

Have started a thread on nginx's forums https://forum.nginx.org/read.php?15,265204

ros-tel commented 8 years ago 
$ curl -I -H "Connection: Upgrade" http://im.ilp.moe:5280/admin/
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
Content-Length: 333
WWW-Authenticate: basic realm="ejabberd"
$ curl -I -H "Connection: Upgrade" http://im.ilp.moe:5280/http-ws/
curl: (52) Empty reply from server

my:

$ curl -I -H "Connection: Upgrade" http://127.0.0.1:5280/http-ws/
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/xml; charset=utf-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type
Cache-Control: max-age=0, no-cache, no-store
ros-tel commented 8 years ago 
$ curl -i -H "Connection: Upgrade" -H "Upgrade: WebSocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Protocol: xmpp" -H "Sec-WebSocket-Key: cyiRtCT1uYakKgIKSSG07w==" -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" -H "DNT: 1" http://127.0.0.1:5280/http-ws/
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-Websocket-Protocol:xmpp
Sec-WebSocket-Accept: a+TgKVlAae8GCAMxJJhe8mplFag=
belldandu commented 8 years ago

why is the server returning an empty reply?

ros-tel commented 8 years ago

no reply

unpack kami.zip back.cap open in Wireshark Filter: http screen

ros-tel commented 8 years ago 
$ ejabberdctl debug
> ejabberd_http_ws:module_info().
belldandu commented 8 years ago 
root@Index:~# ejabberdctl debug --node ejabberd@Index
--------------------------------------------------------------------

IMPORTANT: we will attempt to attach an INTERACTIVE shell
to an already running ejabberd node.
If an ERROR is printed, it means the connection was not successful.
You can interact with the ejabberd node if you know how to use it.
Please be extremely cautious with your actions,
and exit immediately if you are not completely sure.

To detach this shell from ejabberd, press:
  control+c, control+c

--------------------------------------------------------------------
To bypass permanently this warning, add to ejabberdctl.cfg the line:
  EJABBERD_BYPASS_WARNINGS=true
Press any key to continue

Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:true]

Eshell V6.2  (abort with ^G)
(ejabberd@Index)1> ejabberd_http_ws:module_info().
** exception error: undefined function ejabberd_http_ws:module_info/0
(ejabberd@Index)2> mod_http_ws:module_info().     
** exception error: undefined function mod_http_ws:module_info/0
ros-tel commented 8 years ago

I hope now you understand why it can not be workable?

belldandu commented 8 years ago

Can someone tell me why ejabberd is not up to date for debian jessie 8.3?

The node ejabberd@Index is started with status: started
ejabberd 14.07 is running in that node

https://www.ejabberd.im/forum/25396/ejabberd-1602-happy-leap-day

16.02 is the latest version and yet the package for debian jessie is clearly out of date and is clearly the cause for these issues. as the module ejabberd_http_ws is only supported in version 15.03 or newer as stated here http://stackoverflow.com/questions/32070678/javascript-websockets-and-xmpp-client-how-to-make-them-work-together (which btw should definetly be pointed out in the docs)

belldandu commented 8 years ago

apparently this is why https://packages.qa.debian.org/e/ejabberd.html

belldandu commented 8 years ago

gonna try to get the newer version from backports

ros-tel commented 8 years ago

make from source It works for me 15.10

belldandu commented 8 years ago

So i did this.

Dumped database to /etc/ejabberd/ejabberd.dmp with ejabberdctl dump /etc/ejabberd/ejabberd.dmp moved /etc/ejabberd/ejabberd.yml to /etc/ejabberd/ejabberd.yml.backup ran ejabberdctl stop changed directory to /etc/apt/sources.list.d/ made the file backports.list and put in it deb http://http.debian.net/debian jessie-backports main ran apt-get update ran apt-get -t jessie-backports install ejabberd compared the changes between the newly created /etc/ejabberd/ejabberd.yml with /etc/ejabberd/ejabberd.yml.backup and added any configs i had originaly changed ran ejabberdctl start ran ejabberdctl load /etc/ejabberd/ejabberd.dmp ran ejabberdctl status Got back

The node ejabberd@Index is started with status: started
ejabberd 16.01 is running in that node

So now my version is greater then the required 15.03

Hopefully now it will work.

belldandu commented 8 years ago

and nope 

curl -i -H "Connection: Upgrade" -H "Upgrade: WebSocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Protocol: xmpp" -H "Sec-WebSocket-Key: cyiRtCT1uYakKgIKSSG07w==" -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" -H "DNT: 1" http://127.0.0.1:5280/http-ws/
curl: (52) Empty reply from server
belldandu commented 8 years ago

At least this returned something

root@Index:~# ejabberdctl debug --node ejabberd@Index
--------------------------------------------------------------------

IMPORTANT: we will attempt to attach an INTERACTIVE shell
to an already running ejabberd node.
If an ERROR is printed, it means the connection was not successful.
You can interact with the ejabberd node if you know how to use it.
Please be extremely cautious with your actions,
and exit immediately if you are not completely sure.

To detach this shell from ejabberd, press:
  control+c, control+c

--------------------------------------------------------------------
To bypass permanently this warning, add to ejabberdctl.cfg the line:
  EJABBERD_BYPASS_WARNINGS=true
Press return to continue

Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:true]

Eshell V6.2  (abort with ^G)
(ejabberd@Index)1> ejabberd_http_ws:module_info().
[{exports,[{start,1},
           {start_link,1},
           {send_xml,2},
           {setopts,2},
           {sockname,1},
           {peername,1},
           {controlling_process,2},
           {become_controller,2},
           {close,1},
           {socket_handoff,6},
           {init,1},
           {handle_event,3},
           {handle_sync_event,4},
           {handle_info,3},
           {code_change,4},
           {terminate,3},
           {opt_type,1},
           {module_info,0},
           {module_info,1}]},
 {imports,[]},
 {attributes,[{vsn,[130009529784351066649753598160352844685]},
              {lager_records,[{state,[socket,ping_interval,ping_timer,
                                      pong_expected,timeout,timer,input,waiting_input,
                                      last_receiver,ws,rfc_compilant]},
                              {ws,[socket,sockmod,ip,host,port,path,headers,local_path,q,
                                   buf,http_opts]},
                              {request,[method,path,q,us,auth,lang,data,ip,host,port,opts,
                                        tp,headers]},
                              {xmlelement,[name,attrs,children]},
                              {rsm_out,[count,index,first,last]},
                              {rsm_in,[max,direction,id,index]},
                              {iq,[id,type,xmlns,lang,sub_el]},
                              {jid,[user,server,resource,luser,lserver,lresource]},
                              {xmlel,[name,attrs,children]},
                              {scram,[storedkey,serverkey,salt,iterationcount]}]},
              {behaviour,[ejabberd_config]},
              {author,['ecestari@process-one.net']},
              {behaviour,[gen_fsm]}]},
 {compile,[{options,[{outdir,"ebin"},
                     debug_info,nowarn_deprecated_function,
                     {d,'LAGER'},
                     {src_dirs,[asn1,src]},
                     {i,"include"}]},
           {version,"5.0.2"},
           {time,{2016,1,31,15,30,4}},
           {source,"/build/ejabberd-16.01/src/ejabberd_http_ws.erl"}]}]
belldandu commented 8 years ago

funny thing though is that my watchdog has been sent a message twice now

(01:43:29 AM) im.ilp.moe/watchdog: (ejabberd@Index) The process <0.382.0> is consuming too much memory:
[{old_heap_block_size,833026},
 {heap_block_size,196650},
 {mbuf_size,0},
 {stack_size,8464},
 {old_heap_size,15185},
 {heap_size,77320}]
[{current_function,{scram,hi_round,3}},
 {initial_call,{mnesia_schema,schema_coordinator,3}},
 {message_queue_len,0},
 {links,[<0.168.0>,<0.150.0>]},
 {dictionary,
     [{mnesia_activity_state,
          {mnesia,{tid,1889,<0.382.0>},{tidstore,409709,[],1}}}]},
 {heap_size,196650},
 {stack_size,9299}]
(01:43:29 AM) im.ilp.moe/watchdog: (ejabberd@Index) The process <0.382.0> is consuming too much memory:
[{old_heap_block_size,833026},
 {heap_block_size,318187},
 {mbuf_size,0},
 {stack_size,18359},
 {old_heap_size,92500},
 {heap_size,91039}]
[{current_function,{lists,zipwith,3}},
 {initial_call,{mnesia_schema,schema_coordinator,3}},
 {message_queue_len,0},
 {links,[<0.168.0>,<0.150.0>]},
 {dictionary,
     [{mnesia_activity_state,
          {mnesia,{tid,1889,<0.382.0>},{tidstore,409709,[],1}}}]},
 {heap_size,318187},
 {stack_size,15095}]
ros-tel commented 8 years ago 
# netstat -atpln | grep 5280
belldandu commented 8 years ago

The modules don't seem to be working anymore now though. http://im.ilp.moe:5280/http-bind/ and the like all return empty responses now.

Also the response to that command was: tcp6 0 0 :::5280 :::* LISTEN 26005/beam.smp

belldandu commented 8 years ago

I noticed that there is now a modules.d folder but it only has a README.modules file in it.

root@Index:/etc/ejabberd/modules.d# cat README.modules
# This directory may contain configuration files for additional ejabberd modules
# but they must be named mod_foo.yml, where mod_foo is the module they configure.
#
# For example the config file of any ejabberd-mod-* package can be placed
# here.
belldandu commented 8 years ago

Ah Trailing slashes were removed in this version so instead its. http://im.ilp.moe:5280/http-bind

ros-tel commented 8 years ago 
ejabberdctl stop
rm -f /lib/ejabberd/*
apt-get -t jessie-backports install --reinstall ejabberd
ejabberdctl start
belldandu commented 8 years ago

Doing that didn't fix it. I had to go back into the yml and disable the new option tls. as it was forcing https on port 5280 which nginx didn't like.

also http://im.ilp.moe/http-ws/ works \o/

belldandu commented 8 years ago

Also i noticed that http-poll no longer works. not sure why but wte. In anycase my issue has been fixed so i will close this now. 

root@Index:/etc/ejabberd# curl -i -H "Connection: Upgrade" -H "Upgrade: WebSocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Protocol: xmpp" -H "Sec-WebSocket-Key: cyiRtCT1uYakKgIKSSG07w==" -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" -H "DNT: 1" http://127.0.0.1:5280/http-ws/
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-Websocket-Protocol:xmpp
Sec-WebSocket-Accept: a+TgKVlAae8GCAMxJJhe8mplFag=

root@Index:/etc/ejabberd# curl -i -H "Connection: Upgrade" -H "Upgrade: WebSocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Protocol: xmpp" -H "Sec-WebSocket-Key: cyiRtCT1uYakKgIKSSG07w==" -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" -H "DNT: 1" http://im.ilp.moe/http-ws/
HTTP/1.1 101 Switching Protocols
Server: nginx/1.6.2
Date: Fri, 11 Mar 2016 07:10:23 GMT
Connection: upgrade
Upgrade: websocket
Sec-Websocket-Protocol: xmpp
Sec-WebSocket-Accept: a+TgKVlAae8GCAMxJJhe8mplFag=
lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.