Prosody crashes if someone tries to create an TLS connection with my muc service

[GoogleCodeExporter]

My Prosody repeatably crashes when one client (always the same, but unknown IP) 
tries to create a client connection with my muc service.

This happened more than one time. The last lines in debug log are always the 
same. (I don't know what this client is doing exactly) Blocking this specific 
IP solved the issue short term.

I attached the relevant lines from the log file.

What is the expected output? What do you see instead?
not crash...

What version of the product are you using? On what operating system?

0.9.4 on Gentoo/Linux

Please provide any additional information below.

Original issue reported on code.google.com by dan...@gultsch.de on 27 Jul 2014 at 3:10

Attachments:

• 2TntWKGK.txt

[GoogleCodeExporter]

Hi, thanks for the report.

Firstly can you confirm that the Prosody process crashes completely? I.e. it is 
no longer running and there is no entry at the end of the log saying that it is 
shutting down? The definition of "crash" can be quite broad :)

Actual crashes of Prosody are rare, but generally indicate a problem with our 
dependencies. Could you list the versions you have installed? The output of 
"prosodyctl about" would be extremely helpful.

Finally, are you using libevent?

Original comment by MWild1 on 27 Jul 2014 at 5:43

[GoogleCodeExporter]

Hi

the process is gone. the attached log file contains the last lines. there are 
no further lines with something like 'shutdown' or any other indication of a 
graceful end.

prosody is compiled with the libevent useflag und libevent is set to true in 
the config file. Would you like me to disable that for testing purposes? 
(although it might become difficult to replicate in the case that one client 
stopped making connections to my server)

and now the output of prosodyctl about

Prosody 0.9.4

# Prosody directories
Data directory:     /var/spool/jabber
Plugin directory:   /usr/local/lib64/prosody/modules/;/usr/lib64/prosody/modules/
Config directory:   /etc/jabber
Source directory:   /usr/lib64/prosody

# Lua environment
Lua version:                Lua 5.1

Lua module search paths:
  /usr/lib64/prosody/?.lua
  /usr/share/luajit-2.0.2/?.lua
  /usr/local/share/lua/5.1/?.lua
  /usr/local/share/lua/5.1/?/init.lua
  /usr/share/lua/5.1/?.lua
  /usr/share/lua/5.1/?/init.lua

Lua C module search paths:
  /usr/lib64/prosody/?.so
  /usr/local/lib/lua/5.1/?.so
  /usr/lib/lua/5.1/?.so
  /usr/local/lib/lua/5.1/loadall.so

LuaRocks:           Not installed

# Lua module versions
lfs:        LuaFileSystem 1.5.0
lxp:        LuaExpat 1.3.0
pposix:     0.3.6
socket:     LuaSocket 3.0-rc1
ssl:        0.4.1

Original comment by dan...@gultsch.de on 27 Jul 2014 at 6:52

[GoogleCodeExporter]

Thanks for the extra info. Unfortunately it doesn't really give any leads.

libevent is certainly a candidate for crashes. Testing with use_libevent = 
false would help eliminate that at least.

The only other thing I can suggest is trying to get a core dump when the 
process crashes, which would hopefully point us to where the problem lies. 
Unfortunately I don't know how to do this on gentoo, though I'm sure guides are 
probably available if you want to attempt it.

Also check your syslog for the time period that the process crashes, in case 
there is any useful information there.

Original comment by MWild1 on 28 Jul 2014 at 9:47

[GoogleCodeExporter]

Hi, did you get any further crashes with use_libevent = false?

Original comment by MWild1 on 7 Sep 2014 at 9:09

[GoogleCodeExporter]

a small status update: When this happend back in July I just blocked the IP 
that caused this. I didn't do much further experimenting. This is and was a 
production server and I was trying to avoid further restarts. And blocking that 
IP was effective enough. Yesterday this happend again. From a different IP 
address. My first instinct was to just block the IP again. But when I woke up 
today the server has crashed again. This time because of another IP (but from 
the same city like yesterday - so maybe same guy?) In neither of the three 
cases it seemed to be a server. (no open ports)
I disabled libevent this time. However since I can not control the interval of 
that person connecting it has hard to say if libevent = false fixed this or 
not. 

Original comment by dan...@gultsch.de on 22 Sep 2014 at 9:27

[GoogleCodeExporter]

Ok, thanks for the update and extra information.

If this issue is getting more frequent, a packet capture might be feasible and 
would help to see if there is a pattern in the traffic that causes the crash. 
Something like:

  tcpdump -i eth0 -s0 -w prosody-tls-crash.pcap port 5222

When the crash occurs, email the pcap to developers@prosody.im (a private 
address for the core Prosody team) and we'll take a look.

Original comment by MWild1 on 22 Sep 2014 at 9:42

[GoogleCodeExporter]

ok this might be interesting:
with libevent disabled it doesn't crash but reports a 'general error'

Sep 22 11:52:03 socket  debug   server.lua: accepted new client connection from 
203.170.73.214:6201 to 5222
Sep 22 11:52:03 c2s40119460 info    Client connected
Sep 22 11:52:03 c2s40119460 debug   Client sent opening <stream:stream> to 
conference.siacs.eu
Sep 22 11:52:03 c2s40119460 debug   Sent reply <stream:stream> to client
Sep 22 11:52:03 c2s40119460 debug   Received[c2s_unauthed]: <starttls 
xmlns='urn:ietf:params:xml:ns:xmpp-tls'>
Sep 22 11:52:03 socket  debug   server.lua: we need to do tls, but delaying until 
send buffer empty
Sep 22 11:52:03 c2s40119460 debug   TLS negotiation started for c2s_unauthed...
Sep 22 11:52:03 socket  debug   server.lua: attempting to start tls on 
tcp{client}: 0x41292430
Sep 22 11:52:05 socket  debug   server.lua: ssl handshake done
Sep 22 11:52:05 c2s40119460 debug   Client sent opening <stream:stream> to 
conference.siacs.eu
Sep 22 11:52:05 c2s40119460 debug   Sent reply <stream:stream> to client
Sep 22 11:52:05 general error   Top-level error, please report:
/usr/lib64/prosody/core/usermanager.lua:104: attempt to index field 'users' (a 
nil value)
Sep 22 11:52:05 general error   
stack traceback:
    [C]: in function 'parse'
    /usr/lib64/prosody/util/xmppstream.lua:255: in function 'feed'
    /usr/lib64/prosody/modules/mod_c2s.lua:230: in function 'data'
    /usr/lib64/prosody/modules/mod_c2s.lua:252: in function 'readbuffer'
    /usr/lib64/prosody/net/server_select.lua:848: in function </usr/lib64/prosody/net/server_select.lua:830>
    [C]: in function 'xpcall'
    /usr/lib64/prosody/../../bin/prosody:373: in function 'loop'
    /usr/lib64/prosody/../../bin/prosody:403: in main chunk
    [C]: at 0x7fdf26e3ba90

and then continues running like normal. I'm not entirely sure if this is the 
exact same client doing the same thing but the IP address that causes this is 
in the same range.

Original comment by dan...@gultsch.de on 22 Sep 2014 at 9:51

[GoogleCodeExporter]

Ah, perfect, thanks! Don't worry about the packet capture - I think this was 
likely the issue.

What is the output of:

 lua -e 'print(require"luaevent"._VERSION)'

and what architecture is your system?

Original comment by MWild1 on 22 Sep 2014 at 11:34

[GoogleCodeExporter]

luaevent is at version 0.4.0

I'm running a 64bit version of gentoo in a virtual machine

Original comment by dan...@gultsch.de on 22 Sep 2014 at 11:40

[GoogleCodeExporter]

Hi, sorry I lost track of this issue. Are you still experiencing problems?

If possible, please upgrade to luaevent 0.4.3 or higher, as a number of issues 
have been fixed since 0.4.0.

Original comment by MWild1 on 20 Feb 2015 at 4:01