Socket send buffer limitation

[mohierf]

We (@ddurieux and I) noticed problems with Livestatus and Thruk for large requests. For instance, it is possible to "GET hosts" up to 100 hosts but not over it ... 101 hosts make Thruk hangs !

After some testing, we found that :

1/ The underlying system has a socket buffer suze limitation when sending data to remote host (eg. 32768 bytes).

2/ The livestatus broker is sending ouput data without taking care about effective data sent:

                                try:
                                    s.send(output)
                                    self.write_protocol('', output)
                                except Exception, e:
                                    pass

3/ All the output data are not sent and then Thruk remains waiting for missing data.

One should test how many bytes have been sent and then attempt to deliver the remaining data ... I will make a test on my configuration.

[naparuba]

+1, we should try a while True like on this example: https://docs.python.org/2/howto/sockets.html#using-a-socket

and os we don't need to change the buffer size :)

On Fri, Sep 19, 2014 at 11:36 AM, Frédéric MOHIER notifications@github.com wrote:

We (@ddurieux https://github.com/ddurieux and I) noticed problems with Livestatus and Thruk for large requests. For instance, it is possible to "GET hosts" up to 100 hosts but not over it ... 101 hosts make Thruk hangs !

After some testing, we found that :

1/ The underlying system has a socket buffer suze limitation when sending data to remote host (eg. 32768 bytes).

2/ The livestatus broker is sending ouput data without taking care about effective data sent:

                            try:
                                s.send(output)
                                self.write_protocol('', output)
                            except Exception, e:
                                pass

3/ All the output data are not sent and then Thruk remains waiting for missing data.

One should test how many bytes have been sent and then attempt to deliver the remaining data ... I will make a test on my configuration.

— Reply to this email directly or view it on GitHub https://github.com/shinken-monitoring/mod-livestatus/issues/37.

[mohierf]

Maximum send buffer, on Free BSD:

   sysctl -a net.inet.tcp.sendspace

Maximum send buffer, on Linux Debian:

   cat /proc/sys/net/core/wmem_max

This comment may be helpful ... for me, it is :+1:

[mohierf]

Test en cours ;-)

[mohierf]

Success !

@naparuba : for such a bug, you win a tuto about Python socket ;-) You owe me a beer and for @ddurieux it will be a full truck of Coca ... he lost his nerves on this problem :-)

I make some more tests and then I will commit ...

                                try:
                                    totalsent = 0
                                    while totalsent < len(output):
                                        sent = s.send(output[totalsent:])
                                        if sent == 0:
                                            raise RuntimeError("socket connection broken")
                                        self.write_protocol('', output[totalsent:], sent)
                                        totalsent = totalsent + sent

                                except Exception, e:
                                    if self.debug_queries:
                                        print "EXCEPTION: %s \n\n\n" % str(e)
                                    pass

[naparuba]

Cool congrats :)

On Fri, Sep 19, 2014 at 12:33 PM, Frédéric MOHIER notifications@github.com wrote:

Success !

@naparuba https://github.com/naparuba : for such a bug, you win a tuto about Python socket ;-) You owe me a beer and for @ddurieux https://github.com/ddurieux it will be a full truck of Coca ... he lost his nerves on this problem :-)

I make some more tests and then I will commit ...

                            try:
                                totalsent = 0
                                while totalsent < len(output):
                                    sent = s.send(output[totalsent:])
                                    if sent == 0:
                                        raise RuntimeError("socket connection broken")
                                    self.write_protocol('', output[totalsent:], sent)
                                    totalsent = totalsent + sent

                            except Exception, e:
                                if self.debug_queries:
                                    print "EXCEPTION: %s \n\n\n" % str(e)
                                pass

— Reply to this email directly or view it on GitHub https://github.com/shinken-monitoring/mod-livestatus/issues/37#issuecomment-56161389 .