livestatus can't start listener

[silvergroover]

Hi all,

I just install shinken with the livestatus module in the aim to use it with Thruk. I configure the broker module : -bash-4.1$ grep modules broker-master.cfg

Interesting modules that can be used:

modules             webui,livestatus

I configure the livestatus module (I let the default) : define module { module_name livestatus module_type livestatus host * ; * = listen on all configured IP addresses port 50000 ; port to listen

socket /usr/local/shinken/var/rw/live ; If a Unix socket is required

## Available modules:
# - logstore-sqlite: send historical logs to a local sqlite database
# - logstore-mongodb: send historical logs to a mongodb database
# - logstore-null : send historical logs to a black hole
modules         logstore-sqlite
#debug           /tmp/ls.debug   ; Enable only for debugging this module
#debug_queries   0   ; Set to 1 to dump queries/replies too (very verbose)

}

When I start the broker, I have this error in the log : [1428051481] INFO: [broker-master] Setting the module livestatus to restart [1428051481] INFO: [broker-master] Trying to init module: livestatus [1428051482] INFO: [broker-master] Trying to init module: livestatus [1428051482] INFO: [broker-master] [Livestatus Broker] Init of the Livestatus 'livestatus' [1428051482] INFO: [broker-master] I'm stopping module u'livestatus' (pid=19228) [1428051482] INFO: [broker-master] Starting external process for instance livestatus [1428051482] INFO: [broker-master] livestatus is now started ; pid=19886 [1428051483] INFO: [broker-master] [livestatus[19886]]: Now running.. [1428051483] INFO: [broker-master] Modules directory: /var/lib/shinken/modules [1428051483] INFO: [broker-master] [Logstore SQLite] Get an LogStore Sqlite module for plugin logstore-sqlite [1428051484] INFO: [broker-master] Trying to init module: logstore-sqlite [1428051484] INFO: [broker-master] I correctly loaded the modules: [logstore-sqlite] [1428051484] INFO: [broker-master] [Livestatus Broker] Go run [1428051484] INFO: [broker-master] [Livestatus Broker] Livestatus query thread started [1428051485] ERROR: [broker-master] the module livestatus just crash! Please look at the traceback: [1428051485] ERROR: [broker-master] Traceback (most recent call last): File "/var/lib/shinken/modules/livestatus/module.py", line 243, in main self.do_main() File "/var/lib/shinken/modules/livestatus/module.py", line 307, in do_main self.manage_lql_thread() File "/var/lib/shinken/modules/livestatus/module.py", line 516, in manage_lql_thread self.create_listeners() File "/var/lib/shinken/modules/livestatus/module.py", line 440, in create_listeners server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) File "", line 1, in setsockopt error: [Errno 92] Protocol not available

[1428051487] ERROR: [broker-master] The external module livestatus goes down unexpectedly!

I tried to enable the socket parameter but it's always the same issue ... Do you have any idea ?

[david-guenault]

Hello, could you provide more informations Which linux distro Which version of livestatus (from shinken.io or from github ?) Full configuration of broker and livestatus

Thx for reporting

[gst]

According to this :

File "/var/lib/shinken/modules/livestatus/module.py", line 440, in create_listeners
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
File "", line 1, in setsockopt
error: [Errno 92] Protocol not available

I'm guessing it's your OS which doesn't probably support SO_REUSEPORT ; which can effectively be. A proper fix would be certainly to make this setsockopt() call optional and/or "failable".

waiting your infos about your distribution, versions, ..

gst.

[gst]

I'm guessing it's your OS which doesn't probably support SO_REUSEPORT ; which can effectively be.

looks like it is well that : some refs:

http://bugs.python.org/issue19901 https://hg.python.org/cpython/rev/9791c5d55f52 etc..

so the easy fix is to wrap the setsockopt() call inside a try: .. except OSError: block (and log a warning in case it fail)

[silvergroover]

Hello,

I'm using a RedHat RHEL 6.4 x86_64, kernel : 2.6.32-358.6.2.el6.x86_64 with shinken 2.2

I'm working with the last version of the livestatus module available on Github.

I just try your fix in my module.py file like this : if hasattr(socket, 'SO_REUSEPORT'):

server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

            try:
                server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
            except OSError as err:
                logger.warning("Can't set SO_REUSEPORT on socket: %s, "
                               "is it an old kernel ?", err)
        server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

and I always have the same message in the log : [1428338081] INFO: [broker-master] Setting the module livestatus to restart [1428338081] INFO: [broker-master] Trying to init module: livestatus [1428338082] INFO: [broker-master] Trying to init module: livestatus [1428338082] INFO: [broker-master] [Livestatus Broker] Init of the Livestatus 'livestatus' [1428338082] INFO: [broker-master] I'm stopping module u'livestatus' (pid=20049) [1428338082] INFO: [broker-master] Starting external process for instance livestatus [1428338083] INFO: [broker-master] livestatus is now started ; pid=20681 [1428338083] INFO: [broker-master] [livestatus[20681]]: Now running.. [1428338083] INFO: [broker-master] Modules directory: /var/lib/shinken/modules [1428338083] INFO: [broker-master] [Logstore SQLite] Get an LogStore Sqlite module for plugin logstore-sqlite [1428338084] INFO: [broker-master] Trying to init module: logstore-sqlite [1428338084] INFO: [broker-master] I correctly loaded the modules: [logstore-sqlite] [1428338084] INFO: [broker-master] [Livestatus Broker] Go run [1428338084] INFO: [broker-master] [Livestatus Broker] Livestatus query thread started [1428338085] ERROR: [broker-master] the module livestatus just crash! Please look at the traceback: [1428338085] ERROR: [broker-master] Traceback (most recent call last): File "/var/lib/shinken/modules/livestatus/module.py", line 243, in main self.do_main() File "/var/lib/shinken/modules/livestatus/module.py", line 307, in do_main self.manage_lql_thread() File "/var/lib/shinken/modules/livestatus/module.py", line 521, in manage_lql_thread self.create_listeners() File "/var/lib/shinken/modules/livestatus/module.py", line 442, in create_listeners server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) File "", line 1, in setsockopt error: [Errno 92] Protocol not available

[1428338087] ERROR: [broker-master] The external module livestatus goes down unexpectedly!

this is my broker and livestatus config file : define broker { broker_name broker-master address localhost port 7772 spare 0

## Optional
manage_arbiters     1   ; Take data from Arbiter. There should be only one
                        ; broker for the arbiter.
manage_sub_realms   1   ; Does it take jobs from schedulers of sub-Realms?
timeout             3   ; Ping timeout
data_timeout        120 ; Data send timeout
max_check_attempts  3   ; If ping fails N or more, then the node is dead
check_interval      60  ; Ping node every N seconds

## Modules
# Default: None
# Interesting modules that can be used:
# - simple-log              = just all logs into one file
# - livestatus              = livestatus listener
# - tondodb-mysql           = NDO DB support
# - npcdmod                 = Use the PNP addon
# - graphite                = Use a Graphite time series DB for perfdata
# - webui                   = Shinken Web interface
# - glpidb                  = Save data in GLPI MySQL database
modules             webui,livestatus

# Enable https or not
use_ssl               0
# enable certificate/hostname check, will avoid man in the middle attacks
hard_ssl_name_check   0

## Advanced
realm   All

}

define module { module_name livestatus module_type livestatus host * ; * = listen on all configured IP addresses port 50000 ; port to listen

socket /usr/local/shinken/var/rw/live ; If a Unix socket is required

## Available modules:
# - logstore-sqlite: send historical logs to a local sqlite database
# - logstore-mongodb: send historical logs to a mongodb database
# - logstore-null : send historical logs to a black hole
modules         logstore-sqlite
#debug           /tmp/ls.debug   ; Enable only for debugging this module
#debug_queries   0   ; Set to 1 to dump queries/replies too (very verbose)

}

Thanks for your support

[gst]

@silvergroover what ? damn this should have not..

can you try that :

            ...
            server.setblocking(0)
            if hasattr(socket, 'SO_REUSEPORT'):
                try:
                    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
                except Exception as err:
                    logger.warning("DEBUG: %s, ", type(err).mro())
                    logger.warning("Can't set SO_REUSEPORT on socket: %s, "
                                   "is it an old kernel ?", err)

and give me the "DEBUG" line you would get in logs... ?

thx.

[silvergroover]

humm.. ok, I modified the code :-1: if hasattr(socket, 'SO_REUSEPORT'):

server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

            try:
                server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
            except Exception as err:
                logger.warning("DEBUG: %s, ", type(err).mro())
                logger.warning("Can't set SO_REUSEPORT on socket: %s, "
                               "is it an old kernel ?", err)
        server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

and restarted the broker ... and it works now ! but I don't understand why o_o

netstat -an | grep 50000

tcp 0 0 0.0.0.0:50000 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:9882 127.0.0.1:50000 TIME_WAIT tcp 0 0 127.0.0.1:9886 127.0.0.1:50000 TIME_WAIT tcp 0 0 127.0.0.1:9892 127.0.0.1:50000 TIME_WAIT tcp 0 0 127.0.0.1:9890 127.0.0.1:50000 TIME_WAIT tcp 0 0 127.0.0.1:9887 127.0.0.1:50000 TIME_WAIT tcp 0 0 127.0.0.1:9894 127.0.0.1:50000 TIME_WAIT

the log of the broker : [1428340617] INFO: [broker-master] [Livestatus Broker] Init of the Livestatus 'livestatus' [1428340617] INFO: [broker-master] Starting external module livestatus [1428340617] INFO: [broker-master] Starting external process for instance livestatus [1428340617] INFO: [broker-master] livestatus is now started ; pid=18199 [1428340617] INFO: [broker-master] [livestatus[18199]]: Now running.. [1428340617] INFO: [broker-master] Modules directory: /var/lib/shinken/modules [1428340617] INFO: [broker-master] [Logstore SQLite] Get an LogStore Sqlite module for plugin logstore-sqlite [1428340618] INFO: [broker-master] Trying to init module: logstore-sqlite [1428340618] INFO: [broker-master] I correctly loaded the modules: [logstore-sqlite] [1428340618] INFO: [broker-master] [Livestatus Broker] Go run [1428340618] INFO: [broker-master] [Livestatus Broker] Livestatus query thread started [1428340618] WARNING: [broker-master] DEBUG: [<class 'socket.error'>, <type 'exceptions.IOError'>, <type 'exceptions.EnvironmentError'>, <type 'exceptions.StandardError'>, <type 'exceptions.Exception'>, <type 'exceptions.BaseException'>, <type 'object'>], [1428340618] WARNING: [broker-master] Can't set SO_REUSEPORT on socket: [Errno 92] Protocol not available, is it an old kernel ? [1428340618] INFO: [broker-master] [Livestatus Broker] listening on tcp port: 50000 [1428340618] INFO: [broker-master] [Livestatus Broker] listening on unix socket: /opt/shinken/live [1428340620] INFO: [broker-master] [Logstore SQLite] at Mon Apr 6 17:17:00 2015 we rotate the database file [1428340620] INFO: [broker-master] [Logstore SQLite] next rotation at Tue Apr 7 00:05:00 2015 [1428340621] INFO: [broker-master] Connection OK to the scheduler scheduler-master [1428340621] INFO: [broker-master] Connection OK to the poller poller-master [1428340621] INFO: [broker-master] Connection OK to the reactionner reactionner-master [1428340626] INFO: [broker-master] Connection OK to the receiver receiver-master

[gst]

and restarted the broker ... and it works now ! but I don't understand why o_o

hmm.. are you sure you had correctly restarted the broker the try before ? otherwise I may understand it, but I need more infos (well I need one specific info):

[1428340618] WARNING: [broker-master] DEBUG: [, , , , , , ],

damn, it does not have given what I wanted :s ("[, , , , , , ]" should not be that :s well there should be some text between the commas .. :s )

So, can you retry the same but with logger.warning("DEBUG: %s + %s " % (str(type(err)), str(type(err).mro())) instead of the previous one ?

( if I don't get what I want I'll then keep the except Exception as it is right now and that's all )

[silvergroover]

yes i'm sure to have restarted the boker each time (you can see that the number of the line in the log file has changed). Below the log with the new code : [1428343767] INFO: [broker-master] [Livestatus Broker] Init of the Livestatus 'livestatus' [1428343767] INFO: [broker-master] Starting external module livestatus [1428343767] INFO: [broker-master] Starting external process for instance livestatus [1428343767] INFO: [broker-master] livestatus is now started ; pid=11858 [1428343767] INFO: [broker-master] [livestatus[11858]]: Now running.. [1428343767] INFO: [broker-master] Modules directory: /var/lib/shinken/modules [1428343767] INFO: [broker-master] [Logstore SQLite] Get an LogStore Sqlite module for plugin logstore-sqlite [1428343768] INFO: [broker-master] Trying to init module: logstore-sqlite [1428343768] INFO: [broker-master] I correctly loaded the modules: [logstore-sqlite] [1428343768] INFO: [broker-master] [Livestatus Broker] Go run [1428343768] INFO: [broker-master] [Livestatus Broker] Livestatus query thread started [1428343768] WARNING: [broker-master] DEBUG: <class 'socket.error'> + [<class 'socket.error'>, <type 'exceptions.IOError'>, <type 'exceptions.EnvironmentError'>, <type 'exceptions.StandardError'>, <type 'exceptions.Exception'>, <type 'exceptions.BaseException'>, <type 'object'>] [1428343768] WARNING: [broker-master] Can't set SO_REUSEPORT on socket: [Errno 92] Protocol not available, is it an old kernel ? [1428343768] INFO: [broker-master] [Livestatus Broker] listening on tcp port: 50000 [1428343768] INFO: [broker-master] [Livestatus Broker] listening on unix socket: /opt/shinken/live [

[silvergroover]

oh ... it seems that I can't post the correctly the debug line, the real content is this one : DEBUG: <class 'socket.error'> + [<class 'socket.error'>, <type 'exceptions.IOError'>, <type 'exceptions.EnvironmentError'>, <type 'exceptions.StandardError'>, <type 'exceptions.Exception'>, <type 'exceptions.BaseException'>, <type 'object'>

[silvergroover]

humm ... I try without the special characters : DEBUG: class 'socket.error' + class 'socket.error', type 'exceptions.IOError', type 'exceptions.EnvironmentError', type 'exceptions.StandardError', type 'exceptions.Exception', type 'exceptions.BaseException', type 'object'

[gst]

Great thanks ! :)