Contacts serialization error

[nanomad]

This happens while thruk tries to retrieve contacts info via livestatus

[1424087756] ERROR: [broker-master] [Livestatus] Unexpected error during process of request 'GET contacts\nColumns: name alias email pager service_notification_period host_notification_period\nOutputFormat: json\nResponseHeader: fixed16\n\n' : <object object at 0x7f3dfc07d280> is not JSON serializable
[1424087756] ERROR: [broker-master] [Livestatus] Back trace of this exception: Traceback (most recent call last):
  File "/var/lib/shinken/modules/livestatus/livestatus_obj.py", line 75, in handle_request
    return self.handle_request_and_fail(data)
  File "/var/lib/shinken/modules/livestatus/livestatus_obj.py", line 132, in handle_request_and_fail
    output, keepalive = query.process_query()
  File "/var/lib/shinken/modules/livestatus/livestatus_query.py", line 283, in process_query
    return self.response.respond()
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 147, in respond
    responselength = 1 + self.get_response_len() # 1 for the final '\n'
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 142, in get_response_len
    if isinstance(rsp, LiveStatusListResponse)
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 83, in total_len
    for generated_data in value:
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 278, in make_live_data_generator
    for value in self.make_live_data_generator2(result, columns, aliases):
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 263, in make_live_data_generator2
    yield row_handler(self, l, line_nr)
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 183, in _json_end_row
    return (',' if line_nr else '') + dumps(row)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 354, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 262, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 340, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 239, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <object object at 0x7f3dfc07d280> is not JSON serializable

Also, running:

echo -e "GET contacts\n\n" | netcat localhost 50000

Results in

address1;address2;address3;address4;address5;address6;alias;can_submit_commands;custom_variable_names;custom_variable_values;custom_variables;email;host_notification_period;host_notifications_enabled;in_host_notification_period;in_service_notification_period;modified_attributes;modified_attributes_list;name;pager;service_notification_period;service_notifications_enabled
none;none;none;none;none;none;none;1;;;;shinken@localhost;<object object at 0x7f3dfc07d280>;1;0;0;0;;admin;0600000000;<object object at 0x7f3dfc07d280>;1
none;none;none;none;none;none;none;0;;;;guest@localhost;<object object at 0x7f3dfc07d260>;1;0;0;0;;guest;none;<object object at 0x7f3dfc07d260>;1
Internal server error: UnboundLocalError("local variable 'output' referenced before assignment",)

This happens with:

• shinken 2.2 (pip install shinken)
• livestatus 1.2
• logstore-sqlite 1.4.1
• python 2.7.6

[mpedeupe]

I am also seeing this issue.

To add some more data to this issue, I added a debug line before:

 File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 183, in _json_end_row
    return (',' if line_nr else '') + dumps(row)

It produces the following output:

[1424984369] INFO: [broker-master] row = ['name', 'alias', 'email', 'pager', 'service_notification_period', 'host_notification_period']
[1424984369] INFO: [broker-master] row = [u'admin', 'none', u'shinken@localhost', u'0600000000', <object object at 0x7f3e9b74aaa0>, <object object at 0x7f3e9b74aaa0>]

So it looks like Thruk is querying the service_notification_period and host_notification_period when it is generating the contact list. Those are both objects that get passed to the json serializer and that causes the error.

[david-guenault]

Here is a patch to address this bug. Can you validate that it is working ?

patch content

--- mapping.py.orig 2015-04-04 09:55:11.586885213 +0200
+++ mapping.py  2015-04-04 09:55:48.633657815 +0200
@@ -1726,7 +1726,7 @@
         },
         'host_notification_period': {
             'description': 'The time period in which the contact will be notified about host problems',
-            'function': lambda item, req: item.host_notification_period,
+            'function': lambda item, req: item.host_notification_period.get_name(),
         },
         'host_notifications_enabled': {
             'description': 'Whether the contact will be notified about host problems in general (0/1)',
@@ -1763,7 +1763,7 @@
         },
         'service_notification_period': {
             'description': 'The time period in which the contact will be notified about service problems',
-            'function': lambda item, req: item.service_notification_period,
+            'function': lambda item, req: item.service_notification_period.get_name(),
         },
         'service_notifications_enabled': {
             'description': 'Whether the contact will be notified about service problems in general (0/1)',

how to apply

create the patch file (say file.patch for exemple) and run the following command

patch < file.patch

[david-guenault]

Patch submited as PR : https://github.com/shinken-monitoring/mod-livestatus/pull/51

[mpedeupe]

This patch does fix the error condition that had been introduced and the livestatus module now returns results consistent to the previous version. While this has fixed the crashing issue, it doesn't appear to actually return any valid data. In my tests, the time period names are always returned as nothing. I was expecting time period names like 24x7, etc...

[david-guenault]

did not notice that. Sorry i reopen until it is fixed.

[david-guenault]

Confirmed as a livestatus bug since contacts notification periods are handled corectly internaly (from a shinken point of view). Need to dig a little ....

[ryepup]

spent some time trying to track this down, couldn't figure out where the host_notification_period was coming from. Traced it back to the input to manage_initial_contact_status_brok, but I couldn't find who call calls that

[gst]

@ryepup

manage_initial_contact_status_brok (and any other manage_XXX_brok) is (indirectly) called by the broker daemon :

there : https://github.com/naparuba/shinken/blob/master/shinken/daemons/brokerdaemon.py#L272 :

        for mod in self.modules_manager.get_internal_instances():
            try:
                mod.manage_brok(b)

and the module manage_brok is defined here : https://github.com/naparuba/shinken/blob/master/shinken/basemodule.py#L248 :

    def manage_brok(self, brok):
        """Request the module to manage the given brok.
        There a lot of different possible broks to manage.
        """
        manage = getattr(self, 'manage_' + brok.type + '_brok', None)
        if manage:
            # Be sure the brok is prepared before call it
            brok.prepare()
            return manage(brok)