unrecorevable error in livestatus module

[Seb-Solon]

Comes from : naparuba/shinken#1254

Hi, We set Apache (Thruk) and Shinken in en_US.UTF-8, and added a comment with an accent in Thruk. The result is a crash of the livestatus module

Maybe it is related with this issue: * naparuba/shinken#628 Fix utf8 research bug in LiveStatus broker module naparuba/shinken#634

[dgilm]

Confirmed:

==> /var/log/shinken/brokerd.log <==
2014-09-24 10:01:19,494 [1411545679] Error :   [Livestatus] Back trace of this exception: Traceback (most recent call last):
  File "/var/lib/shinken/modules/livestatus/livestatus_obj.py", line 52, in handle_request
    return self.handle_request_and_fail(data)
  File "/var/lib/shinken/modules/livestatus/livestatus_obj.py", line 119, in handle_request_and_fail
    response.format_live_data(result, query.columns, query.aliases)
  File "/var/lib/shinken/modules/livestatus/livestatus_response.py", line 187, in format_live_data
    self.output = dumps(lines)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 286, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 226, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 296, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 5: invalid continuation byte

[dgilm]

I know that this is not the right solution (there are more codifications than latin-1), but for your information, now I can use livestatus with thruk making the following change:

--- livestatus_response.py.orig 2014-09-24 10:11:44.524597684 +0200
+++ livestatus_response.py  2014-09-24 14:13:34.885018044 +0200
@@ -184,7 +184,7 @@
                 else:
                     lines.insert(0, columns)
             if self.outputformat == 'json':
-                self.output = dumps(lines)
+                self.output = dumps(lines, encoding='latin-1')
             else:
                 self.output = str(lines)

[naparuba]

We should focus on utf8 only I think. It should be up to UIs to be sure they manage encoding. We have to choice for one standard encoding, and I think it's utf8 that must be choose, because it's the defacto standard, even if it means droping some characters if the UI don't follow it (but at leat it don't break the module).

On Wed, Sep 24, 2014 at 2:15 PM, David Gil notifications@github.com wrote:

I know that this is not the right solution (there are more codifications than latin-1), but for your information, now I can use livestatus making the following change:

--- livestatus_response.py.orig 2014-09-24 10:11:44.524597684 +0200 +++ livestatus_response.py 2014-09-24 14:13:34.885018044 +0200 @@ -184,7 +184,7 @@ else: lines.insert(0, columns) if self.outputformat == 'json':

• self.output = dumps(lines)
• self.output = dumps(lines, encoding='latin-1') else: self.output = str(lines)

— Reply to this email directly or view it on GitHub https://github.com/shinken-monitoring/mod-livestatus/issues/33#issuecomment-56660850 .

[dgilm]

Livestatus supports both utf8 and latin1 encodings. https://mathias-kettner.de/checkmk_livestatus.html#Character%20encoding

Maybe adding a module option (_dataencoding set to utf8 by default) to the livestatus module like the described in the previous link, might be useful for 'latin' users and it's quite easy to implement. In fact, the decode error only ocurrs using json/python.

[dgilm]

I think we can implement a similiar aproach: shinken-monitoring/mod-import-mysql#7

Detect a different encoding than 'utf8' and encode/decode again to utf8 to avoid this kind of errors.

What do you think guys?