Object of type bytes is not JSON serializable

[Sc0th]

Hi,

Would really appreciative if someone can assist me with this. (mqttwarn 0.34.1.post10+g6a24a8a.d20231004)

I have a simple feed to collect all mqtt data and feed to graylog

targets = {
    'graylog' : [ "post", "http://graylog.<my domain>k:12201/gelf", None, None, None ]
  }

[#]
targets = http:graylog
template = graylog.json

Template looks like this:

{% set data = {
        'host'          : 'mqtt',
        '_payload'      : payload,
        'short_message' : topic,
        }
        %}
{{ data | jsonify }}

This has all worked for years, however with the latest update I am seeing this error, and the message is not being pushed to graylog

2023-10-06 11:22:20,521 ERROR    [mqttwarn.core             ] Rendering template failed: graylog.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/mqttwarn/core.py", line 526, in process_job
    text = render_template(template, transform_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mqttwarn/core.py", line 99, in render_template
    text = template.render(data)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/graylog.json", line 16, in top-level template code
    {{ data | jsonify }}
  File "/usr/local/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

However if I remove the payload line from the template, like so:

{% set data = {
        'host'          : 'mqtt',
        'short_message' : topic,
        }
        %}
{{ data | jsonify }}

It springs back to life, although of course I then don't get the payload in the graylog record, which makes the data pretty useless.

Would appreciate any pointers, I have spent waaaay to long on this and making no progress.

[amotl]

Dear @Sc0th,

thanks for your report. When talking about that this "worked for years", may I ask if you are by chance referring to a Python 2 installation of mqttwarn? That might give us some hints already about in which area something might have gone south.

Other than this, may I ask you to submit a full mqttwarn.ini configuration, and maybe a corresponding mosquitto_pub command to be able to reproduce your scenario easily? This will tremendously help to discover and resolve the problem, hopefully quickly.

With kind regards, Andreas.

[ScothularNetwork]

Hi @amotl

Thanks for getting back to me! I backed out to version 0.29.0 and all is well so suspect you are correct.

Requested details below:

config.ini

# -*- coding: utf-8 -*-
# (c) 2014-2018 The mqttwarn developers
#
# mqttwarn example configuration file "mqttwarn.ini"
#

; ------------------------------------------
;             Base configuration
; ------------------------------------------

[defaults]

; ----
; MQTT
; ----

hostname     = 'mqtt.<my domain>'
port         = 1883
username     = None
password     = None
clientid     = 'mqttwarn'
lwt          = 'clients/mqttwarn'
skipretained = False
cleansession = False

# MQTTv31 = 3   (default)
# MQTTv311 = 4
protocol     = 3

; -------
; Logging
; -------

; Send log output to STDERR
;logfile   = 'stream://sys.stderr'

; Send log output to file
logfile   = '/data/logs/mqttwarn.log'

; one of: CRITICAL, DEBUG, ERROR, INFO, WARN
loglevel  = DEBUG

;logformat = '%(asctime)-15s %(levelname)-8s [%(name)-25s] %(message)s'

; --------
; Services
; --------

; path to file containing self-defined functions for formatmap and datamap
;functions = 'samplefuncs.py'

; name the service providers you will be using.
launch    = file, http
directory = '/'

; -------
; Targets
; -------

;[config:file]
;append_newline = True
;targets = {
;    'all'       : ['/data/logs/all.log'],
;    }

[config:http]
targets = {
    'graylog' : [ "post", "http://graylog.<my domain>:12201/gelf", None, None, None ]
  }

[config:file]
append_newline = False
overwrite = True
targets = {
    'changedetection'       : ["/data/{topic}.dat"],
    }

; ------------------------------------------
;                  Basic
; ------------------------------------------

[#]
#;targets = file:all, http:graylog
targets = http:graylog
template = graylog.json

[changedetect/#]
targets = file:changedetection
;template = changedetection.j2

mqtt publish:

mosquitto_pub -h mqtt -t scothnet/rack/racktemp -m 60

error in debug log:

2023-10-10 16:11:27,897 DEBUG    [mqttwarn.core             ] Message received on scothnet/rack/racktemp: 60
2023-10-10 16:11:27,897 DEBUG    [mqttwarn.core             ] Section [#] matches message on scothnet/rack/racktemp, processing it
2023-10-10 16:11:27,897 DEBUG    [mqttwarn.core             ] Decoding JSON failed: 'int' object is not iterable. payload=60
2023-10-10 16:11:27,897 DEBUG    [mqttwarn.core             ] Message on scothnet/rack/racktemp going to http:graylog
2023-10-10 16:11:27,897 DEBUG    [mqttwarn.core             ] New `http:graylog' job: scothnet/rack/racktemp
2023-10-10 16:11:27,897 DEBUG    [mqttwarn.core             ] Processor #0 is handling: `http' for graylog
2023-10-10 16:11:27,898 ERROR    [mqttwarn.core             ] Rendering template failed: graylog.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/mqttwarn/core.py", line 526, in process_job
    text = render_template(template, transform_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mqttwarn/core.py", line 99, in render_template
    text = template.render(data)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/graylog.json", line 17, in top-level template code
    {{ data | jsonify }}
  File "/usr/local/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
2023-10-10 16:11:27,898 INFO     [mqttwarn.core             ] Invoking service plugin for `http'
2023-10-10 16:11:27,898 DEBUG    [mqttwarn.services.http    ] *** MODULE=/usr/local/lib/python3.11/site-packages/mqttwarn/services/http_urllib.py: service
=http, target=graylog
2023-10-10 16:11:27,898 DEBUG    [mqttwarn.services.http    ] before send
2023-10-10 16:11:27,901 DEBUG    [mqttwarn.core             ] Job queue has 0 items to process

template (graylog.json):

{% set data = {
        'version'       : '1.1',
        'host'          : 'mqtt',
        'short_message' : topic,
        '_payload'      : payload,
        }
        %}
{{ data | jsonify }}

Cheers!

[amotl]

GH-686 fixes the flaw which lead to the error. mqttwarn 0.35.0 has been released, including the fix.

[ScothularNetwork]

Confirmed all working again with 0.35 many thanks, super quick!

[amotl]

Wonderful. Thanks for your reply. Closing this.