Windows10 - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

[symonjim]

Followed steps to install mqttwarn on my W10 machine. Everything seemed to work including mqttwarn make-xxx commands. Finally tried running mqttwarn by itself and got strange error messages. Any help?

pip install --upgrade mqttwarn
mqttwarn make-config > mqttwarn.ini
cat mqttwarn.ini
mqttwarn make-samplefuncs > samplefuncs.py

mqttwarn
Traceback (most recent call last):
  File "D:\Program Files (D)\Python37\Scripts\mqttwarn-script.py", line 33, in <module>
    sys.exit(load_entry_point('mqttwarn==0.28.1', 'console_scripts', 'mqttwarn')())
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 93, in run
    run_mqttwarn()
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 132, in run_mqttwarn
    config = load_configuration(name=scriptname)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 196, in load_configuration
    return Config(configfile, defaults=defaults)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 38, in __init__
    self.read_file(f)
  File "d:\program files (d)\python37\lib\configparser.py", line 717, in read_file
    self._read(f, source)
  File "d:\program files (d)\python37\lib\configparser.py", line 1014, in _read
    for lineno, line in enumerate(fp, start=1):
  File "d:\program files (d)\python37\lib\codecs.py", line 714, in __next__
    return next(self.reader)
  File "d:\program files (d)\python37\lib\codecs.py", line 645, in __next__
    line = self.readline()
  File "d:\program files (d)\python37\lib\codecs.py", line 558, in readline
    data = self.read(readsize, firstline=True)
  File "d:\program files (d)\python37\lib\codecs.py", line 504, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

[amotl]

Dear @symonjim,

thank you for writing in. It looks like that this is a specific issue when mqttwarn tries to open the generated configuration file mqttwarn.ini. Do you have any clues why chr(255) could be the first character in the generated file on Windows 10?

For the sake of getting the whole picture, may I ask if you are using the cmd.exe shell, or PowerShell?

With kind regards, Andreas.

[jpmens]

That sounds a bit like a (broken) BOM. Which text editor have you been using, @symonjim, for editing the mqttwarn.ini file?

[amotl]

Do you have any clues why chr(255) could be the first character in the generated file on Windows?

I see, https://github.com/PyCQA/pylint/issues/1663 has many insights into the problem and also offers a solution.

Windows 10 produces a file with UTF-16 encoding when using the default redirection operator, which causes this problem when trying to read it back with assuming UTF-8.

When using PowerShell, you might try this command to generate the default configuration file:

mqttwarn make-config | Out-File -Encoding utf8 mqttwarn.ini

Another proposed solution in the discussion is to invoke chcp 65001 before running mqttwarn make-config > mqttwarn.ini, using either cmd.exe, or PowerShell.

It would be sweet if you could evaluate both proposed solutions and report back to us, so that we can improve the mqttwarn documentation correspondingly.

[symonjim]

notepad

640 Tanager Lane Chapel Hill, NC 27517 @.*** home 919 869-7877 cell 919 649-6337

On 8/17/2022 12:15 PM, Jan-Piet Mens wrote:

That sounds a bit like a (broken) BOM https://en.wikipedia.org/wiki/Byte_order_mark. Which text editor have you been using, @symonjim https://github.com/symonjim, for editing the |mqttwarn.ini| file?

— Reply to this email directly, view it on GitHub https://github.com/jpmens/mqttwarn/issues/571#issuecomment-1218230907, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVI2GEC6C3BHGFQZBAS47J3VZUF25ANCNFSM56UBXL5Q. You are receiving this because you were mentioned.Message ID: @.***>

[symonjim]

That doesn't seem to help. At the end of the error output it says the following. Could it be that the mqttwarn.ini is missing something at the top?

configparser.MissingSectionHeaderError: File contains no section headers.
file: 'mqttwarn.ini', line: 1
'\ufeff# -*- coding: utf-8 -*-\r\n'

I'll attach my mqttwarn.ini file generated with that suggestion of piping it into Out-File.

You sent the info about a similar issue with Python installation. Installing or using mqttwarn involves use of Python, correct? could it be that my Python installation has the problem rather than the mqttwarn installation?

Thanks,

Jim

Exception

PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main> mqttwarn 
make-config | Out-File -Encoding utf8 mqttwarn.ini
PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main> mqttwarn
Traceback (most recent call last):
   File "D:\Program Files (D)\Python37\Scripts\mqttwarn-script.py", line 
33, in <module>
     sys.exit(load_entry_point('mqttwarn==0.28.1', 'console_scripts', 
'mqttwarn')())
   File "d:\program files 
(d)\python37\lib\site-packages\mqttwarn\commands.py", line 93, in run
     run_mqttwarn()
   File "d:\program files 
(d)\python37\lib\site-packages\mqttwarn\commands.py", line 132, in 
run_mqttwarn
     config = load_configuration(name=scriptname)
   File "d:\program files 
(d)\python37\lib\site-packages\mqttwarn\configuration.py", line 196, in 
load_configuration
     return Config(configfile, defaults=defaults)
   File "d:\program files 
(d)\python37\lib\site-packages\mqttwarn\configuration.py", line 38, in 
__init__
     self.read_file(f)
   File "d:\program files (d)\python37\lib\configparser.py", line 717, 
in read_file
     self._read(f, source)
   File "d:\program files (d)\python37\lib\configparser.py", line 1079, 
in _read
     raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: 'mqttwarn.ini', line: 1
'\ufeff# -*- coding: utf-8 -*-\r\n'

My configuration files

# -*- coding: utf-8 -*-
# (c) 2014-2018 The mqttwarn developers
#
# mqttwarn example configuration file "mqttwarn.ini"
#

; ------------------------------------------
;             Base configuration
; ------------------------------------------

[defaults]

; ----
; MQTT
; ----

hostname     = 'localhost'
port         = 1883
username     = None
password     = None
clientid     = 'mqttwarn'
lwt          = 'clients/mqttwarn'
skipretained = False
cleansession = False

# MQTTv31 = 3   (default)
# MQTTv311 = 4
protocol     = 3

; -------
; Logging
; -------

; Send log output to STDERR
logfile   = 'stream://sys.stderr'

; Send log output to file
;logfile   = 'mqttwarn.log'

; one of: CRITICAL, DEBUG, ERROR, INFO, WARN
loglevel  = DEBUG

;logformat = '%(asctime)-15s %(levelname)-8s [%(name)-25s] %(message)s'

; --------
; Services
; --------

; path to file containing self-defined functions for formatmap and datamap
functions = 'samplefuncs.py'

; name the service providers you will be using.
launch    = file, log

; Publish mqttwarn status information (retained)
status_publish = True
; status_topic = mqttwarn/$SYS

; -------
; Targets
; -------

[config:file]
append_newline = True
targets = {
    'f01'       : ['/tmp/f.01'],
    'log-me'    : ['/tmp/log.me'],
    'mqttwarn'  : ['/tmp/mqttwarn.err'],
    }

[config:log]
targets = {
    'debug'  : [ 'debug' ],
    'info'   : [ 'info' ],
    'warn'   : [ 'warn' ],
    'crit'   : [ 'crit' ],
    'error'  : [ 'error' ]
  }

; special config for 'failover' events
[failover]
targets = log:error, file:mqttwarn

; ------------------------------------------
;                  Basic
; ------------------------------------------

[hello/1]
; echo '{"name": "temperature", "number": 42.42}' | mosquitto_pub -h localhost -t hello/1 -l
targets = log:info
format = u'{name}: {number} => {_dthhmm}'

; ------------------------------------------
;                OwnTracks
; ------------------------------------------

[owntracks-location]
topic = owntracks/+/+
targets = log:info, file:f01
datamap = OwnTracksTopic2Data()
format  = OwnTracksConvert()

[owntracks-battery]
topic = owntracks/+/+
targets = log:info, file:f01
datamap = OwnTracksTopic2Data()
filter  = OwnTracksBattFilter()
format  = {username}'s phone battery is getting low ({batt}%)

; ------------------------------------------
;              Dynamic targets
; ------------------------------------------

[robustness-1]
; even if "foo" is considered an invalid service or
; "log:baz" is considered an invalid service target,
; mqttwarn should keep calm and carry on
topic   = test/robustness-1
targets = foo:bar, log:baz

[topic-targets-dynamic]
; interpolate transformation data values into topic target, example:
; mosquitto_pub -t test/topic-targets-dynamic -m '{"loglevel": "crit", "message": "Nur D├╢ner macht sch├╢ner!"}'
topic   = test/topic-targets-dynamic
format  = Something {loglevel} happened! {message}
targets = log:{loglevel}

[topic-targets-func]
; use functions for computing topic targets, example:
; mosquitto_pub -t test/topic-targets-func -m '{"condition": "sunny", "remark": "This should go to a file"}'
; mosquitto_pub -t test/topic-targets-func -m '{"condition": "rainy", "remark": "This should go to the log"}'
topic   = test/topic-targets-func
format  = Weather conditions changed: It's {condition}. Remark: {remark}
targets = TopicTargetList()

; ------------------------------------------
;              Periodic tasks
; ------------------------------------------

[cron]
; Demonstrate periodic task feature.
; Define a function for publishing your public ip address to the MQTT bus each minute.
; mosquitto_sub -t 'test/ip/#' -v
#publish_public_ip_address = 60; now=true

[symonjim]

Oops. That long file inclusion was not a good idea.

I used notepad to rewrite the ini file making sure it was output utf8. Ran mqttwarn and got the following new error messages.

PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main> mqttwarn
Traceback (most recent call last):
  File "D:\Program Files (D)\Python37\Scripts\mqttwarn-script.py", line 33, in <module>
    sys.exit(load_entry_point('mqttwarn==0.28.1', 'console_scripts', 'mqttwarn')())
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 93, in run
    run_mqttwarn()
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 132, in run_mqttwarn
    config = load_configuration(name=scriptname)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 196, in load_configuration
    return Config(configfile, defaults=defaults)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 106, in __init__
    self.functions = load_functions(functions_file)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\util.py", line 219, in load_functions
    py_mod = imp.load_source(mod_name, filepath)
  File "d:\program files (d)\python37\lib\imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ValueError: source code string cannot contain null bytes
PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main>

[amotl]

I see. Thanks for sharing your progress.

Now, it looks like mqttwarn is loading its configuration file just fine, but croaks at "load_functions" when trying to load the Python code file containing the runtime extension functions.

So, I would recommend to apply the same procedure to the .py file as well and create it manually with an editor of your choice, saving it using the utf-8 charset encoding.

I am confident it will get you started with finally getting mqttwarn running. Good luck with that.

If that succeeds, we can get back to investigating and resolving the original problem with the configuration blueprint scaffolding, which apparently goes south on Windows10.

-- Sent from my mind. This might have been typed on a mobile device, so please excuse my brevity.

[symonjim]

I did that with the .py files. I opened them in notepad and immediately did a "save as" choosing utf8 and overwriting the original. Interestingly all the files I had worked with had said they were going to save as utf8 until I got to samplefuncs.py which said it would save as utf16. I changed that to utf8 and...presto! It works!

PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main> mqttwarn
2022-08-17 18:28:26,596 INFO     [mqttwarn.commands         ] Starting mqttwarn
2022-08-17 18:28:26,597 INFO     [mqttwarn.commands         ] Log level is DEBUG
2022-08-17 18:28:26,597 DEBUG    [mqttwarn.core             ] Trying to load built-in service "file" from "file"
2022-08-17 18:28:26,598 DEBUG    [mqttwarn.core             ] Trying to load service "file" from file "d:\program files (d)\python37\lib\site-packages\mqttwarn\services\file.py"
2022-08-17 18:28:26,599 INFO     [mqttwarn.core             ] Successfully loaded service "file"
2022-08-17 18:28:26,599 DEBUG    [mqttwarn.core             ] Trying to load built-in service "log" from "log"
2022-08-17 18:28:26,600 DEBUG    [mqttwarn.core             ] Trying to load service "log" from file "d:\program files (d)\python37\lib\site-packages\mqttwarn\services\log.py"
2022-08-17 18:28:26,601 INFO     [mqttwarn.core             ] Successfully loaded service "log"
2022-08-17 18:28:26,601 DEBUG    [mqttwarn.core             ] Attempting connection to MQTT broker localhost:1883...
2022-08-17 18:28:26,601 DEBUG    [mqttwarn.core             ] Setting LWT to clients/mqttwarn...
2022-08-17 18:28:28,672 INFO     [mqttwarn.core             ] Publishing status information to mqttwarn/$SYS
2022-08-17 18:28:28,673 INFO     [mqttwarn.core             ] Starting 1 worker threads
2022-08-17 18:28:28,678 DEBUG    [mqttwarn.core             ] Job queue has 0 items to process
2022-08-17 18:28:28,679 DEBUG    [mqttwarn.core             ] Connected to MQTT broker, subscribing to topics...
2022-08-17 18:28:28,680 DEBUG    [mqttwarn.core             ] Cleansession==False; previous subscriptions for clientid mqttwarn remain active on broker
2022-08-17 18:28:28,682 DEBUG    [mqttwarn.core             ] Subscribing to hello/1 (qos=0)
2022-08-17 18:28:28,682 DEBUG    [mqttwarn.core             ] Subscribing to owntracks/+/+ (qos=0)
2022-08-17 18:28:28,683 DEBUG    [mqttwarn.core             ] Subscribing to test/robustness-1 (qos=0)
2022-08-17 18:28:28,683 DEBUG    [mqttwarn.core             ] Subscribing to test/topic-targets-dynamic (qos=0)
2022-08-17 18:28:28,684 DEBUG    [mqttwarn.core             ] Subscribing to test/topic-targets-func (qos=0)

and mosquitto seemed to react. Now I just need to make some adjustments for my setup and tell it what to do when it gets a message.

Thanks for all your help.

Jim

[symonjim]

It worked fine until I added my smtp config and topic to the mqttwarn.ini file. Then it gave me another esoteric error message about int having no length??

I added:

[config:smtp]
server   = 'localhost:25'
sender   = "mqttwarn <symonjim@localhost>"
starttls = False
targets = {
    'symonjim'  : ['symonjim@sybr.net']
    }

and:

[IOTgadgettopic]
topic   = IOTgadgettopic
targets = smtp:symonjim

and got this when it failed:

2022-08-17 22:04:55,000 DEBUG    [mqttwarn.core             ] Successfully loaded service "smtp"
2022-08-17 22:04:55,761 DEBUG    [mqttwarn.core             ] Attempting connection to MQTT broker localhost:1883...
2022-08-17 22:04:55,762 DEBUG    [mqttwarn.core             ] Setting LWT to clients/mqttwarn...
2022-08-17 22:04:57,833 ERROR    [mqttwarn.core             ] Cannot connect to MQTT broker at localhost:1883: object of type 'int' has no len()

hmm...

[amotl]

Hi Jim,

got this when it failed: Cannot connect to MQTT broker at localhost:1883: object of type 'int' has no len()

That's weird, but I am confident that we will also find out about the root cause. The corresponding place in the code where this is happening is:

https://github.com/jpmens/mqttwarn/blob/f4e6e0888a2345bb3b0bf7a490081b1e48185609/mqttwarn/core.py#L629-L634

I am assuming that again something might be going wrong with correctly parsing and interpreting the configuration file on your end. I can confirm that, when inserting the snippets you shared into the mqttwarn.ini on my machine, I don't observe such an error.

Currently, I have no idea what exactly might be going wrong on your end. Two things come to mind how to proceed:

a) Maybe the culprit is different newline characters on Windows after saving the configuration file? It is hard to believe, because you already edited and saved it manually beforehand, right?

b) Can you verify everything works flawlessly again, when you remove the snippets you added?

With kind regards, Andreas.

[symonjim]

I found it. Had nothing to do with the smtp stuff I had inserted. I had also put in these lines without single quotes. Single quotes fixed it.

Now I just need to figure out how to specify qos=1.

Fails with that odd error message:

username = mqttwarn password = 123456789

Works fine:

username = 'mqttwarn' password = '123456789'

[symonjim]

Go that. Now this. I guess I need to just plug away at it and not keep clogging up this site with each of my steps.

2022-08-18 10:45:37,047 DEBUG    [mqttwarn.core             ] Message received on IOTgadgettopic:   "This is MQTTX1 with gadget topic with another try"
2022-08-18 10:45:37,048 DEBUG    [mqttwarn.core             ] Section [IOTgadgettopic] matches message on IOTgadgettopic. Processing...
2022-08-18 10:45:37,051 DEBUG    [mqttwarn.core             ] Cannot decode JSON object, payload=  "This is MQTTX1 with gadget topic with another try": dictionary update sequence element #0 has length 1; 2 is required
2022-08-18 10:45:37,053 DEBUG    [mqttwarn.core             ] Message on IOTgadgettopic going to smtp:mqttwarn
2022-08-18 10:45:37,053 DEBUG    [mqttwarn.core             ] New `smtp:mqttwarn' job: IOTgadgettopic
2022-08-18 10:45:37,054 DEBUG    [mqttwarn.core             ] Processor #0 is handling: `smtp' for mqttwarn
2022-08-18 10:45:37,054 INFO     [mqttwarn.core             ] Invoking service plugin for `smtp'
2022-08-18 10:45:37,056 DEBUG    [mqttwarn.services.smtp    ] *** MODULE=d:\program files (d)\python37\lib\site-packages\mqttwarn\services\smtp.py: service=smtp, target=mqttwarn
2022-08-18 10:45:37,056 ERROR    [mqttwarn.core             ] Cannot invoke service for `smtp'
Traceback (most recent call last):
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\core.py", line 517, in processor
    notified = timeout(module.plugin, (srv, st))
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\util.py", line 133, in timeout
    raise it.exception
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\util.py", line 121, in run
    self.result = func(*args, **kwargs)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\services\smtp.py", line 24, in plugin
    username    = item.config['username']
KeyError: 'username'
2022-08-18 10:45:37,063 WARNING  [mqttwarn.core             ] Notification of smtp for `IOTgadgettopic' FAILED or TIMED OUT
2022-08-18 10:45:37,066 DEBUG    [mqttwarn.core             ] Job queue has 0 items to process

[amotl]

Hi again,

I am glad that the issues with the configuration files seem to be resolved now.

not keep clogging up this site with each of my steps

Don't worry, please keep reporting. Every detail counts to improve the software and/or its documentation in one way or another. On the very last detail you reported, I've just created #573.

Cheers, Andreas.

[amotl]

Hi again,

I've just investigated the situation on my workstation using Racker, and wanted to report back about the outcome on Windows10, mainly as a reference for others running into the same problem. #576 also tries to reproduce the problem on CI. In order to finally resolve this issue, we should improve the documentation correspondingly, to make Windows users aware of it.

With kind regards, Andreas.

Introduction

Per https://github.com/PyCQA/pylint/issues/1663, we discovered that:

When using the default redirection operator > on Windows 10, it produces a file with UTF-16 encoding. This obviously causes problems when trying to read it back with assuming the UTF-8 character encoding.

Out-File and > / >> create "Unicode" - UTF-16LE-files by default

Apparently, the problem only happens with PowerShell, it does not occur with Bash nor does it happen with cmd.exe.

Non-solutions

• Using the chcp command for changing the Code page of the active console did not work.
• Using [Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(65001) did not work.

Solution I

Use the Out-File cmdlet.

mqttwarn make-config | Out-File -Encoding utf8 mqttwarn.ini

Solution II

Adjust the settings in $PSDefaultParameterValues. This will be active for all subsequent invocations.

$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'
mqttwarn make-config > mqttwarn.ini

Reset this setting:

$PSDefaultParameterValues.Remove('Out-File:Encoding')

You can also adjust the setting for all cmdlets, not only Out-File:

$PSDefaultParameterValues['*:Encoding'] = 'utf8'

In order to make those settings persistent, you may want to add them to your $PROFILE (current user only) or $PROFILE.AllUsersCurrentHost (all users) file.

References

• https://github.com/PyCQA/pylint/issues/1663
• https://stackoverflow.com/questions/5796339/printing-unicode-characters-to-the-powershell-prompt/5808445#5808445
• https://stackoverflow.com/questions/49476326/displaying-unicode-in-powershell/49481797#49481797
• https://stackoverflow.com/questions/40098771/changing-powershells-default-output-encoding-to-utf-8
• https://stackoverflow.com/questions/57131654/using-utf-8-encoding-chcp-65001-in-command-prompt-windows-powershell-window/57134096#57134096

[amotl]

7b42d6c improves the README correspondingly, so let me close this issue. Thanks again for making us aware of it, Jim.