intermittent struct.error

leres commented 1 year ago

Describe the bug

I'm using tasmota 9.2.0.3. I have a cron job that uses decode-config to backup configs from my devices. Occasionally decode-config blows up with struct.error.

To Reproduce

decode-config -d unit1.example.com -T json --json-indent 4

Expected behavior

This normally outputs json but once in awhile it crashes.

Version Information

decode-config v9.2.0.3 [aeca3c0] by Norbert Richter <nr@prsolution.eu>

Script:   decode-config
Python:   3.11.3
Platform: FreeBSD-13.2-RELEASE-amd64-64bit-ELF - amd64
OS:       FreeBSD 13.2-RELEASE FreeBSD 13.2-RELEASE r13 LBL
Time:     2023-04-22 13:23:08

System' information of your Tasmota data

{"bootcount": 325, "bootcount_reset_time": 1581993339, "cfg_crc": "0x5a49", "cfg_crc32": "0xc4545b4e", "cfg_holder": 4617, "cfg_size": 4096, "cfg_timestamp": 1682178556, "header": {"data": {"crc": "0x2e1e", "crc32": "0xc4545b4e", "platform": "ESP82xx", "size": 4096, "template": {"crc": "0x5a49", "crc32": "0xc4545b4e", "size": 4096, "version": "0x9020003"}, "version": "0x9020003"}, "env": {"platform": "FreeBSD-13.2-RELEASE-amd64-64bit-ELF", "python": "3.11.3", "script": "decode-config v9.2.0.3 [aeca3c0]", "system": "FreeBSD amd64 13.2-RELEASE FreeBSD 13.2-RELEASE r13 LBL"}, "timestamp": "2023-04-22 15:49:16"}, "version": "0x9020003"}

Additional context

Here's an example stack trace:

# get_fieldvalue FAILED fieldname=config_version, fielddef=(15, 'B', 3894, (None, '0 <= $ < len(Platform.STR)', ('Internal', None)), (None, False)), format_=B
Traceback (most recent call last):
  File "/home/ice/u0/leres/scripts/decode-config", line 5020, in <module>
    CONFIG['info'] = get_config_info(CONFIG['decode'])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ice/u0/leres/scripts/decode-config", line 2631, in get_config_info
    config_version = get_field(decode_cfg, Platform.ALL, 'config_version', fielddef, raw=True, ignoregroup=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ice/u0/leres/scripts/decode-config", line 3851, in get_field
    value = get_fieldvalue(fieldname, fielddef, dobj, baseaddr+addroffset)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ice/u0/leres/scripts/decode-config", line 3648, in get_fieldvalue
    raise e
  File "/home/ice/u0/leres/scripts/decode-config", line 3644, in get_fieldvalue
    unpackedvalue = struct.unpack_from(format_, dobj, addr)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack_from requires a buffer of at least 3895 bytes for unpacking 1 bytes at offset 3894 (actual buffer size is 2920)

curzon01 commented 1 year ago

decode-config v9.2.0.3 is outdated, does this also happen using current v12.5.0 with the same device?

leres commented 1 year ago

I'm using tasmota 9.2.0.3 because the bloat in newer versions of tasmota makes the image too large to do OTA updates without first upgrading to a minimal load (an extra step I wanted to avoid). I'll try to upgrade a module for testing but this failure is so rare (I'll see a couple within a few days every 2-6 months) I'm not optimistic I can reproduce it quickly.

I looked at the commit logs and don't see any that look related to the problem and also did a diff between 9.2.0.3 and development and the exception I see in get_fieldvalue() is no longer being caught so the debugging print is no longer there.

curzon01 commented 1 year ago

I'm talking about decode-config version, not Tasmota version.

You wrote your -v output is decode-config v9.2.0.3 [aeca3c0] by Norbert Richter <nr@prsolution.eu> that tells me you are using decode-config v9.2.0.3.

As written in README at the beginning ("decode-config is compatible with all Tasmota versions, starting from Tasmota v5.10.0 up to the current one."): You can use always the current decode-config (master or development) with all current and past Tasmota firmware versions. It is not necessary to use an outdated decode-config version with an older tasmota FW.

Please test the current deocde-config version v12.5.0 in conjunction with your device with the old firmware v9.2.0.3

leres commented 1 week ago

I'm now running tasmota 14.2.0 and have:

% decode-config --version
decode-config v14.2.0.1 [7d54038] by Norbert Richter nr@prsolution.eu

and my daily cron job still sees this error, on average, once every 5 weeks:

Traceback (most recent call last):
  File "/home/ice/u0/leres/scripts/decode-config", line 6873, in <module>
    CONFIG['info'] = get_config_info(CONFIG['decode'])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ice/u0/leres/scripts/decode-config", line 3695, in get_config_info
    config_version = get_field(decode_cfg, HARDWARE.ESP, 'config_version', fielddef, raw=True, ignoregroup=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ice/u0/leres/scripts/decode-config", line 5464, in get_field
    value = get_fieldvalue(fieldname, fielddef, dobj, baseaddr+addroffset)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ice/u0/leres/scripts/decode-config", line 5263, in get_fieldvalue
    unpackedvalue = struct.unpack_from(format_, dobj, addr)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack_from requires a buffer of at least 3895 bytes for unpacking 1 bytes at offset 3894 (actual buffer size is 2920)

decode-config is used about 10 times each day so this error is fairly intermittent using something similar to:

decode-config -d noname.example.com -T json --json-indent 4 > file

As I understand it, struct.unpack_from() is not being called with enough buffer to unpack the data. Here's a stackoverflow post that discusses the issue.

I suspect what's happening is that the config downloaded from the device is randomly ordered and once in a while it produces an ordering that is problematic for [decode-config](struct.error: unpack_from). I've looked at the code but I don't understand what it's doing well enough to

Is there a way to download the config without parsing it? I tried all three options for -t (bin, json, and dmp) and they out result in the same exact json output. (And it looks like -d is deprecated, it is no mentioned in the --full-help output). The only way I can see to obtain a .dmp format file is to use the web gui to connect to a device and click the "backup configuration" button.

curzon01 commented 1 week ago

For download the config without processing use wget or curl (you don't need the gui):

wget http://your-tasmota/dl -O config.dmp -q or curl http://your-tasmota/dl -o config.dmp --no-progress-meter

BTW the reason of your error sometimes is that the download (decode-config use the same way as curl and whet above) delivers absolute garbadge.

leres commented 1 week ago

I rewrote my iot backup script to keep a copy of the .dmp files, with any luck I'll soon have a .dmp file that causes decode-config to fail.

tasmota / decode-config