Failed restoring a JSON from v9.3.1 onto a 9.5.0.8 device

[barbudor]

Describe the bug

Restoring a JSON from v9.3.1 onto a 9.5.0.8 device failed.

To Reproduce

1. Have a device running 9.5.0
2. Backup the configuration as JSON decode-config -s 192.168.y.z -o device_9.5.0.json
3. Upgrade the device to 9.5.0.8
4. Clear the configuration with reset 1
5. Re-upload the backed up configuration with decode-config -s 192.168.y.z -i device_9.5.0.json
6. Fails with:

   Load data from device '192.168.x.y'
   WARNING 9 (@4247): file 'R:\tasmota\conf\skt_xxxxx_9.3.1.json' array 'sensors[3]' exceeds max number of elements [2]
   Premature exit - Restore data error #9
   WARNING 9 (@4261): file 'R:\tasmota\conf\skt_xxxxx_9.3.1.json' array 'sensors' couldn't restore, format has changed! Restore value contains <class 'list'> but an array of size [2] is expected
   Premature exit - Restore data error #9

Expected behavior

Device should be reconfigured with a compatible configuration

Version Information (please complete the following information):

decode-config.py v9.5.0.8 [aa8ab02] by Norbert Richter <nr@prsolution.eu>

Script:   decode-config.py
Python:   3.8.6
Platform: Windows-10-10.0.19041-SP0 - AMD64
OS:       Windows 10 10.0.19041
Time:     2021-09-11 15:48:22

• 'system' information of your Tasmota data (append -g System to your decode-config parameter, e. g. decode-config -f myconfig.dmp -g System):

Didn't found a proper syntax to do it from a JSON file

Additional context

sensors field in v9.3.1:

  "sensors": [
    "0xffffffff",
    "0xffffffff",
    "0xffffffff"
  ],

became in 9.5.0.8

  "sensors": [
    [
      "0xffffffff", 
      "0xffffffff", 
      "0xffffffff", 
      "0xffffffff"
    ], 
    [
      "0xffffffff", 
      "0xffffffff", 
      "0xffffffff", 
      "0xffffffff"
    ]
  ], 

[curzon01]

This is one of the rare examples where the data structure has changed so structurally that you have to choose one of two options:

• convert the data manually within your JSON backup
• ignore the warning using -w. This will restore all except the changed structure

There is currently no functional conversion from old to new data in decode-config. Since data is simply read in on the basis of JSON identifiers, the data structures of old systems must basically be the same. A field can become larger and smaller, a variable can take on a different size, all that's handled. This system weakens when there are structural changes. In the past, I used new JSON identifiers for the few changes of this kind (there were maybe 2 or 3), with the result that this procedure simply did not convert these fields because the new identifier did not exist in the old JSON.

I never wanted to have too much functional data dependency in decode-config, because for this I would not only have to keep track of the data structure changes in the past, but also functional dependencies. That means a lot of work when maintaining the Tasmota cmnds (-T cmnds), because I do not only track the file changes in setting.h, setting.ino and tasmota.h, but also have to pay attention to command implementations in the history. I didn't really want to add more.

[barbudor]

Thanks for the details. Unless I missed it, this is not detailed in the manual. Worth a word and a link to this issue ? So far the workaround I found is probably the only good one:

• Install the firmware matching the JSON backup (could be on a nodemcu if not possible on the device itself)
• Restore the backup
• Upgrade the device with the target version
• Backup JSON

[curzon01]

Install the firmware matching the JSON backup (could be on a nodemcu if not possible on the device itself)

Unnecessary, take it easy: use your old backups and restore it on new device. Use parameter -w if warning occurs. This does what intuitively you would expect except what it can't - here it restores all from your backup from the previous release config data except sensors. This will be output during restore:

./decode-config.py -s v9_5_0_8_device  -i 9_5_0.json -w
Load data from device 'v9_5_0_8_device'
WARNING 9 (@4247): file './ 9_5_0.json' array 'sensors[3]' exceeds max number of elements [2]
WARNING 9 (@4247): file './ 9_5_0.json' array 'sensors[10]' exceeds max number of elements [4]
WARNING 9 (@4261): file './ 9_5_0.json' array 'sensors' couldn't restore, format has changed! Restore value contains <class 'str'> but an array of size [4] is expected
WARNING 9 (@4247): file './ 9_5_0.json' array 'sensors[10]' exceeds max number of elements [4]
WARNING 9 (@4261): file './ 9_5_0.json' array 'sensors' couldn't restore, format has changed! Restore value contains <class 'str'> but an array of size [4] is expected
Restore successful to device 'v9_5_0_8_device' from './ 9_5_0.json'

these are just warnings, not errors

[curzon01]

As I said, there have been two or three objects during the entire Tasmota life since v5.10 that have had such a structural change, sensors is now the fourth. I think we can live with either adapting this single object manually in JSON or skipping it with the -w parameter.

[barbudor]

Thanks