decode-config and Tasmota scripting

[littlebilly]

Have you looked for this feature in other issues and in the docs? yes

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

The decode-config Tool https://github.com/tasmota/decode-config is not able to decode when using your scripting.

UnicodeEncodeError: 'charmap' codec can't encode character '\u0766' in position 17464

I think its related to the compressed script.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

(Please, remember to close the issue when the problem has been addressed)

? Is it possible to get decode-config working with tasmota scripting?

Billy

[gemu2015]

i never used this tool. the only difference between rules and scripting is that rules is using 3 separate sections while scripting uses 1 section of script buffer (containing the 3 rules buffers). i am not very familiar with python so you should better ask the author of decode-config (Norbert Richter) to look at this problem.

[curzon01]

I tried to support compressed scripts but there are some points I'm stuck:

1. script uses the settings rules array char rules[3][512] as one stream and I'm unable to distinguish between rules and script is enabled, 'cause the Tasmota feature flags are not part of confiugration data - this is only questionable from oneline-devices, unfortunately not for stored config data
2. I'm also unable to distinguish between compressed and uncompressed script strings. With rules it's possible 'cause rule string uses the following rules:
   • rule strings having char[0] != '\0' are uncompressed ending with '\0'
   • rule strings having char[0] == '\0' and char[1] != '\0' are compressed strings ending with '\0'
   • rule strings having char[0] == '\0' and char[1] == '\0 are empty

The issue is that compressed strings must be encoded/decoded anyway and UTF does not allow any possible combination of byte order so for the json backup output of decode-config I'm using human readable hex values but these are not readable nor editable and therefor useless for end user.

I'm currently unable to a) detect if the rule array is used by script as one long string b) unable to distinguish between compressed/uncompressed script string

If scripting used the same byte indication as rules (2.) I would go one step further. It would be perfect if Script did not fill the flag (script enabled or not) rule_enabled with a 1 but with a different number so it is possible to distinguish between rule/script used. Script data currently cannot be distinguished from rule data, because of (1.)

[gemu2015]

i use >D as an ID for a valid script. (very first 2 chars) if you find >D without uncompress it is an uncompressed script. if you decode with unishox and find a >D it is a compressed script.

i dont think this imposes a conflict with rules.

hope that helps

[littlebilly]

So sieht das für ein comprimiertes Script aus, nachdem decode config jetzt das Ergebnis abspeichert. (Hatte ich einen Fehler in meiner Batch Datei) "rule_enabled": { "rule1": 1, "rule2": 0, "rule3": 0 }, "rule_once": { "rule1": 0, "rule2": 0, "rule3": 0 }, "rule_stop": { "rule1": 0, "rule2": 0, "rule3": 0 }, "rules": [ "?CpF278|?z,,*+LLCq!܄qQ܄n:,9Dn?τOnj!>n?01E#u|n>|)7;<4a9aA329FiAɏ@EɎBF||8s", "", "" ],

[littlebilly]

Sorry wollte das nicht schließen! Wäre natürlich super wenn man das lesen könnte.

[curzon01]

@gemu2015 - not fully: rules are defined as three strings each 512. The definition devides the script string into 3 parts and therefore compressed script strings are broken. I need to know that a string is one using 1536 bytes or three with each 512. this is impossibe currently

but let me check... will investigate

[gemu2015]

you may simply decode a few bytes to get the >D if you decode >D it must be 1536 bytes script and you can decode the whole block.

[curzon01]

yes, I know but I do not have exception handling for any setting defintion like that within the decoder - it is completely definition driven. I have to look how I can solve this exceptions without breaking the functional code

[curzon01]

sorry, not feasble that way - feeding unishox decode with still decoded content (thats not the way unishox was written for) results in various not detecable arrays in python. I can 100% detect '>D' if it is uncompressed but a compressed string can contain anything. I don't want to have an interpretation within decode-config if a string which do not start with '>D' might be an uncompressed rule, compressed rule or compressed script.

[gemu2015]

>D is compressed always to: (this is independent of chars to follow after >D) [0] char '?' [1] char '\b' so you know when a compressed script is found

[curzon01]

thx, didn't tried that way. If compresses two bytes are allways same Independent of following data I can try top detect that way.

[curzon01]

If user disables unishox compression, scripting saves the script as is and can contain leading comments (e.g. script from cookook) which does not match '>D' - and this requires a substantive interpretation of content I want to prevent. I think as long as I do not have a valid indicator within configuration data for script is compiled in, it will become a newer ending story.

[gemu2015]

>D ist mandatory. (i have to modify script cookbooks old examples) you can not start a script either compressed or uncompressed that does not start with >D

a rule never starts with >D (makes no sense)

[gemu2015]

also features reflect that script is compiled

ifdef USE_SCRIPT

feature_drv2 |= 0x00080000; // xdrv_10_scripter.ino

endif

[curzon01]

yes, but the difference is that the decode-config is a tool to backup/restore config data so it does not check if data content make sense or not so also bad script code which may not executed by your script interpreter is possible and therefore the data handling of decode-config will fail. It's unsafe to assume >D (or the compress variant) is a indicator for scripts. As I wrote above feature_drv2 is only available for online devices, not for stored configuration data, 'cause the feature flags are not part of Tasmota config data

issues for config data access are

• script using rule[] array is the only setting which is used as an union
• there is no save indicator whithin config data which leads an external program if rules is used as rules[3][512] or rules[1536]

[gemu2015]

i dont think so. if you find a script indicator you assume a script and decode one block instead of 3

it would be extremely rare that a config file is messed up showing >D an not being a script

the rules enable flag is a uint8 and currently only 3 bits used.

if that helps i can use e.g. bits 6 and 7 for script and compressed script markers

is there already a unishox marker in the config file ?

[curzon01]

no unishox marker directly, unishox compression used by rules can be detected by:

• char[0] == 0 and char[1] == 0: empty string
• char[0] == 0 and char[1] != 0: compressed string starting with char[1]
• char[0] != 0: uncompressed string

this is backward compatible with older Tasmotas

I'd also thought about searching for '>D' but that seems very uncertain to me - it can for example also be part of a string which sets a Mem/Var or Publish command. That doesn't happen very often, but I don't like such functional dependencies.

might be possible to use compressed/uncompressed strings for scripts in the same manner as rules? it would be compatible with rule strings so a compressed/uncompressed flag is not necessary.

only a flag which indicates script usage (that rules[3][512] is rules[1536]) would be helpfull

[gemu2015]

why searching >D ???? there is only one >D in a script and that is at char[0] and char[1] per definition else script is not accepted by scripter

char[0]=='>' and char[1]=='D' = uncompressed script char[0]=='?' and char[1]=='\b' = compressed script

this is completely easily discriminated from rules

[curzon01]

nobody prevents you from entering an invalid script. The only thing that happens to you is that you don't run the script. This is not helpful for the detection of whether the variable is evaluated either way

[gemu2015]

the script won't run and give a console message. on restart the defective script is even cleared to zero and replaced by a short message script starting with >D and a comment.

[curzon01]

I used the wrong script example from Sensor Logging without unishox compressin and the config binary data contains now thw following (sorry line feeds are not in but this might be explain the sitation):

"; define all vars here; reserve large strings>D 48hum=0temp=0fr=0res=0; moving average for 60 secondsM:mhum=0 60M:mtemp=0..."

using your rule this is not a script and therefore the rules var will be processed by decode-config as rules[3][512] which results in 3 strings traling with 0 which corrupt your script.

Why I should implement backup of data which can not be handled?

[gemu2015]

the script is not valid and would be cleared on restart. backing up before a restart would generate 3 rules sets which would not harm because a restore would install them as rules again. and on restart would again be cleared.

but anyhow i will implement a flag bit in rules enabled bit 7 that scripter is in place.

[curzon01]

if you didn't still worked on that, can I make the PR (I would like using the unused 'rule_once' bits for indication, so rules bits are not missused for feature expanding). I can make it...

[gemu2015]

i did not make a pr yet. feel free to pr .

[curzon01]

decode-config adj in progress, needs refactoring...

[curzon01]

Implemented in decode-config developer v8.3.1.7 [00179] commit aff1983

[littlebilly]

Tested it, runs as expected even with compressed script! Thanks a lot for implementation

[curzon01]

my pleasure - so if this issue is solved you can close it - you opened it, you close it, it's so easy :wink:

[littlebilly]

I just did another testing. Tried to restore with decode-config. Firstly I restored a script with a restore-Subset! Command: decode-config-development>decode-config.py -d 192.168.148.213 --restore-file Config_Rules_Gosund_4_213.json The script was restored correctly, but on console I get an error Output! ERROR 11 (@4287): Config data upload failed - <div style='text-align:center;'>Gerät wird jetzt neu gestartet</div><br></div> Premature exit - Upload error #11

When restoring a complete Backup the script was also restored and activated correctly, but I got the same error output!

May be you have an idea to solve this issue.

[curzon01]

pls open an issue on https://github.com/tasmota/decode-config and close this one. This is not the repository of decode-config, I can't manage here anything.