Cannot run firmware on ESP32 with MAX_DOMOTICZ_IDX = 7 (or above) to control relays via MQTT

ESPCustomizer commented 3 years ago

PROBLEM DESCRIPTION

Not possible to run firmware with MAX_DOMOTICZ_IDX = (7 or higher) on ESP32 to control more than 6 relays via MQTT

REQUESTED INFORMATION

After getting enthusiastic about tasmota with successful flashing a Sonoff 4ch pro and connecting it using MQTT to my Domotics controller I planned a new project for garden watering. I assembled my own hardware using an ESP32 board connected to 8 relay board (and later want to add some more switches/sensors) Starting pre-compliled firmware I could operate the device but only connect 4 relays via MQTT :-( After digging into the code I found that this corresponds to a setting in tasmota.h default: MAX_DOMOTICZ_IDX = 4

So I updated it to 8 and compiled it but the device didn't start. Since I had no experience with compiling my own firmware I took a step back and compiled it with original value of 4 which worked. After that I tried different values and found that for 6 or below it works as expected. At first start tasmota can be configured and I see number of IDX in GUI as set in tasmota.h

From serial monitor I also see normal behavior: rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DOUT, clock div:2 load:0x3fff0018,len:4 load:0x3fff001c,len:1044 load:0x40078000,len:8896 load:0x40080400,len:5828 entry 0x400806ac ÿ00:00:00 CFG: Loaded, Count 2 00:00:00 Project tasmota Tasmota Version 8.3.1.7(tasmota)-1_0_4 00:00:00 WIF: WifiManager active for 3 minutes 00:00:00 HTP: Web server active on tasmota_FC3B10-6928 with IP address 192.168.4.1

Updating MAX_DOMOTICZ_IDX to 7 or above again does not work. From serial monitor it looks like it doesn't start correctly.

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DOUT, clock div:2 load:0x3fff0018,len:4 load:0x3fff001c,len:1044 load:0x40078000,len:8896 load:0x40080400,len:5828 entry 0x400806ac ÿ00:00:00 CFG: Loaded, Count 4

So it looks like there is either a dependency which I cannot find or above 7 it runs out of memory. With Google, I saw suggestions to disable functionality that you don'tneed but don't know how :-(. I have a bit hobby experience with Python coding but I'm not familiar enough with this coding and how everything connects to know what/where to change. In the meantime I spend many evenings searching in the code but I don't manage to understand how to read it or debug it.

Does anybody know how to fix it and can explain me in "dummy" language such that I can move on with my project?

[ ] Read the Contributing Guide and Policy and the Code of Conduct
[ x] Searched the problem in issues
[ x] Searched the problem in the docs
[ x] Searched the problem in the forum
[ x] Searched the problem in the chat
[ x] Device used: ESP32 DEVKITV1
[ ] Tasmota binary firmware version number used: _____
- [ ] Pre-compiled
- [x ] Self-compiled
- [ ] IDE / Compiler used: _____
[x] Flashing tools used: ESPeasy

[x] Provide the output of command: Backlog Template; Module; GPIO 255:


00:01:00 CMD: Backlog Template; Module; GPIO255
00:01:00 RSL: stat/tasmota_FC3B10/RESULT = {"NAME":"ESP32-DevKit","GPIO":[1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,0,1,1,1,0,0,0,0,1,1,1,1,1,0,0,1,0,768,0,0],"FLAG":0,"BASE":117}
00:01:00 RSL: stat/tasmota_FC3B10/RESULT = {"Module":{"1":"ESP32-DevKit"}}
00:01:00 RSL: stat/tasmota_FC3B10/RESULT = {"Command":"Error"}

- [ ] If using rules, provide the output of this command: `Backlog Rule1; Rule2; Rule3`:

Rules output here:

- [ ] Provide the output of this command: `Status 0`:

00:01:48 CMD: STATUS 0 00:01:48 RSL: stat/tasmota_FC3B10/STATUS = {"Status":{"Module":1,"DeviceName":"Tasmota","FriendlyName":["Tasmota"],"Topic":"tasmota_FC3B10","ButtonTopic":"0","Power":0,"PowerOnState":0,"LedState":1,"LedMask":"0000","SaveData":1,"SaveState":1,"SwitchTopic":"0","SwitchMode":[0,0,0,0,0,0,0,0],"ButtonRetain":0,"SwitchRetain":0,"SensorRetain":0,"PowerRetain":0}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS1 = {"StatusPRM":{"Baudrate":0,"SerialConfig":"5N1","GroupTopic":"tasmotas","OtaUrl":"http://thehackbox.org/tasmota/release/tasmota.bin","RestartReason":"Software reset CPU","Uptime":"0T00:01:48","StartupUTC":"","Sleep":0,"CfgHolder":4617,"BootCount":4,"BCResetTime":"1970-01-01T00:00:00","SaveCount":8}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS2 = {"StatusFWR":{"Version":"8.3.1.7(tasmota)","BuildDateTime":"2020-07-26T21:59:01","Core":"1_0_4","SDK":"v3.2.3-14-gd3e562907","CpuFrequency":80,"Hardware":"ESP32","CR":"346/699"}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS3 = {"StatusLOG":{"SerialLog":0,"WebLog":0,"MqttLog":0,"SysLog":0,"LogHost":"","LogPort":0,"SSId":["WIFI_INTERNET",""],"TelePeriod":300,"Resolution":"00000000","SetOption":["00008009","2805C8000100060000005A00000000000000","00000000","00000000"]}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS4 = {"StatusMEM":{"ProgramSize":1046,"Free":1984,"Heap":111,"PsrMax":0,"PsrFree":0,"ProgramFlashSize":4096,"FlashSize":4096,"FlashFrequency":40,"FlashMode":3,"Features":["00000809","8FDAC787","00140001","000000CD","010013C0","40000181","00004000"],"Drivers":"1,2,3,4,5,7,8,9,10,12,20,21,24,26,27","Sensors":"1,2,3,5,6"}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS5 = {"StatusNET":{"Hostname":"tasmota_FC3B10-6928","IPAddress":"0.0.0.0","Gateway":"0.0.0.0","Subnetmask":"0.0.0.0","DNSServer":"0.0.0.0","Mac":"24:62:AB:FC:3B:10","Webserver":0,"WifiConfig":0,"WifiPower":0.0}} 00:01:48 RSL: stat/tasmotaFC3B10/STATUS6 = {"StatusMQT":{"MqttHost":"","MqttPort":0,"MqttClientMask":"DVES%06X","MqttClient":"DVES_FC3B10","MqttUser":"DVES_USER","MqttCount":0,"MAX_PACKET_SIZE":1200,"KEEPALIVE":30}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS7 = {"StatusTIM":{"UTC":"1970-01-01T00:01:48","Local":"1970-01-01T00:01:48","StartDST":"1970-01-01T00:00:00","EndDST":"1970-01-01T00:00:00","Timezone":"+00:00","Sunrise":"19:06","Sunset":"07:12"}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS10 = {"StatusSNS":{"Time":"1970-01-01T00:01:48"}} 00:01:48 RSL: stat/tasmota_FC3B10/STATUS11 = {"StatusSTS":{"Time":"1970-01-01T00:01:48","Uptime":"0T00:01:48","UptimeSec":108,"Heap":111,"SleepMode":"Dynamic","Sleep":0,"LoadAvg":179,"MqttCount":0}}

- [ ] Provide the output of the Console log output when you experience your issue; if applicable:
  _(Please use_ `weblog 4` _for more debug information)_

Console output here:



### TO REPRODUCE
_Steps to reproduce the behavior:_

Compile ESP32 firmware with MAX_DOMOTICZ_IDX = 7 (0r above 7)
Upload to ESP32 
Try to restart..........fails, as shown in prolem description at the beginning

### EXPECTED BEHAVIOUR
Firmware starting and possible to configure more than 6 IDX for Domoticz MQTT

### SCREENSHOTS
_If applicable, add screenshots to help explain your problem._

### ADDITIONAL CONTEXT
_Add any other context about the problem here._

**(Please, remember to close the issue when the problem has been addressed)**

sfromis commented 3 years ago

In my limited understanding, MAX_DOMOTICZ_IDX is not designed for you to change, unless you very carefully make sure that all the places it is used is changed to accommodate more than 4, without breaking the data structures (like settings) which the data is part of. Tasmota does not support more than 4 without such work.

ESPCustomizer commented 3 years ago

@sfromis: certainly true. With my limited knowledge I only found some locations where this max is used to define or check a loop. Since increasing to 6 seems to work as expected it looks like it is not a fundamental problem and probably easy to support higher numbers but I cannot find what limit I exceed at 7 and above.

Nog sure how it works at Github. Am I supposed to close the item or is there still a chance that somebody can point me at the line to update?

arendst commented 3 years ago

Your change will break future tasmota upgrades but then that's what you've choosen.

First, while changing MAX_DOMOTICZ_IDX from 4 to anything else you must have read the header above it saying:

/*********************************************************************************************\
 * Constants
\*********************************************************************************************/

// Changes to the following MAX_ defines will impact settings layout
const uint8_t MAX_SWITCHES = 8;             // Max number of switches
const uint8_t MAX_RELAYS = 8;               // Max number of relays
const uint8_t MAX_INTERLOCKS = 4;           // Max number of interlock groups (MAX_RELAYS / 2)
const uint8_t MAX_LEDS = 4;                 // Max number of leds
const uint8_t MAX_KEYS = 4;                 // Max number of keys or buttons
const uint8_t MAX_PWMS = 5;                 // Max number of PWM channels
const uint8_t MAX_COUNTERS = 4;             // Max number of counter sensors
const uint8_t MAX_TIMERS = 16;              // Max number of Timers
const uint8_t MAX_PULSETIMERS = 8;          // Max number of supported pulse timers
const uint8_t MAX_DOMOTICZ_IDX = 4;         // Max number of Domoticz device, key and switch indices

As the settings layout is a fixed area of exactly 4k memory any increase of the above values will overflow the area with unstable system result.

In your case if you increase the value you will need to go over the settings area and find the impact of the change. You'll notice the define MAX_DOMOTICZ_IDX pops up three times in settings.h. Any increase needs extra storage space you will need to calculate how many bytes are needed for a single increment. Then find a location within the 4k settings to decrement the same amount to keep the total exactly 4k (4096 bytes). Look for variable free_f43[117] and decrement the index with your calculated number of bytes. I leave the calculation to you to get some feeling with the procedure. Oh and keep in mind to have the uint32_t variables exactly on a 4-byte binary so as the define also increases a uint16_t I would recommend to increase by steps of 2. Again, dive in C as a real ESPCustomizer and learn a lot.

Good luck and remember any default Tasmota will binary will break your system as the settings have been moved from their default location.

ascillato2 commented 3 years ago

Closing this issue as it has been answered.

Support Information

See Docs for more information. See Chat for more user experience. See Community for forum. See Code of Conduct

ESPCustomizer commented 3 years ago

Thanks. Indeed got 8 relays working with suggestions of Theo

The-A74 commented 1 year ago

@ESPCustomizer sorry for jumping in this post after some time :) How did you get it to work? I'm trying to compile tasmota 12.4.0 release, followed @arendst suggestions, firmware builds (4096 bytes setting size is ok) but it doesn't work as expected: on Domoticz section conficuration I can't see any idx for relais. Default settings.h configuration states MAX_DOMOTICZ_IDX=4 and MAX_DOMOTICZ_SNS_IDX=12: firmware builds and works fine with domoticz page letting me configure 4 relais (I have 8 on board all functional on the web interface) Trying to play with MAX_DOMOTICZ_IDX , analyzing tasmota_types.h, i saw that every increase of 1 requires 10 bytes and that every MAX_DOMOTICZ_SNS_IDX unit requires 2 bytes. So just to experiment I decreased MAX_DOMOTICZ_SNS_IDX by 10 (freeing 20 bytes) and increased MAX_DOMOTICZ_IDX by 2. I did not modify any free_xxx variable for the moment. Compilation completes but as I said above, domoticz settings page is not working as expected: I canìt see any switch idx available and i cannot control any relais on the main tasmota page too.....

My goal is to have idx from 1 to 8 in domoticz settings

My device is a Kincony KC868-A8

ESPCustomizer commented 1 year ago

For me it indeed worked by following suggestions. Ik you need details on changes I have to check after my holiday if I can trace back the changes

The-A74 commented 1 year ago

Finally i got it to work as expected on my KC868-A8 (ESP32 base hardware). Tasmota release 12.4.0, I modified:

in tasmota.h :
MAX_DOMOTICZ_IDX from 4 to 8 ,requires 40 byres MAX_DOMOTICZ_SNS_IDX from 12 to 2 (i need only one temp sensor, so it's ok in my case) freeing 20 bytes
in tasmota_types.h :
free_esp32_448[4] --> removed (commented) this line freeing 4 bytes free_esp32_484[17] --> free_esp32_484[1] freeing 16 bytes

Compiled with a warning in domiticz xdrv, flashef the firmware (always use "Erase Flash and Upload" to avoid settings retaining causing some glitches and instability) et voilà, idx from 1 to 8 for relais and inputs are present and working. Sensors from 1 to 11 are still displayed but only 1+2 are usable and working. Do not change the idx values displayed for sensors from 3 to 11 as they refer to other settings of tasmota now abd it will corrupt the firmware.

Just in case someone else is struggling with a simiar config

Ciao

ESPCustomizer commented 1 year ago

I checked my old notes and my changes to modify Tasmota 8.x.x. were slightly different and probably and not limiting other functionality but I didn’t check if this is still possible in version 12 or if the memory was already taken by something else.

In Tasmota.h I changed the number of switches from 4 to 8 as below const uint8_t MAX_KEYS = 8; // Max number of keys or buttons const uint8_t MAX_DOMOTICZ_IDX = 8; // Max number of Domoticz device, key and switch indices

This requires additional memory for following items: uint32_t shutter_button[MAX_KEYS]; // FDC unsigned long domoticz_relay_idx[MAX_DOMOTICZ_IDX]; // 344 unsigned long domoticz_key_idx[MAX_DOMOTICZ_IDX]; // 354 uint16_t domoticz_switch_idx[MAX_DOMOTICZ_IDX]; // 454

Which means uint32_t = 4 byte unsigned long =8 byte unsigned long =8 byte unit16_t = 2 byte

                                     =  22 bytes per switch

So for 4 additional switches 88 bytes is required.

To free this I modified below line from 89 to 1 uint8_t free_f5a[1]; // F5A - Decrement if adding new Setting variables just above and below

This worked for me but as said it was some time ago with Tasmota v8.x.x. and not sure if this still applies fo newer version

arendst commented 1 year ago

Just a note. The unsigned long is also just 4 bytes as it's an uint32_t too.

arendst / Tasmota