search

letscontrolit / ESPEasy

Easy MultiSensor device based on ESP8266/ESP32
http://www.espeasy.com
Other
3.25k stars 2.2k forks source link

Discussion on how to define 24 task settings #2621

Closed TD-er closed 1 year ago

TD-er commented 4 years ago

There has been a lengthy discussion already about how to implement the patch to use 24 tasks on ESP8266 ESPEasy. However this was posted below a pull request, which makes it hard to follow and rather invisible to others unless they track all changes and updates on this repo.

So let's continue here.

The most useful parts of the discussion:

@uzi18 has some set to define 24 tasks: https://github.com/letscontrolit/ESPEasy/commit/f7e0d7cfdde65d9503ff08fd92d48681dc8da702#commitcomment-35122941

#define TASKS_MAX                          24
#define DAT_OFFSET_CONTROLLER            4096    // 0x1000 each controller = 1k, 4 max
#define DAT_OFFSET_CUSTOM_CONTROLLER     8192    // 0x2000 each custom controller config = 1k, 4 max.
#define DAT_OFFSET_TASKS                16384    // 0x4000 each task = 2k, (1024 basic + 1024 bytes custom), 12 max

@clumsy-stefan has another (incompatible) one: https://github.com/letscontrolit/ESPEasy/commit/f7e0d7cfdde65d9503ff08fd92d48681dc8da702#commitcomment-35126444

#define TASKS_MAX                          24 // max 12! 
#define DAT_OFFSET_CONTROLLER            DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX)                        // each controller = 1k, 4 max 
#define DAT_OFFSET_CUSTOM_CONTROLLER     DAT_OFFSET_CONTROLLER + (DAT_CUSTOM_CONTROLLER_SIZE * CONTROLLER_MAX)  // each custom controller config = 1k, 4 max
TD-er commented 4 years ago

but the exchangable thing is a good point, why not...

Will do :) Only thing is, it does then make the config file smaller (or at least it doesn't need 128k it is right now)

Maybe we should use different names for the files to indicate the nr of tasks?

This will make it clear what file is compatible and which isn't. We can also make conversions between them (in binary) and it makes detection of a config a lot easier and simpler to grasp. So then I can make it dynamic at load which version should be used and we can do a manual convert to another version if required. The newer firmware should be able to handle all and I think it can be done without much resources being used.

By the way, I managed to get rid of a few kB RAM being wasted, since some code was including the ESPEasy-Globals.h while it shouldn't. This also may have caused some bugs in handling commands from rules when they depend on settings and also loading of ExtraSettings may have been buggy when called from commands (not sure if they did)

clumsy-stefan commented 4 years ago

Sounds great! Still lots of ideas to integrate though... Priorisation is everything 😄

I updated to the latest esp8266 core this morning, which unfortunately increased bin. size some 20k... My 1M builds are by now some 40k too big for 2step OTA... so saving binary size is still an important thing... on the other hand, I started to throw out 1M units anyways, the hazzle with OTA updates is just too much and all newer units have anyways much more mem...

TD-er commented 4 years ago

If the chips can be reached, you can also replace the flash chips.

Still size reduction is important, since also the dev and test build approach the limits.

clumsy-stefan commented 4 years ago

Did that with the sonoff-basics... but with the 4ch. and dual's I couldn't find the mem chips... I think it uses only the internal mem... but even with the basics it's much more complicated to exchange the mem chip than buying the pow R2 with is a few dollars more but therefore gives power measurments (ans has 4M mem)

TD-er commented 4 years ago

The ESP8285 has 1M flash internal.

clumsy-stefan commented 4 years ago

the ESP8266 also, not? if not then it's probably a 8285..... but I'll throw them out anyways... there are enough 4/8/16M bpards out there...

TD-er commented 4 years ago

Can you test what I have committed so far?

clumsy-stefan commented 4 years ago

I flashed all my units with the commit from this morning (I'm currently away, so I can't say which one exactly that is).

Currently all units are running fine except the two deep sleep units that don't deliver the analog value when walking up.

I'm back tomorrow an can report more then.

clumsy-stefan commented 4 years ago

All my (40) nodes are now running for some 24h with up to commit #213fae7718dd614e2ded0feebe3015c45b8be42e without issues. also included PR #2605 for the dallas improvments.

I'll add the new commits now since then and update some nodes...

TD-er commented 4 years ago

Things you need to check are:

clumsy-stefan commented 4 years ago

Ok, I see how far I get... I'll try as much as possible.

one thing I can already say (I'm still compiling) is, that all the includes need to be changed to relative paths if it should work out of the box with arduino. Eg. all "src//.h" need to be "..//.h" and stuff that is in the top directory need to have "../../fwdecl.h" or similar... I think I found all of them now and it's compiling..

TD-er commented 4 years ago

OK, good to know. Arduino IDE is a strange tool.

clumsy-stefan commented 4 years ago

I think that's just gcc/C... includes with " are alwys relative to the directory where it's included or you have to specify with -I additional directories to search for includes..

clumsy-stefan commented 4 years ago

after a few tests with my test nodes everything seem to be fine..

Problem is, that I don't use MQTT at all, so I can't really say anything about that. But at least I use some 20 different plugins/sensors and remote commands, rules etc. so I think it's quite a good test. Also I have 4 different type of nodes and running 40nodes in parallel...

I'm going to update the rest of them (so rules and external commands are executed and testted) and will report back.

clumsy-stefan commented 4 years ago

A few quick remarks after updating:

I'll graph the memory, rssi and CPU usage and will report on this after they have run for a day or so.

clumsy-stefan commented 4 years ago

@TD-er I found at least one issue which is different from before. On deepsleep units it seems that the tasks are (sometimes) executed before the wifi connection is establieshed (especially when it takses a litte longer for the connection), therefore not all values are sent to the controller (see log, events occur before connection). Also the analog value is not available or so at that time..

I think this should be adjusted somehow.

INIT : Booting version: SMY_5.11DP Sep 22 2019 10:54:49 (ESP82xx Core 5920ce61, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2)
70 : INIT : Free RAM:32248
71 : INIT : Rebooted from deepsleep #8 Last Task: Background Task - Restart Reason: Deep-Sleep Wake
73 : FS   : Mounting...
95 : FS   : Mount successful, used 75802 bytes of 957314
167 : FS   : Success garbage collection
242 : FS   : Success garbage collection
318 : FS   : Success garbage collection
657 : CRC  : program checksum       ...OK
688 : CRC  : SecuritySettings CRC   ...OK 
786 : INIT : Free RAM:28880
788 : INIT : I2C
790 : INIT : I2C custom clockstretchlimit:1000
791 : INIT : SPI not enabled
793 : EVENT: System#NoSleep=15
803 : INFO : Plugins: 26 [Normal] [4M OTA SMY] 24tasks (ESP82xx Core 5920ce61, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2)
805 : EVENT: System#Wake
912 : WIFI : Set WiFi to STA
944 : WIFI : Connecting clumsy_ap2 attempt #0
948 : OTA  : Arduino OTA enabled on port 18266
963 : EVENT: System#Boot
1070 : WIFI : Connecting clumsy_ap2 attempt #1
1211 : SYS  : 31.00,26408.00,100.00,0.00
1217 : EVENT: sysinfo#rssi=31.00
1245 : EVENT: sysinfo#freeheap=26408.00
1250 : EVENT: sysinfo#load=100.00
1254 : EVENT: sysinfo#uptime=0.00
1404 : SYS  : 1.00,3376.00,0.00,0.00
1407 : EVENT: sysinfo#web=1.00
1433 : EVENT: sysinfo#freestack=3376.00
1437 : EVENT: sysinfo#ip3=0.00
1440 : EVENT: sysinfo#ip4=0.00
1586 : ADC  : Analog value: 0 = 0.000
1589 : EVENT: sysinfo#lipo=0.00
3110 : WD   : Uptime 0 ConnectFailures 0 FreeMem 25112 WiFiStatus 6
5029 : WIFI : Connected! AP: clumsy_ap2 (E4:8D:8C:49:4F:BA) Ch: 5 Duration: 3756 ms
5030 : EVENT: WiFi#ChangedAccesspoint
5055 : EVENT: WiFi#ChangedWiFichannel
5057 : WIFI : DHCP IP: 10.0.10.116 (wemos-mini-16-16) GW: 10.0.0.2 SN: 255.255.0.0   duration: 125 ms
5094 : NTP  : NTP replied: delay 20 mSec Accuracy increased by 0.628 seconds
5095 : Time adjusted by -1569162449568.58 msec. Wander: -435878458.21 msec/second
5097 : Current Time Zone:  DST time start: 2019-03-31 02:00:00 offset: 120 minSTD time start: 2019-10-27 03:00:00 offset: 60 min
5100 : EVENT: Time#Initialized
5104 : EVENT: WiFi#Connected
5110 : Webserver: start
5111 : firstLoopConnectionsEstablished
5113 : EVENT: System#NoSleep=15
5118 : WD   : Uptime 1 ConnectFailures 0 FreeMem 19368 WiFiStatus 3
5121 : EVENT: Clock#Time=Sun,16:27
20116 : EVENT: System#Sleep
20141 : SLEEP: Powering down to deepsleep...
TD-er commented 4 years ago

One thing that has been changed is when tasks have the same name. I'm now using a cache to keep track of what task index has what name. So it will only refer to the first one to match a name. Not sure if it should match only enabled ones, or just looking at the name. But you're being warned when trying to save a task with a name that's not unique, so I guess it has been made clear those should be unique.

clumsy-stefan commented 4 years ago

hmm.. yes, but it worked before ;) the problem is, when I use multiple names for the same "thing" (eg. system values) I get 2 different "devices" on my server and need to do silly workarounds there... (each task name refers to a single device in fhem)... so, when I need more than 4 system values I get them in different devices serverside..

however what's stange is, that I have the same things defined in my wemos-pro deepsleep node, where it still works as before!

Also if you look at the serial log above, the ADC is read and creating an event, but it's always 0! It seems the ADC is taking longer to initialize. Obvisously the same with the IP-Adress, as it's run before the connection is set up it always returns zero.

On deep-sleep units, probably the tasks should not run immediately, but som time later, like after 2 sec. or even something like awake-time/2 so all the sensors get time to power-up and initialize (and WLAN gets time to connect). just as a proposal...

EDIT: obviously there is a "workaround" for this. If you set the "send to controller" value small enough (eg. 2sec. when staying 15sec. awake) then it will send the values multiple times, and at some point, the connection is established and the sensors alive....

TD-er commented 4 years ago

When waking from deepsleep, all tasks are executed at least once, so that's not addressed by "calling" the values. ADC is working when powered on long enough? One of the things I also changed (not sure if it is in this PR, or changed before) is that the ADC is not read when WiFi connection is being executed. (connect to AP) This because the WiFi RF calibration is also using the same ADC, so a read value can be off quite a bit or the ESP may crash.

clumsy-stefan commented 4 years ago

ADC is working when powered on long enough?

yes, or probably as you write, after WiFi is established (RF Calibration?)...

When waking from deepsleep, all tasks are executed at least once, so that's not addressed by "calling" the values.

Not sure if I understand correctly, but yes, the tasks are called, but even before the WiFi connection is established (see log above), therefore some values (like RSSI, IP, etc.) are not yet initialized. That's why I think it should get more time... Only for "standalone" units which do not make any (wifi) connection, it doesn't care if already conencted or not...

but as said, as a workaround, I set the "send to controlelr" values to 3 sec. and the awake time to 15, so the tasks get executed at least once, after the wifi has been initialized.

TD-er commented 4 years ago

Yep, a task should always be executed, or else no measurement will be taken if the WiFi cannot connect. That's a bit nasty when the same ESP node should switch on/off something based on measurements taken by a sensor on the same node.

There is a wifi connected event, which you can use in the rules. Oh and the wake time is "reset" when the connection is made, so you can also play with that setting.

clumsy-stefan commented 4 years ago

That's a bit nasty when the same ESP node should switch on/off something based on measurements taken by a sensor on the same node.

or if you power-on with a GPIO set an external sensor, and only want to take the reading after some init-time..

therefore the tasks in the "single-run" should not be started as first thing... probably even just before going to sleep...

clumsy-stefan commented 4 years ago

Memory seems ok with the latest commits:

image

the update was this morning at 11... no visible increas or similar on any node.

TD-er commented 4 years ago

Hmm, I would expect an increase of free RAM by about 2 kB at least. Your charts don't show that.

clumsy-stefan commented 4 years ago

probably because I use a different set of plugins and no MQTT... depends where you would expect the increase... the graphs shows the free heap!

but still, after 12 hours running, I think that's a very good PR and a lot of improvments...

you would (from my pov) just need to make sure that the includes are all right (I had to chang about 25 files... other thatn that I think it could be merged, if someone els tests the MQTT part...

TD-er commented 4 years ago

Well, problem was that the ESPEasy-Globals.h was included from more than 1 file. And since it had the global variables, it means variables declared in there would be allocated more than once. One of the variables is the Settings struct, which alone is over 2 kB.

I am now (in this PR) declaring the global variables in the header files using extern and then construct them in a .cpp file. This means you can then include such a header file more than once and it will be the same instance of the object you will be referring to.

Edit: Hmm... Just at the moment I press the submit button of my reply I realize what's going on. Those global variables are allocated on the stack, not the heap. And not just any stack, but the sys stack (Core library territory) We use the cont stack, which is just 4 kB. N.B. This reduction may also prevent issues when the sys stack reaches stack overflow.

uzi18 commented 4 years ago

nice to see you found solution for header files and objects so now road is more straightforward then before =)

clumsy-stefan commented 4 years ago

I had not a single unit rebooting (due to HW-WD or similar) since updating yesterday, so that's already an improvment! As soon as you have pushed this new commit I can recompile and try again.

TD-er commented 4 years ago

I added a new commit which has even more stuff stripped from the ESPEasy-Globals.h into separate file.

@clumsy-stefan Would be nice if you could also so your tests with the last one, mainly to see what has to be changed in the #include paths. If you then could make a PR based on the last commit, then I can merge it this evening. This PR is already getting quite big and more changes could end up making it very hard for PRs to be merged.

If the fixes of include paths is as simple as renaming everything in a subdir from #include src/ to something like #include ../ then it is just a matter of some very basic query replace.

Edit: I made a new commit to make the include paths relative to the file location.

clumsy-stefan commented 4 years ago

Your're latest commit is more or less what I had in my stash before. Basically replacing all src/ with ../ as you wrote..

I have only one more change in SettingsStruct.hline 66 changed from #ifdef USE_NON_STANDARD_24_TASKS to #if defined(USE_NON_STANDARD_24_TASKS) && defined(ESP8266)

and I added an error line in _P068_SHT3x.inoline 192: addLog(LOG_LEVEL_ERROR, F("SHT3x: not initialised!"));

I'm going to compile, install and test it now...

clumsy-stefan commented 4 years ago

@TD-er commit 0b24c7179b637cfeb62720b0e02877bef32ac52d has src/ again in the includes....

EDIT: also src/Globals/MQTT.h and src/Globals/Plugins.cpp are missing a ../../ in the include for define_plugin_sets.h and ESPEasy_plugindefs.h

TD-er commented 4 years ago

@TD-er commit 0b24c71 has src/ again in the includes....

That's in files which are in ESPEasy/src. so the file they include actually has src/.... as prefix.

clumsy-stefan commented 4 years ago

That's in files which are in ESPEasy/src. so the file they include actually has src/.... as prefix.

hmm... No, it's in files that are in src/Globals/ (see above)

TD-er commented 4 years ago

Then it must be a different commit, the one you link only has .ino files in it.

clumsy-stefan commented 4 years ago

Sorry, you're right, these ones are ok.. It's the two files mentioned above (from a different commit) which are wrong... sorry..

TD-er commented 4 years ago

Ah you had edited the comment... Will add them in a new commit

clumsy-stefan commented 4 years ago

yes... also if you could change the define in SettingsStruct.hon line 66... to make it compile on esp32...

TD-er commented 4 years ago

Done in https://github.com/letscontrolit/ESPEasy/pull/2617/commits/75f24711be05e92a770dc5e2221a582e651c37ca

clumsy-stefan commented 4 years ago

wow, you're quicker commiting source changes, than me writing comments 😀

Commit 75f24711be05e92a770dc5e2221a582e651c37ca makes it compile again for ESP32 and ESP8266.. So I'm going to flash some units now, do some tests and tehn flash the rest of them to see how they run with commands, rules, sensors, etc...

PS: crc32.py does not seem to work on esp32 builds, if I'm correct?

TD-er commented 4 years ago

I have never tested that Python script on ESP32, or at least never checked it.

clumsy-stefan commented 4 years ago

FIrst tests show, that all my unit's seem to run fine... commands via controller (fhem) and via web-intrerface seem to work, also the basic rules I have... Memory graph shows more or less flat...

Only issue with the deep-sleep units as descibed. tasks are run before the connection is established therefore no ADC, RSSI, IP, etc. values!

I'll keep it running and can report tmorrow if there were any reboots or anything...

Great work!!!

TD-er commented 4 years ago

Not bad for a pull request containing:

:)

clumsy-stefan commented 4 years ago

did you test the MQTT part? I can't do that one...

TD-er commented 4 years ago

Did you also test changing settings on ESP32? I'm changing the controller and then the settings of a plugin get mangled. Also the other way around, if I edit settings of a plugin, the controller settings get corrupted.

TD-er commented 4 years ago

Ah, bug found.... image Thats not good.....

Found it where it was mangled. Apparently the labels and values got mixed up in the StorageLayout.h Will test it again, but this should now work fine again. Will commit in a minute, but I guess it is best not yet to merge this evening.

clumsy-stefan commented 4 years ago

Oh, as I only have one esp32 I did not check this... but good you found it ;) I'll test later on.

Generally I'm still a bit undecided if I should move to esp32 or stay with esp8266.

Regarding esp32 there are a couple of things that I found:

TD-er commented 4 years ago

Arduino OTA can be enabled on the ESP8266, but isn't by default. It does take quite a bit of resources.

I'm also doing some tests here on ESP32 and the builds are quite stable there. Only thing is the WiFi response time. The ESP32 does not send Gratuitous ARP packets, so it may act slow sometimes. But it is really stable in my tests. Also nice that it has way more memory, GPIO pins and 3 HW uarts. Also proper (de)bounce detection in hardware. It also has Bluetooth :) I do not yet use the 2nd core, since we don't have proper locking in our data structures.

And some boards have PSRAM, which allows to have 4 MB of heap or even more. I already saw some boards with 8 MB PSRAM or even more.

I have one board here running GPS on ESP32 and it is really really stable.

clumsy-stefan commented 4 years ago

Arduino OTA can be enabled on the ESP8266, but isn't by default.

I have it enabled, and it works (espota.py tells ans shows me it uploads succesfully) after that it reboots, but stimm with the old version active... I have no clue why it doesn't work..

TD-er commented 4 years ago

The default port was 8266, if you also use that port for ESPeasy p2p, you have a conflict :)

I just tried ArduinoOTA on my ESP32 unit and it does work, but had to try it at least 3 times to complete. The last time I did send pings during the update and it did succeed. (not sure if that was the reason it worked)

clumsy-stefan commented 4 years ago

I use non-standard port and upload succeeds on both, esp32 and esp8266, but only on esp32 it gets activated, on esp8266 it just loads the old version....