Discussion on how to define 24 task settings

[TD-er]

There has been a lengthy discussion already about how to implement the patch to use 24 tasks on ESP8266 ESPEasy. However this was posted below a pull request, which makes it hard to follow and rather invisible to others unless they track all changes and updates on this repo.

So let's continue here.

The most useful parts of the discussion:

• Arduino IDE compilation was (again) broken by my attempt to clean up the code.
• Splitting defines into separate files made it hard to apply a user defined patch for 24 tasks which had been working for a while.
• How to add/suggest some kind of universal define to make switching to a different set of tasks more reliable and reproducible
• Other defines a user may want to set using a simple header file for creating a personalized build.

@uzi18 has some set to define 24 tasks: https://github.com/letscontrolit/ESPEasy/commit/f7e0d7cfdde65d9503ff08fd92d48681dc8da702#commitcomment-35122941

#define TASKS_MAX                          24
#define DAT_OFFSET_CONTROLLER            4096    // 0x1000 each controller = 1k, 4 max
#define DAT_OFFSET_CUSTOM_CONTROLLER     8192    // 0x2000 each custom controller config = 1k, 4 max.
#define DAT_OFFSET_TASKS                16384    // 0x4000 each task = 2k, (1024 basic + 1024 bytes custom), 12 max

@clumsy-stefan has another (incompatible) one: https://github.com/letscontrolit/ESPEasy/commit/f7e0d7cfdde65d9503ff08fd92d48681dc8da702#commitcomment-35126444

#define TASKS_MAX                          24 // max 12! 
#define DAT_OFFSET_CONTROLLER            DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX)                        // each controller = 1k, 4 max 
#define DAT_OFFSET_CUSTOM_CONTROLLER     DAT_OFFSET_CONTROLLER + (DAT_CUSTOM_CONTROLLER_SIZE * CONTROLLER_MAX)  // each custom controller config = 1k, 4 max 

[clumsy-stefan]

One thing I'd like to add to your list is the possibility to export/import settings in some structured way (json, etc.), this would make changes in the config structure much less of an issue for users when they can somehow easily backup/restore a config across versions!

[TD-er]

That's another issue, which has been a feature request for a long time already. The problem with serializing/de-serializing structs is that you also need to include their variable names as text in the binary. Since we're already struggling with the binary size, I don't think that's a good idea right now.

[clumsy-stefan]

Not sure if I understand, but the variable-name is in the settings file not the binary? Basically a json export of the config page would be enough? Ialready thought about hacking something together which exports the config-pages in some kind of HTML and then does a series of post-request to put in the values again... I think even if it's an external script (python, perl, shell, etc.) that fetches the configs in some text format and pushes it again if needed, would already help! IMHO...

[TD-er]

For example, if I want to export this struct into JSON:

struct Foo {
int bar;
};

Then I need to create a JSON with the name "bar" in it. Also when reading the JSON, I need to look for a name "bar". So this means the string "bar" must be included in the binary. The same for all other variables in the struct. Also the string name for these variables may not change, so you cannot use the "display name" string for it, since someone may want to change that if there is a typo in it for example.

This means you will have some "internal string" and a "display string" to represent the same variable.

[TD-er]

The so called "new GUI" feature actually does interpret the binary file in JavaScript, thus in the browser.

[TD-er]

About the defines you use. They depend on other defines. Is it possible those defines are not yet known when your define is being initiated?

[clumsy-stefan]

Yes, I also realised thet, that's why I also included in my Custom.h files all #define's that I use in the new definitions further up above... It's quite likely however that I missed something there, even after double-checking... or that one of these fixed defines had been changed before somewhere dynamically... I'm also still digging there...

[TD-er]

Maybe just define these values with an integer value instead of computing it.?

[clumsy-stefan]

I hate fixed-values for things that can be computed 😄 I think that's error prone and double work... :grin . And as it worked before, there has to be a mistake somewhere (which probably also has an influence to other things)...

[TD-er]

Well, before you were using it in the globals header file, but now the order in which things may be defined has changed.

And I do agree it is error prone, that's why we should handle it in the core with a single define. I think @uzi18 has the more useful layout then, since he has the tasks at the end.

[clumsy-stefan]

but this would mean to manually reconfigure some 50 nodes currently, when I change the config-file structure... :(

[TD-er]

Well, it is not being said we should only use one ;) There may be others using those settings if you gave them via a forum post or something? The same applies for @uzi18 's layout parameters.

[clumsy-stefan]

There may be others using those settings if you gave them via a forum post or something?

I got mine from a forum posting actually...

[clumsy-stefan]

Tried PR #2617 as suggested by @TD-er and included my changes to have 24 Tasks as I had them before the cleanup merge. Unfortunately I still experience the same issues (runtime) again, which I did not have before the cleanup merge.

One of them is, that task nr. 13 (and it seems only that one) "looses" task- and variable-names after pushin "submit". However they are still present and when closing the config page they show up again.

The worse thing is, that there seem to be an issue with (some) I2C (probably due to config-corruptions). Even though in "scan I2C bus" the sensor shows up, the task that reads from the sensor (an SHT30) is eitehr never started or does not provide a result. In the debug log it does not create an event. some screenshots attached.

So it seems, that moving around the defines had other impacts on Custom.h defines. I n my Custom.h I use (as before the cleanup merge):

  #define DAT_TASKS_DISTANCE               2048  // DAT_TASKS_SIZE + DAT_TASKS_CUSTOM_SIZE
  #define DAT_TASKS_SIZE                   1024
  #define DAT_TASKS_CUSTOM_OFFSET          1024  // Equal to DAT_TASKS_SIZE
  #define DAT_TASKS_CUSTOM_SIZE            1024
  #define DAT_CUSTOM_CONTROLLER_SIZE       1024
  #define DAT_CONTROLLER_SIZE              1024
  #define DAT_NOTIFICATION_SIZE            1024

  #define DAT_BASIC_SETTINGS_SIZE          4096

  #if defined(ESP8266)
    #define DAT_OFFSET_TASKS                 4096  // each task = 2k, (1024 basic + 1024 bytes custom), 12 max
    #define DAT_OFFSET_CONTROLLER           28672  // each controller = 1k, 4 max
    #define DAT_OFFSET_CUSTOM_CONTROLLER    32768  // each custom controller config = 1k, 4 max.
    #define CONFIG_FILE_SIZE                65536
  #endif
  #if defined(ESP32)
    #define DAT_OFFSET_CONTROLLER            8192  // each controller = 1k, 4 max
    #define DAT_OFFSET_CUSTOM_CONTROLLER    12288  // each custom controller config = 1k, 4 max.
    #define DAT_OFFSET_TASKS                32768  // each task = 2k, (1024 basic + 1024 bytes custom), 32 max
    #define CONFIG_FILE_SIZE               131072
  #endif

  #define CONTROLLER_MAX                      3 // max 4!

#ifdef TASKS_MAX
#undef TASKS_MAX
#endif
#define TASKS_MAX                          24 // max 12! 

#ifdef DAT_OFFSET_CONTROLLER
#undef DAT_OFFSET_CONTROLLER
#endif
#define DAT_OFFSET_CONTROLLER            DAT_OFFSET_TASKS + (DAT_TASKS_DISTANCE * TASKS_MAX)                        // each controller = 1k, 4 max 

#ifdef DAT_OFFSET_CUSTOM_CONTROLLER
#undef DAT_OFFSET_CUSTOM_CONTROLLER
#endif
#define DAT_OFFSET_CUSTOM_CONTROLLER     DAT_OFFSET_CONTROLLER + (DAT_CUSTOM_CONTROLLER_SIZE * CONTROLLER_MAX)  // each custom controller config = 1k, 4 max 

[TD-er]

Hmm, this is probably not a good idea:

#ifdef TASKS_MAX
#undef TASKS_MAX
#endif
#define TASKS_MAX                          24 // max 12! 

Then you have no idea what value for TASKS_MAX is being used. In PlatformIO, you can define such things on the "command line" to make sure they are defined before any other define is being called. Not sure if something like that can be done in Arduino IDE?

[uzi18]

#ifndef TASKS_MAX
#ifndef TASKS_24
#define TASKS_MAX                          12 // max 12!
#else
#define TASKS_MAX                          24 // max 12!
#endif
#endif

will read all discussion later ...

[clumsy-stefan]

Why not? Like this it's made sure, that TASKS_MAX is always 24, independent if it's already defined before or not...

[TD-er]

Why not? Like this it's made sure, that TASKS_MAX is always 24, independent if it's already defined before or not...

Nope, for the simple reason that we don't really know when that change of you is being interpreted. It is becoming more clear to me these global defines are really hard to predict when they will be evaluated.

I have to think about what to do here, since it does seem these will be evaluated at some time when the structures depending on them may already being handled by the pre-processor.

[clumsy-stefan]

ok, understand.. but it still seems related to moving the things around, as it worked before...

one other thing I just found, I flashed another node with my 24-task test build (with your PR) that has multiple I2C sensors. It seems, that only the SHT30 has an issue (see screenshots). Probably something directly in that Plugin (it uses direct I2C access and not only the ESPEasy functions). However this did not happen before the cleanup-merger... so this could likely be another sepparate issue...

[TD-er]

ok, understand.. but it still seems related to moving the things around, as it worked before...

It IS related, although I am now wondering to WHY it is related.

[clumsy-stefan]

did you test a SHT30 with your "standard" build? I can't check if it works with teh standard 12Task build see if it's related to the cleanup or to the 24tasks....

[uzi18]

@clumsy-stefan @TD-er about why my patch is different:

• if use original offsets with 24 tasks some data overlap another from tasks on controllers data, as we use max 4 so it could be safely moved just after general settings, this provide incompatibility
• it does not change config file size

@clumsy-stefan idea is far more clean maybe to move controller settings just after tasks in structure, but config file may grow a little

[uzi18]

about configuration file, with python script it should be possible to convert one to another

[TD-er]

about configuration file, with python script it should be possible to convert one to another

Sure, but then we should write one to interpret the binary blob. Almost all of it has already been done by @ppisljar but then in JavaScript, which is then run in the browser. We will need the browser part in the future I guess, so I am a bit reluctant in having 2 interpreters for the data, since we then have to maintain 2 versions (and I already don't understand the JavaScript version, or at least how to maintain it)

[uzi18]

I think more about Stefans automation for configuration conversion =) just in case

[clumsy-stefan]

it's actually not a "conversion" issue but a general "backup-restore" in binary independant format... eg. if you want to move from a ESP8266 to a ESP32 with the same sensors attached... it would be cool to just dump and restore the config... or even modify the baclkup file and upload it again... or for fully automated configurations etc...

[TD-er]

True, but where should this be executed? The vagrant environment could do it I guess...

[clumsy-stefan]

P068_SHT3x plugin seems to fail here:

      SHT3X* sht3x = static_cast<SHT3X*>(getPluginTaskData(event->TaskIndex));
            if (nullptr == sht3x) {
        addLog(LOG_LEVEL_ERROR, F("SHT3x: not initialised!"));
              return success;
            }

sht3x always returns a nullptr, not sure yet why...

[TD-er]

It could be this object is considered not to have changed so the linker may include one already built into an .o file. Can you make a clean build?

[clumsy-stefan]

ArduinoIDE always does clean builds, as I think, could not find any .o files anywhere... reverting git to before this merger and compiling it again (in the same tree) makes it work again...

[ppisljar]

adding to the discussion, the new GUI (as TD-er mentioned) parses the configuration from espeasy into json. we can store this json and load it on another ESPeasy (maybe running on esp32) and save it back into binary format. We could even add managing capabilities to the ui which would allow you to have more than allowed tasks defined in your browser and then store just the ones you want to ESP.

i plan to spend some time on moving the new UI forward in the following month.

[TD-er]

I think I have an idea what's happening here. Not sure yet if it is due to the way how Arduino is handling these files or maybe it is a very standard C++ thing.

What we're seeing here is this:

• Some code operates like TASKS_MAX is set to 12.
• Some other code uses TASKS_MAX = 24.

I am not entirely sure yet if the preprocessor is working in 1 run or multiple runs. It could be the preprocessor is just filling in the blanks it already knows and later evaluated the left over #ifdef parts. We already know that (at least PlatformIO) is not always handling the existing #ifdef wrappers the way we expect when generating forward declarations. Maybe there is also some issue here where defines set in other header files only will be evaluated by the preprocessor when they are being processed. For example:

Globals.h:

#include "limits.h"  <-- Defines TASKS_MAX
#include "custom.h" <-- Updates TASKS_MAX

byte array[TASKS_MAX];  <-- What size is it?

byte getValue(byte index);

In globals.cpp:

byte getValue(byte index) {
  if (index < TASKS_MAX) {return array[index];}  <--- What is TASKS_MAX here?
  return 0;
}

[clumsy-stefan]

ok, I thought that's handled serially.... at least it seemd to have before...

[TD-er]

ok, I thought that's handled serially.... at least it seemd to have before...

I think that has to do with the fact it used to be all in a single .h file.

[clumsy-stefan]

Ah, understand... now it could be that src/Datastruct/* could be precompiled before the Custom.h is actually included... (or so..)?

[TD-er]

It does seem indeed that some of these values may change after some parts of the code are already being dealt with by the preprocessor.

[TD-er]

So either we must make sure a define is only defined once and not changed, or we must use get functions on runtime parts. I think we may also have similar issues with the selection of which plugin to include, since the build size does increase a lot more than I would expect it to be.

[clumsy-stefan]

I guess that's an issue with all #define's (especially in Custom.h) that are specified in different places (centrally and lcoally), and could lead to all kind of strange effects..

[TD-er]

Well the difference is that they should be defined first and if not defined then set to the defaults.

[clumsy-stefan]

So you would need to make sure, that the Custom.h (or wherever the global defines are) are always and everywhere included somehow.... (can be casted if already included somewhere else) eg. some kind of an universal #include at the start of every file....

[TD-er]

Or we should stop using custom.h as something optional, but something that has to contain all defines or otherwise use the defaults. Or go the PlatformIO way used in the custom build by using Python to define all from the build command line.

[TD-er]

What if you move this part into the globals.h file

/*
    To modify the stock configuration without changing this repo file :
    - define USE_CUSTOM_H as a build flags. ie : export PLATFORMIO_BUILD_FLAGS="'-DUSE_CUSTOM_H'"
    - add a "Custom.h" file in this folder.

*/
#ifdef USE_CUSTOM_H
#include "../../Custom.h"
#endif

[TD-er]

I've thought about it and to me it does look the best if we make all user configurable stuff be handled in the Custom.h and make it mutual exclusive. So either you use the Custom.h and have to define all, or you use the values from ESPEasy. If you're missing a value in the Custom.h, you will have a failing build.

Or should all defines be made using #ifndef statements? The include of the Custom.h should then be one of the first lines in the code compilation.

Edit: I think including it here in ESPEasy-Globals.h would be enough:

#ifndef ESPEASY_GLOBALS_H_
#define ESPEASY_GLOBALS_H_

#ifndef CORE_POST_2_5_0
  #define STR_HELPER(x) #x
  #define STR(x) STR_HELPER(x)
#endif

#ifdef __GCC__
#pragma GCC system_header
#endif

/*
    To modify the stock configuration without changing this repo file :
    - define USE_CUSTOM_H as a build flags. ie : export PLATFORMIO_BUILD_FLAGS="'-DUSE_CUSTOM_H'"
    - add a "Custom.h" file in this folder.

*/
#ifdef USE_CUSTOM_H
#include "Custom.h"
#endif

#include "ESPEasy_common.h"
....

[uzi18]

should be like this:

#include "custom.h" <-- Defines TASKS_MAX
#include "limits.h"  <-- Updates if not definied out of limit

[TD-er]

I think it should be like you suggested earlier and also discussed before. We must have one to define the 24 tasks parameter, which does set them all.

[clumsy-stefan]

Or should all defines be made using #ifndef statements? The include of the Custom.h should then be one of the first lines in the code compilation.

I'd vote in favor of this one, but I'm not a really structured coder ;)

I think including it here in ESPEasy-Globals.h would be enough:

Not sure if that's enough, as this file does not get included everywhere I think...

[TD-er]

See latest commit and try building using the USE_NON_STANDARD_24_TASKS flag.

@uzi18 Should we also make a USE_NON_STANDARD_24_TASKS_alt version for you?

[clumsy-stefan]

See latest commit and try building using the USE_NON_STANDARD_24_TASKS flag.

this works as good (or bad) as with what I had in my Custom.h previously... Meaning that task 13 still shows empty names after pushing "submit" and SHT30 not working...

but at least someone else should verify, if its because of this the 24 tasks or if there are other issues in my environment (which I tried to rule out as much as possible)...

[TD-er]

I'm going to test myself too, but that's using PlatformIO.

You can also use some tests like these at the places where you suspect some issues:

  #if defined(USE_NON_STANDARD_24_TASKS) && defined(ESP8266)
    static_assert(TASKS_MAX == 24, "TASKS_MAX invalid size");
  #endif

These static_asserts will fail the build if they are not passing the test.

[clumsy-stefan]

I'll try that... I just put it in every file at the beginning then 😁