Dynamic loading of modules

[wolfgangr]

Free of the creative restrictions I felt in the tasmota ecosystem, I dropped at least 10 proposals from the IoT-vision-corner of my mind to the issue list here within 24 hrs.
Just 100 other people doing the same, module number might easily grow into the thousands. Tough task to manage....

At the moment, module configuration is performed by preprocessor directives at compile time, as far as I can figure out. Even after dropping the one-binary-fits-it-all policy, users might try to include at least all modules they might have user for in the foreseeable futures. However, Flash size, update time etc still pose their limitations. OTA is still in the Project Label - roughly doubling flash space requirements.

So were it possible to have modues loaded / linked dynamically? Similar to that what the linux kernel does? One small binary with the core functionality like "generic ESP8266" in tasmota, and any sensors etc loaded as separate files (or bundles) over the net? As far as my understanding goes, there are a few *event-like callbacks in any module, hooked into the "kernel" aka sonoff.ino. I think the compiler translates that to functions, called by their memory adress pointers. Could that not be manipulated at runtime? OK, I have no clue yet how flash space is managed. But this is required for the OTA functionality, anyway. And isn't there some functionality of a file system (spiffs) in the arduino core?

[WelterRocks]

As far as I can say, it is currently not possible to load modules, like the kernel does. This is not only a compiletime limitation by the dev environment. I dropped Arduino IDE support to make things a bit more flexible. But PIO also has its limitations, which starts with the file ecosystem you have to use. One thinkable possibility to work around that, is to adapt the PIO pre compilers, e.g.

[wolfgangr]

I'm afraid PIO will stay in alpha...to...beta state the next years. Always moving, always buggy, great vision, attracting many people but always a moving target.

I have a X-windows where I can move around as many edit windows I like. I have consoles as many I like, and edit windows as many I like, with the editor I prefer. And any projects I started from scratch now I managed to build with makefiles. And as far as I can remember, even Nodemcu with a similiar config complexity than tamota but much more mature has a makefile. Keep it as simple as possible, and under full control. Leave the coloured blinky stuff for the kids at tasmota (sorry....)

[wolfgangr]

But the issue is not about IDE here - its about loading of modules. As far as I can remember from my first looks into sonoff.ino (which I consider to be the "kernel") there are some preprocessor hooks that link the drivers into relevant event processing (loop, 25 ms, 100 ms, init...) at compile time.

As far as I can remember from my Nodemcu, the modules there expose pointers to functions. The kernel then can manage a list of them (presumably as chained list, I think) and call the function at run time. I'm neither C nor C++ professional, but in one of my earlier lifes, I used to work in assembler (handcoded at checkerd paper... ) so I am sure it is possible. Maybe some nanny settings in "modern" compilers try to tell you not to do so, until you explicitely force them. So some RTFM may be required. And of course, the code must be small and thoroughly tested, because with bending pointers to function you easily screw your system by sending the CPU into the forest. That's what bluescreens where about.....

[wolfgangr]

Just as an example, how it looks like in NodeMCU.

https://github.com/nodemcu/nodemcu-firmware/wiki/%5BDRAFT%5D-How-to-write-a-C-module#debug-and-error-messages

Here is an Example https://github.com/wolfgangr/nodemcu-firmware/blob/net_if-000to-stdout/app/modules/net_if.c (This one I wrote myself two years ago, so I hope I can manage to understand it again, when required ;-) )

In the body there are ordinary C-function as in the tasmota XYZ-99.ino files. The interface in the end looks slightly different:

Basically you have pairs of label -> function pointers. Of course, the interface (parameters required) has to be standardized. In NodeMCU that is handled the LUA way - implemented in the LSTRKEY and LFUNCVAL macros. So any of the functions has to pull its parameters from the LUA stack, which requires to aquire a LUA state first, which opens the quagmire of ESP multithreading .

So, I would prefer it more C'ish. Maybe just a single pointer, or no variable at all. Global variables are a nice thing on small gadgets.

The last directive is called by the "kernel" at startup. It is the variable that is linked into the module list. Thus you have a two level namespace: "module.function" in this case "net_if.list_adr", "net_if.first_adr" and so on.

// Module function map
static const LUA_REG_TYPE net_if_map[] = {
  { LSTRKEY( "first_IF" ),             LFUNCVAL( first_IF ) },
  { LSTRKEY( "first_adr" ),             LFUNCVAL( first_adr ) },
  { LSTRKEY( "first_adr2" ),             LFUNCVAL( first_adr2 ) },
  { LSTRKEY( "list_adr" ),             LFUNCVAL( list_adr ) },

  { LSTRKEY( "__metatable" ),      LROVAL( net_if_map ) },
  { LNILKEY, LNILVAL }
};

// Register the module - NODEMCU_MODULE()  
NODEMCU_MODULE(NET_IF, "net_if", net_if_map, NULL);

As far as I remember, you still have to decide which modules are compiled by some_comfig.h So you do not save flash space this way. But when you do not load modules you don not need at runtime, they will not occupy RAM, GPIO, CPU time ....

In the next step, I would like to be able to add modules at run time, that were not compiled in beforehand. If it were that sinple, why did they not include in NodeMCU? hm....

Well, I see at least these challenges:

• The code must be relocatable - so only relative, no absolute jumps inside
• we must manage flash space - mabe we would like a file system. There is a spiffs library in Arduino-ESP8266-core. Maybe that fits the purpose.
• we must manage file transfer. I think that we can model along the OTA mechanism already implemented.

I dont see that dynamic loading as an urgent requirement in the near future. But I think architectural decisions should be wise to keep this road open. Above all the way modules are linked into the kernel.

[wolfgangr]

So for comparison, the tasmota module interface: There is obviously only one function - Xrdv20 - with a tokenized sub-selector (byte value defined in mnemonic #defines) I haven't checked yet how the kernel accesses it. Actually, I expected it also to be done by function ref... But, Yes, you could link it statically by conditional defines.

boolean Xdrv20(byte function)
{
....
  switch(Settings.module)
  {
      case LC_TECH_RELAY1:
        LCTNumDevs = 1;
        LCTBaudrate = 9600;
        break;
    ....
  }

  if (LCTNumDevs > 0)
  {
    switch (function)
    {
      case FUNC_MODULE_INIT:
        result = LCTModuleSelected();
        break;
...
      // hook update handling function into main loop
      case FUNC_LOOP:
        result = LCTLoopHandler();
        break;
....

Mh.... myself corious for the answer....

ah, here it is: Sonoff-Tasmota/sonoff/xdrv_interface.ino

#ifdef XFUNC_PTR_IN_ROM
boolean (* const xdrv_func_ptr[])(byte) PROGMEM = {   // Driver Function Pointers
#else
boolean (* const xdrv_func_ptr[])(byte) = {   // Driver Function Pointers
#endif

#ifdef XDRV_01
  &Xdrv01,
#endif

So they have a linkable interface in the drivers, already. But they use this crutch to convert it to static links - no, not really - to a static list of pointers at compile time.

Ah, and at the end of the same file, there is how this list is processed:

boolean XdrvCall(byte Function)
{
  boolean result = false;
  for (byte x = 0; x < xdrv_present; x++) {
//    WifiAddDelayWhenDisconnected();
    result = xdrv_func_ptr[x](Function);
    if (result) break;
  }
  return result;
}

OK, traversing a linked list is not much more complicated than a for loop. Building it is a nice programmers exercise, but not a prohibitive one.

But first the questions are about architectural design. I definety would like to dismiss this funny XDRV_## chain. I would prefer to allow a namespace with at least free ascii string. Maybe some kind of hierarchy.

Do we want to stick with the distinction of xdrv, xdsp, xnrg, xplg, xsns, type drivers? Or can we lump them all together? Create a superset of all interfaces and the driver itself decides by the subset of its implementation whether it is more drv, nrg, sns...?

[wolfgangr]

Ah, now I begin to understand how this relates to IDE constraints. My next thought was some shell or perl script at make time, scanning all drivers and the config at hand, and producing some output of how and what these were to be included.

Basically a autogenerated x???_interface.ino

If your IDE does not have a widget for such a script to run, you are bad off, aren't you?

[WelterRocks]

As you can see in WESPOTA src directory, I even tried to change the way, modules are loaded. First, I thought of making a sub directory tree, where the modules are placed, compiled and then dynamically linked, when they are needed.

But PIO throws its middle finger to me. Automatic inclusion is only done for a specified path and only for c or cpp code, not ino files. Only the main directory you specify in platformio.ini is scanned for ino files to preprocess. Therefore the tasmota guys made the XYZ_ file stuff. The files have to be preprocessed by PIO in correct order or linking will fail. Also automatic sub directory compiling is not an option, because PIO doesnt support it, yet. Also, a creepy poitner file exists, where the loaded drivers and modules have to be hardcoded, so that the PIO preprocessor understands it.

If we change the PIO preprocess subroutines, we could make module autoloading possible. No other idea yet, how this should work, without completly dropping PIO.

[wolfgangr]

As you can see in WESPOTA src directory,

I'm really running out of time already. I trust you we share fundamental Ideas - so I hope you will do it right. But I will put this glance on my todo, of course...

But PIO throws its middle finger to me.

no surprise at all. Good Idea to get rid of that and regain control.

scanned for ino files

Do we need the .ino extension? I feel there is a lot of obscurity connected to this old heritage of early arduino. I don't know for sure, but that's what obscurity is about. If possible, I'd prefer to gain control instead of playing around other peoples work.

without completly dropping PIO.

Have my 3 cents: drop PIO. Immediately.

Also drop the arduino IDE. Keep the arduino-ESP8266-framework, however. At least for the next round. There is a lot of experience with coexistence to the esp-core distilled in it. Don't want to reinvent that all.

[wolfgangr]

First, I thought of making a sub directory tree,

I really had preferred that... I encountered the problem of the opaque building machine as well, but didn't spend enough time to penetrate the issue. I think I would love to see you staying at your decision, overcoming the automagicalism and craft a customized makefile for a subdir tree.

You optimised the tasmota file naming scheme a little bit, so the two tier naming scheme becomes a bit more clear and maybe more flexible that way. But is this gain enough to justify a complete break with tasmota? No chance to merge any more, as I'm afraid?

This is a high price, and I think, the gain to be expectes must be worth paying it.

I've seen there is a makfile based arduino build arund ..... https://github.com/esp8266/Arduino#building-with-make ... BUT .... having a look at it ..... https://github.com/plerup/makeEspArduino/blob/master/makeEspArduino.mk .... it still feels like loooooots of obscurity. Well, at least concentrated at one single point of maintenance. 583 lines of condensed obscurity to work through, so to say. I would not be surprised if PIO under the hood would use right this makefile.

Oh, I just wanted to click it away, when I saw it mentioning spiff. Maybe I can learn from it...???

And it starts with a comment

# Include possible project makefile. This can be used to override the defaults below

.... hm... I'd still prefer learning from it over using it in it's whole complexity...

A clearly organized makefile where everey file to include in the build is maintained manually would be my prefernce. Maintainig it is not more work than maintain any other format that feeds some automagic monster. Building it bottom up, starting with an empty file. So it will not contain anything you do not thoroughly understand. I know it might be hard with the esp-machine, which will remain opaque to some level. Mabe you could learn from the NodeMCU which is built by a much shorter makefile than the arduino one, as far as I can remember.

[wolfgangr]

@VerboteneZone Hi Oliver,

Do you have expertise / tools / informations / pointers on the management and monitoring of memory management in ESP8266?

My best source regarding theses issues up to now: https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot/posts/1501224 However, it raises more questions than answers.

Before that, I haven't realized that (spi serial) flash access in the ESP8266 architecture is tremendously slow and some ram caching mechanism is implemented, therefore. I may conclude that severe cache access timing penalties may arise if code is spread over a large area of flash space.

Could this explain the long loop time (100 ms) I measured with my LCtech experiments? May it be that any module that is compiled into tasmota, even if not used, poses such high caching load? In addition to RAM footage? How can we measure that? If so, how can we avoid that?

[wolfgangr]

A quick primer on the problem: http://www.danielcasner.org/guidelines-for-writing-code-for-the-esp8266/

loading code from SPI flash is relatively slow and there is a limited code cache in local chip RAM. Hence inner loops and code which executes often should be kept relatively small to reduce cache thrashing.

Application code should have the ICACHE_FLASH_ATTR decorator unless it is executed very often.

A quick GIT search in our tree reveals that ICACHE_FLASH_ATTR is only used in mqtt lib, not in tasmota core code. So, is it safe to conclude that 'till now, the issue of flash caching performance has not yet been properly addressed?

[wolfgangr]

This supposed-to-be-datasheet is not really revealing more on the issue https://cdn-shop.adafruit.com/product-files/2471/0A-ESP8266__Datasheet__EN_v4.3.pdf ... just compare to a datasheet for AVR or STM ....