search

WelterRocks / WESPOTA

WESPOTA is a Sonoff-Tasmota fork, designed to support as many devices as possible, not only Sonoff and equivalent.
GNU General Public License v3.0
15 stars 0 forks source link

Advanced rule processing #15

Open wolfgangr opened 5 years ago

wolfgangr commented 5 years ago

Yes, There is a rule processor available in tasmota. Haven't worked yet with it, but I'm not sure whether I really want to do so. To me, it appears both quite quirked and very limited in capabilities.

Do we have better options? And - do we WANT better options, at all?

Yes, I would like a generic programming language, with a touring capable command interpreter, capable of anything that might be programmed. But do we end up then with people inclined to pull their whole automation system to one tiny IoT device? If so, is it a problem? It's any users own responsibility to explore limits of the platform at hand.

What are the alternatives? I don't want to reinvent the wheel.

There is LUA in the NodeMCU framework. I really liked it, when I first encountered it, although it is not a well known language at all. While coined to live on small systems, at least in NodeMCU, the LUA dominates anything else, severely inhibiting hardware side development. Have been there, have tried that. Lesson learned: I'd prefer a command interpreter living below hardware modules, not hardware modules living below the command interpreter.

Long time (5yrs?) ago, I played with microPython on GPS modules. No Idea, how it is implemented and how big its footprint is.

Long long ago (> 30 yrs) I played with FORTH language. Weird thing, easily expandable, small startup footprint. Fast interactive implementation of close-to-metal tasks. Recently, I encountered it as IP-kernel implemented on FPGAs: Processors, implemented as firmware on gate arrays, running FORTH as their native machine language. Quite sure, the ESP chip would be capable of doing things like that, but far beyond my knowledge and time budget. I'll search for a contemporary C implementation.

There is the infamous bash command syntax, implemented by the busybox module on small systems. Is something like this available in a even more stripped down version? On the plus side, many linuxed admins are quite familiar with that. Admittedly, it's quirkiness even outperforms tasmota rules. On the other hand, bash programming heavily relies on the power of linux commands, files, pipes ans other goodies of a full fledged multitasking system - not that what we have.

There is my favoured PERL, but I've learned that this is an old men's problem ... and PHP, and JAVA, and BASIC, ...

Any better Ideas? Or may be, the tasmota rule processor is alread a good choice? Where does it come from? What are its limitations?

wolfgangr commented 5 years ago

I'm just doing research, and the get more and more enthusiastic by the idea of implementing FORTH within the tasmota-framework.

My vision:

I'd try to insert FORTH into the framework as a driver. So it's called in the loop and can do whatever the user tells it to do. Including low level access to all modules - be they active or not. This way, for a transitional state, there is no need to drop the existing implementations. They can either be used as standalone as they are till now, or as backends for FORT snippets, if there is more complex stuff to do.

I don't expect anybody to share my enthusiasm at this early stage. Mabe it is safe not to do so. But if I could infect sbdy, I'd be glad for a short note.

wolfgangr commented 5 years ago

There are more challenges in the search for a proper FORTH implementation to start with than I expected.

The well known FORTH implementations advertised for Microcontrollers e.g.

The "old forth people" consider writing forth for any platform as "simple and easy". So I started analyzing this project: https://github.com/kristopherjohnson/cxxforth It consist of basically one very well documented c++ file with about 3000 lines, 3/4 of them resembling comments.

I'll keep you informed.

wolfgangr commented 5 years ago

I think that it would be possibe to have something like cxxforth compiled and threaded into the tasmota framework.

To get an idea whether it's worth the effort, I try some tests now

wolfgangr commented 5 years ago

I'm still in the stage of getting somewhat acquainted to the language. Have not yet tried to play with the I/O, still practicing on a naked NoceMCU module.

But I think it is already safe to assume that basically any driver might be implemented in FORTH. In an interactive way, at a much lower development time (once one has mastered the language, of course), at less memory footprint, at ~ 30 ... 50 % runtime performance penalty, compared to C (this is what the ads say).

Compare to the current situation: When I debugged my LCTech-4-CH-patch, I learned that the main loop in sonoff.ino was only called once every ~100 ms. For a 80 MHz Processor doing basically nothing, this is a lot of work - done for nobody knows what...

The question is whether it is worth to press forth into this apparently already overcrowded tasmota - and maybe end up with similiar probems than NodeMCU. Forth is capable of running standalone, being it's own operating system. So I f I could find or craft a simpe web server and MQTT module ON TOP of Forth, I might seriously consider to refrain from a complete rework of tasmota.

Instead I'd:

What I do not yet know

wolfgangr commented 5 years ago

btw: interesting article: http://www.complang.tuwien.ac.at/anton/euroforth/ef15/genproceedings/papers/peri.pdf

IoT widgets not only exchanging information to each other, but even programming each other. Not sure whether I want to administer stuff like that, but it highlights the power and flexibility :-) of Forth on IoT widgets.

wolfgangr commented 5 years ago

Glanced over the building and installation process of cforth. While it looks quirky when it runs, it is scripted by a couple of makefiles in a quite transparent and straightforward way.

wolfgangr commented 5 years ago

I looked for this embed.o on my build tree, but could not find it. There are two binaries flashed with esptool.py 

-rw-r--r-- 1 wrosner users  24768 Jan 18 23:41 0x00000.bin
-rw-r--r-- 1 wrosner users 324436 Jan 18 23:41 0x10000.bin

This includes the complete ESP-SDK core, not only the Forth. There is a tembed.o in the build dir with 343 kByte and tforth.o with 156 kByte, among others, but on their exact interpretation I cold only speculate. Should spend some time with xtensa-lx106-elf-objdump, presumably.

wolfgangr commented 5 years ago

So even if this 350 k were the extra size, this is still less than the half of flash size tasmota tries to spare for OTA. I think with a high level language, OTA is not really necessary, since the development / deployment cycle in most cases will not require reflashing.

I envision the "bootstrapping" of a new driver or any other application as follows:

wolfgangr commented 5 years ago

My grep just found this line in the source code of cforth:

./src/cforth/util.fth:599:' noop to pause               \ No multitasking for now

I'm afraid this could be a game killer. Without any basic multitasking support from the language, coding of concurrent threads waiting for different events becomes a burden we do not want to face, I think. May have a look to the examples, may be it is a question of 'how-to'.

But nevertheless, cforth just lost mosts of its points in the contest.

In comparison: https://github.com/zeroflag/punyforth#tasks-experimental

Punyforth supports cooperative multitasking which enables users to run more than one task simultaneously. For example one task may wait for input on a socket, while another one receives commands through the serial port. Punyforth never initiates a context switch by its own. Instead, tasks voluntarily yield control periodically using the word pause. Tasks are executed in a round robin fashion.

In order to run some code in the background, one must create a new task first, using the task: parsing word.

sounds much better to me....

wolfgangr commented 5 years ago

punyforth tests are not more encouraging either, yet. Simple beginner's error (pulling from an empty stack) lets the module reboot and obvioulsy screw up it's dictionary permanently. Even reflashing it doesn't help. But the module knew my wifi config from earlier trials. So I assume that has to do with some broken config in protected space.

Complete flash erase does not help. Trying some forth words from the standard (e.g ." or words) which are not implemented - throw error messages - OK. Examples from the readme throw errors as well - not so OK. But after that, the thingie remains in some unresponsive state and needs reboot??? not OK at all? Ah, it still is in compiling state after the error .... a simple [ will help.

OK, we have so far

... still not encouraging ..... but i woll not give up that easy :-)

wolfgangr commented 5 years ago

Ok, we are on FOSS. So - if there is no manual to do RTFM, RTFS (Read The F....antastic SOURCEcode ) will help.

The punyforth core source tree https://github.com/zeroflag/punyforth/tree/master/generic looks quite manageable:

       ...../punyforth/punyforth$ tree generic/
|-- data.S
|-- forth
|   |-- core.forth
|   |-- decompiler.forth
|   |-- deprecated.forth
|   |-- punit.forth
|   |-- ringbuf.forth
|   |-- ringbuf_test.forth
|   `-- test.forth
|-- macros.S
|-- outerinterpreter.S
`-- words.S

The assembler files here are not seriuous bare to metal codes, but concatenated pointers. I suppose it is a "handycrafted" basic forth dictionary - without the need for a forth residing on the build host. And the rest is .... well ... forth ..., of course.

Ah, there is more in punyforth/punyforth/arch/esp8266/. Here we have 'serious' assembler code for the low level primitives. Most of them are just snippets of typical 2 ... 5 lines. I don't tink there is any need to change. Just a nice view how simple it is to craft a language :-).

Here it's going to be C-ish punyforth/punyforth/arch/esp8266/rtos/user .... roughly just forth_{evt,flash,gpio,i2c,io,math,netconn,spi,systime,wifi,ws2612}.{c,h} Most of them are straightforward simple 2-liner wrappings around well known or obvious library code. This is the 'lower' side of the interface, where forth calls into the metal. So if any hardware glue we might need is not there, we +- easily might change or extend that.

This looks like the "upper side" of the interface - the central hook to get the whole thing up and running:

    ..../punyforth/punyforth/arch/esp8266/rtos/user$ cat user_main.c 
#include "FreeRTOS.h"
#include "espressif/esp_common.h"
#include "espressif/esp_softap.h"
#include "task.h"
#include "esp/uart.h"
#include "espressif/esp8266/esp8266.h"
#include "punyforth.h"
#include "forth_evt.h"
#include "forth_io.h"

static void forth_init(void* dummy) {
    forth_start();   
}

void user_init(void) {
    uart_set_baud(0, 115200);
    printf("\nLoading Punyforth\n");
    forth_load(0x52000 / 4096);
    init_event_queue();
    xTaskCreate(forth_init, "punyforth", 640, NULL, 2, NULL); 
}

It is the name of the directory RTOS, that bothers me, and the name of the last function call xTaskCreate. I dont't find this symbol anywhere else on my ESP8266 related build tree. Neither in punyforth, neither in espressif-sdk, neither in nodemcu nor in the PlatformIO-Arduino-tasmota-tree. Google reveals https://www.freertos.org/a00125.html

So the next questions coming up:

What we want in the end, is at least one interactive interpreter task for interactive on gadget development that we can hook to some tasmota text stream (serial, commands, mqtt, syslog, tcp, udp, telnet ....) and 0..n compiled runtime tasks we can hook in as sensor or generic modules. And at best, a way to generate one of the latmentiones ones on the fly by the firstmenioned interactive task.

wolfgangr commented 5 years ago

OK, as expected, the answer for the first question is here https://github.com/zeroflag/punyforth/wiki/Build-environment-setup

Clone esp-open-rtos then copy punyforth to its example directory.

https://github.com/SuperHouse/esp-open-rtos

I'm inclined to like this senctence, but I'm not sure, whether this is a good idea:

Originally based on, but substantially different from, the Espressif IOT RTOS SDK.

In the end, it relies on https://github.com/pfalcon/esp-open-sdk/ I hope - but still have to check - that at least this is the same arduino-ESP8266 builds upon.

wolfgangr commented 5 years ago

.... deeply woven in ??

       .../punyforth/punyforth/arch/esp8266/rtos/user$ grep -n "FreeRTOS.h" *
forth_evt.c:2:#include "FreeRTOS.h"
forth_evt.h:4:#include "FreeRTOS.h"
forth_i2c.c:1:#include "FreeRTOS.h"
forth_io.c:1:#include "FreeRTOS.h"
forth_math.c:1:#include "FreeRTOS.h"
forth_netconn.c:3:#include "FreeRTOS.h"
forth_spi.c:1:#include "FreeRTOS.h"
forth_sys.c:1:#include "FreeRTOS.h"
forth_time.c:1:#include "FreeRTOS.h"
forth_wifi.c:4:#include "FreeRTOS.h"
forth_ws2812.c:1:#include "FreeRTOS.h"
user_main.c:1:#include "FreeRTOS.h"
wolfgangr commented 5 years ago

OK, to get a hands on it, lets start from the user interface of task management https://github.com/zeroflag/punyforth#tasks-experimental extracted those buzzwords: 'task' 'activate' 'deactivate' 'pause' ' multi' 'mailbox' 'mailbox-receive' 'mailbox-send' 

..../punyforth/punyforth/arch/esp8266/rtos/user$ grep -ni "pause" *

maybe pause calls upon some lower level primitive?

...//punyforth/punyforth/arch/esp8266/rtos/user$ grep -ni "task" *
forth_evt.c:3:#include "task.h"
forth_evt.h:5:#include "task.h"
forth_gpio.c:31:        .event_time_ms = xTaskGetTickCountFromISR() * portTICK_PERIOD_MS,
forth_io.c:5:#include "task.h"
forth_io.c:92:   taskYIELD();
forth_spi.c:3:#include "task.h"
forth_sys.c:3:#include "task.h"
forth_sys.c:11:    taskYIELD();
forth_sys.c:15:    taskENTER_CRITICAL();
forth_sys.c:20:    taskEXIT_CRITICAL();
forth_time.c:2:#include "task.h"
forth_time.c:6:    vTaskDelay(millis / portTICK_PERIOD_MS);
forth_time.c:10:    return xTaskGetTickCount() * portTICK_PERIOD_MS;
forth_ws2812.c:3:#include "task.h"
user_main.c:4:#include "task.h"
user_main.c:20:    xTaskCreate(forth_init, "punyforth", 640, NULL, 2, NULL);

OK, we have a yield() in arduino, as far as I can remember. And the xTaskGetTickCount() presumably does the same as arduionos millis(), I hope.

wolfgangr commented 5 years ago

all other buzzwords have no relevant matches here:

...$ grep -ni "activate" *
...$ grep -ni "deactivate" *
...$ grep -ni "pause" *
...$ grep -ni "multi" *
forth_io.c:12:#define BUFFER_SIZE 4096 // should be multiple of 4
...$ grep -ni "mailbox" *
wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_time.c

these two can be implemented by arduino millis()

void forth_delay_ms(int millis) {
    vTaskDelay(millis / portTICK_PERIOD_MS);
}

int forth_time_ms() {
    return xTaskGetTickCount() * portTICK_PERIOD_MS;
}

looks like microsecond delay is not implemented collaborative - so we can keep it

void forth_delay_us(unsigned int microseconds) {
    sdk_os_delay_us(microseconds & 0x0ffff);
}

let's create a tick list: rtos/user$ ls -1 forth_*.c

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_sys.c provides just bare wrappers around

forth_sys.c:11: taskYIELD(); forth_sys.c:15: taskENTER_CRITICAL(); forth_sys.c:20: taskEXIT_CRITICAL();

https://freertos.org/a00020.html#taskYIELD https://freertos.org/taskENTER_CRITICAL_taskEXIT_CRITICAL.html

The taskENTER_CRITICAL() and taskEXIT_CRITICAL() macros provide a basic critical section implementation that works by simply disabling interrupts,

OK, this is not the whole story, but might suffice for a start

[ * ] forth_sys.c implemented by call to arduino yield() and enable / disable interrupts

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_spi.c

Do those functions block?

forth_spi.c:11:    return spi_init(bus, (spi_mode_t)mode, (uint32_t)freq_div, forth_bool(msb), (spi_endianness_t)endian, forth_bool(minimal_pins));
forth_spi.c:15:    return spi_transfer_8(bus, data & 0xFF);
forth_spi.c:19:    return spi_transfer(bus, out_data, in_data, size, (spi_word_size_t) word_size);

I don't think so... but should be double checked, of course [ * ] forth_spi.c no blocking functions

wolfgangr commented 5 years ago

This one https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_io.c will require some attention.

let's look for blocking functions first: Of couse, we may wait for a character:

int forth_getchar_nowait() {
   if (loading) return next_char_from_flash();
   taskYIELD();
   char buf[1];
   return sdk_uart_rx_one_char(buf) != 0 ? check_enter() : buf[0];
}

We have a yield that we may take from Arduino - fine.

So for our multitasking check list: [*] forth_io.c

But there is more in it:

void err(char *msg) {
    printf(msg);
    sdk_system_restart();
}

This was the weir'd behaviour of a system restart upon pull from an empty stack. So this is not a bug, it's a feature. We might change this, but the README says that there is some exception hadling scheme implemented. So it will be preferrable to catch errors there and leave this final rescue in place. RTFS working at its best ....

Next stanza grabbing my attention:

void forth_putchar(char c) { printf("%c", c); fflush(stdout); }
void forth_type(char* text) { printf("%s", text); fflush(stdout); }
void forth_uart_set_baud(int uart_num, int bps) { uart_set_baud(uart_num, bps); }

This is fine for the first test - leave forth connected to the serial line. On a deployed tasmota gadget, we will have this redirected. But I think this does not necessarily mean rewriting this low level interface, but providein additional ones and manage the switching from within forth .... or so... we'll see.... so I hope... ####TODO

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_gpio.c

configuration, read and write from GPIO is not blocking. Maybe we should cross check port number schemes, though.

pwm_set_duty and pwm_set_freq are not blocking either, I think nor nasty in any other way.

This macro looks blocking:

#define WAIT_FOR_PIN_STATE(state) \
    while (gpio_read(pin) != (state)) { \
        if (xthal_get_ccount() - start_cycle_count > timeout_cycles) { \
            return 0; \
        } \
    }

xthal_get_ccount seems to be part of xtensa libs, and calls can be found in arduion as well. So I'd say this code is not corerctly implemented in an multitasking environment yet. On the next lines it says // max timeout is 26 seconds at 80Mhz clock or 13 at 160Mhz That's not cooperative in multitasking terms, I'd say... Well, the README says "experimental", right?

I think the correct way would be, to limit this wait to some ms and implement an nonbreaking alternative with yield() calls during the wait.

This one will require real hard work, I'm afraid - or at least much more thought: Interrupt handlers, written in forth, attached to 

void IRAM gpio_intr_handler(uint8_t gpio_num) {
    struct forth_event event = {
    .event_type = EVT_GPIO,
    .event_time_ms = xTaskGetTickCountFromISR() * portTICK_PERIOD_MS,
    .event_time_us = sdk_system_get_time(),
    .event_payload = (int) gpio_num,
    };
    forth_add_event_isr(&event);
}

But sure, irq-handlers in forth is something we really will want.

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_i2c.c wrappers around low level calls:

$ grep -ni " return i2c_" *
forth_i2c.c:5:    return i2c_init(bus, scl_pin, sda_pin, (i2c_freq_t)freq);
forth_i2c.c:9:    return i2c_write(bus, byte);
forth_i2c.c:13:    return i2c_read(bus, (bool)ack);
forth_i2c.c:17:    return i2c_start(bus);
forth_i2c.c:21:    return i2c_stop(bus);
forth_i2c.c:25:    return i2c_slave_write(bus, slave_addr, data, buf, len);
forth_i2c.c:29:    return i2c_slave_read(bus, slave_addr, data, buf, len);

don't look like arduino calls, but presumably perform similiar stuff. MAY all be blocking - we should be sure....

Where can I find refernce doc for that? RTFS, of course... https://github.com/SuperHouse/esp-open-rtos/blob/a8c60e096093e9e9f4a60b885676adc2cf5b790a/extras/i2c/i2c.c https://github.com/SuperHouse/esp-open-rtos/blob/a8c60e096093e9e9f4a60b885676adc2cf5b790a/extras/i2c/i2c.h

2c.c, Line 362: I see delays...

line 404: static int i2c_bus_test(uint8_t bus) looks like this tests the bus and yield()s if it has to wait. I think it is only called for slave side communication. Well, makes some sort of sense: slave has to wait - blocking implementation master will be served immediately - non blocking implementation (tiny µſ waits are neglected) Anyway, this looks like bit banging stuff. Do we need this? Or do we have hardware aka buffered i2c in arduino-ESP?

Summary on i2c:

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_flash.c

#include "espressif/esp_common.h"
#include "espressif/spi_flash.h"

Obviously only direct calls into sdk, no rtos stuff - fine :-) [*] forth_flash.c

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_evt.c

forth_evt.c:10:    event_queue = xQueueCreate(12, sizeof(struct forth_event));
forth_evt.c:14:    xQueueSendToBackFromISR(event_queue, event, 0); // queued by copy, not reference
forth_evt.c:18:    return (xQueueReceive(event_queue, event, timeout_ms / portTICK_PERIOD_MS) == pdTRUE) ? 1 : 0;

hm..... Events are a central elements of multitasking frameworks, I'm afraid. This will require real LOTS of thought, I'm afraid....

[ ###TODO### ] forth_evt.c

well, or may be we do not really need them, at least not in the first instance? Basic interprocess communication can be implemented by other tools like shared variables or fifos aka ring buffers. But aren't queues just ring buffers?

RtfM: Queue Management xQueueCreate xQueueReceive xQueueSendToBackFromISR

Obvioulsy this mechanism is used to desynchronize (fast) interrupt handlers from (slow) processors. I think we may postpone that. If really required at some time, it will not be impossible, albeit cumbersome, to implement.

It looks like just a fifo ring buffer of 12x4 integers (i. e. 768 byte of maybe precious ram) of type

struct forth_event {
    int event_type;
    unsigned int event_time_ms;
    unsigned int event_time_us;
    int event_payload;
};
wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_wifi.c In the essence:

$ grep -ni "forth_wifi_" *
forth_wifi.c:13:int forth_wifi_set_opmode(int mode) {
forth_wifi.c:17:int forth_wifi_station_connect() {
forth_wifi.c:21:int forth_wifi_station_disconnect() {
forth_wifi.c:25:int forth_wifi_set_station_config(char* ssid, char* pass) {
forth_wifi.c:34:int forth_wifi_set_softap_config(char* ssid, char* pass, AUTH_MODE auth_mode, int hidden, int channel, int max_connections) {
forth_wifi.c:49:void forth_wifi_set_ip(int ipv4) {
forth_wifi.c:57:void forth_wifi_get_ip_str(int interface, char * buffer, int size) {
forth_wifi.c:74:void forth_wifi_stop() {

These are +- thin wrappers around sdk-calls (not RTOS). As far as I can remember from my work on nodemcu, they should be nonblocking. But I'm not sure.

wolfgangr commented 5 years ago

https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/forth_netconn.c

#include "lwip/api.h"
#include "lwip/ip_addr.h"

Ah, this is the interface to lwip - fine. Just what I am looking for to implement such things as wired networking. Would be great, to have a interactive higl-level language at hand for debugging :-) So, yes, maybe not everybody needs it, but I will.

As far as I can remember, all calls are implemented nonblocking. lwip hat its own protothread implementation anyway. Postpone details and clarification for the ethernet job #1

wolfgangr commented 5 years ago

Intermediate summary: Up till here, I haven't yet found anything in the C-layer between punyforth and the system that would look like absolutely prohibiting the integration into the tamota-arduino-framework. But I haven't yet figured out the way control is switched to and from forth and between tasks and how forth then is supposed to regain processor time e.g. in some kind of main loop call.

I just see that there is a Forth file https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/forth/tasks.forth which contains the definition of task:, pause, activate, deactivate and multi, associated with task management. Next candidate for scrutiny.

wolfgangr commented 5 years ago

May just have hit a no-go barrier for punyforth-into-tasmota: lack of memory 

cr osfreemem . cr
18144
cr freemem . cr
17100

If those figures refer to avaialable dictionary / total ram space (out of 128k total) we may run into a barrier. Sitll have to figure out the mem scheme of punyforth. Can we estimate the RAM footprint of tasmota (above esp-core and arduino) ?

wolfgangr commented 5 years ago

NEVER GIVE UP :-[|]

https://github.com/zeroflag/punyforth/wiki/Developing-and-deploying-Punyforth-applications#how-does-everything-work punyforth does not appear to be tuned for efficient RAM usage. All expansion modules are compiled into RAM. Just decompile:-ed the silly LED blinker-app and found that one single silly simple loop directive is compiled into 12 cells, 32 bit each, thus eating 48 bytes of precious RAM.

I need to understand the task switcher anyway - maybe it is even portable. This would reopen the contest for other candidates - at foremost cforth and cxxforth, in the hope that they are somewhat more careful with RAM.

So let's have a look: https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/forth/tasks.forth

The whole task handling is modeled on top of this 8-word = 32 byte data structure:

struct
    cell field: .next
    cell field: .status
    cell field: .sp
    cell field: .rp
    cell field: .ip
    cell field: .s0
    cell field: .r0
    cell field: .handler
constant: Task

A next-pointer for crafting a (cyclic) linked list, a status flag field, 2x2 pointers for top and bottom of ordinary and return stack, interpretion pointer and an exception handler.

Looking at the definition of the task: instantiator below, we see that for everey task we need one such struct and an own stack, plus some optional user-space size. The stack size is configurable and determinded by the value of those variables at task creation time:

112 init-variable: task-stack-size
112 init-variable: task-rstack-size

With these defaults, we end up at a RAM footage of 112+112+8 = 256 cells = 1kByte per task. Quite a lot, but I think for simple I/O handler, these figures may be tuned town considerably.

The running task - supposedly the interactive REPL - is instantiating itself as task 0. pause is a placeholder that is either linked to pause-multi when switched to multi-tasking oepration, or to nop in single-tasking mode.

In addition to that, there is an assignment of the pause execution pointer to xpause This symbol can be grepped here: https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/ext.S just below the readchar definition.

I think it makes sense to assume that a task switch is called when forth is waiting for the next char on the serial line, but I haven't yet figured out the syntax of those .S files in detail.

wolfgangr commented 5 years ago

So: the task manager of punyforth looks manageable. It might be ported to other forth implementations. It may need some rework, though, since punyforth considerably deviates from standard forth in the syntax details of higher layers. The struct datatype might need (re)implementation.

The punyforth task handler is forth only. The integration of task scheduling with non-forth applications has to be crafted, anyway. Possible hooks are either the pause-multi handler, which is always called by a forth task when it likes to yield.

Another idea were to instantiate an own task for the OS, thus inserting it in the round robin scheme of the forth task handler.

I could envision additional tasking modes beyond mere single and multi:

At the moment, only 1 out of 32 bits of the status cell are used. Maybe we can assign some other bit to a runlevel, so for each task we can select in which runlevel it is active. The runlevel could then become a parameter supplied with the startup call of forth.

... things gaining shape in mind :-) ....

wolfgangr commented 5 years ago

How can we hook into the other side? Let's have a look at tasmotas task handler: https://github.com/arendst/Sonoff-Tasmota/blob/development/sonoff/sonoff.ino

It boils down to this stanza, beginning at line 2542: a (not really so) simple arduino loop().

void loop(void)
{
  uint32_t my_sleep = millis();

  XdrvCall(FUNC_LOOP);
  XsnsCall(FUNC_LOOP);

  OsWatchLoop();

  ButtonLoop();
  SwitchLoop();
  RotaryLoop();

  if (TimeReached(state_50msecond)) {
    SetNextTimeInterval(state_50msecond, 50);
    XdrvCall(FUNC_EVERY_50_MSECOND);
    XsnsCall(FUNC_EVERY_50_MSECOND);
//....

First approach: We keep arduino in top control. We let forth return after it cycles through its tasks. Then we might need two different call into forth: A setup and a loop call. OK - when forth keeps track of its own setup state, then it may well do this itelf when it is called the first time in a loop. Any way, the loop call has to yield when it encounters either a pause or a complete loop cycle:

Second Approach: Let Forth be the top level task handler My initial idea was to call forth at the end of the arduino setup(). It will stay there forever. So we have to call arduino loop() from within forth. Simply as a thin layered wrapper around loop() such as any other C-wrapper encountered yet. We even might inject more than one such 'outcall' in the task list, thus giving the arduino loop a higher share of cpu runtime.

However, as I recall, hidden in arduino, there are other loop tasks to do, invisible in sonoff.ino loop(). The same might be true after setup() ends, before it starts loop()ing We might dig into arduino, find out and call that our selves. But if that is not doucmented behaviour, it may be subject to change, and thus break maintainability.

Third idea: We combine both options: We define skd of resume task like any other task, that does only a return to arduino and manages the regain of control, when forth is called again. This looks like forth were in control from inside forth, but in reality, arduino is.

Hard questions: Can we rely on the anything beeing intact that is required when forth returns like any small subroutine? Or do we need to store C-compiled system state, before we call forth? What? stack? pointers? CPU registers? Maybe this questions are easier in a C-only implementation of forth?

There is an additional question how we handle abort and exceptions in forth. I don't like the idea of a complete module restart as the one and only option. I'd like to see the possibility of a forth restart only, the other tasmota stuff going on.

wolfgangr commented 5 years ago

Question still lurking: Priorities.

In sonota, they are implemented by the 50 / 100 / 250 msecond stanzas: They driver may decide on priority and defer low prio tasks to the timed calls, only time critical tasks to the main loop. Recalling my own tests I found the loop running once every 100 ms, so this concept does not really work. Maybe it is because I have enabled syslog, webserver and TLS mqtt all together. And still a lot of modules not activated, but called in every loop and wanting to be asked, at least. Good idea in the beginning, but outgrown by the increased complexity of exploding tasmota.

So, even if we colud implement a priority scheduler in forth, it will not tackle the real issue. To gain real CPU time savings from task prioritisation, we needed to call any single module handler from the forth task schedule, not the whole arduino loop. I think this may be a perspective in the long run, but not for the first implementation, There is the option of having more and more drivers to be pulled from C-land to Forth-land, anyway, maybe rendering this idea irrelevant, in the end, anyway.

Next consideration: More or less standalone forth snippets as driver implementation, inserted into the different time/priority hooks of tasmota. Actually, this was on my vision quite early. Looks nice, lots of work and complicated interface, but does not help much do to the same reason: If the loop is called not more than once everey 100 ms anyway, we can as well leave all in one priority layer.

wolfgangr commented 5 years ago

TaTAAA !! :-) Glad to welcome Attila Magyar @zeroflag , Master of punyforth https://github.com/zeroflag/punyforth/issues/43#issuecomment-455905709 in the dark realms of my weird thoughts.

wolfgangr commented 5 years ago

OH MY GOD! It was an up and down 'til now, anyway, but this smashes the record: https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot/posts/1501224

Essence: The ESP8266 where we spend our precious life time on, is a real pile of scrap. No serious documentation, just a collection of reengineered hacks. No idea how the memory management works, if not figured out by trial an error .. CRY... I just want to compile my forth dictionary from precios RAM to ample Flash. Got 4 MB on my ESP-12, anyway. I mean, what good is it for, if I can't use it??? GIVING UP IS NOT AN OPTION

wolfgangr commented 5 years ago

OK. take care of memory. That's what embedded programming is about, right?

Reconsidering choice of forth implementations:

=> pendulum swings towards cforth again

wolfgangr commented 5 years ago

Memory footprint of cforth: tembed.o has 351168 byte flash requirement so, could be linked to tasmota, if we sacrifice OTA. Or other stuff.

350 k flash footprint break down:

RAM footpring:

So - may be, it could work....

wolfgangr commented 5 years ago

Reappraisal of cforth regarding cooperative multitasking

starting here: https://github.com/MitchBradley/cforth/blob/esp32-v1.0/src/cforth/embed/BuildProcess.txt I tracked the rabbit hole.

j) Link, using the target (cross) linker, the object files compiled in
   steps c and i.  The result is an object file "embed.o" that can
   subsequently be linked into a target main application.  "embed.o"
   has only a very small number of external dependencies that must
   be provided by the main application, namely "getchar()" and "putchar()".
   embed.o in turn provides two entry points that the main app can call:

      init_forth()      - must be called once to set things up

      execute_word(str) - call this to execute the Forth word "str"
For the interactive interpreter, use execute_word("quit")

init_forth is non-blocking and should be fine in arduinos init(). execute_word(str) calls - some layers higher - the inner interpreter (line 73) here: https://github.com/MitchBradley/cforth/blob/esp32-v1.0/src/cforth/forth.c

I think I see 2+ special cases, when it leaves:

So, we do not even need a multitasking framework within forth, I think. We can hookindependent snippets of code to the tasmota framework events and thus utilizing its existent multitasking behaviour.

wolfgangr commented 5 years ago

Well, not true - yet. The innerinterpreter, as far as it now, is not reentrant. So, when it leaves in the main loop, waiting for a key pressed, and is called as interrupt handler then (or even while it runs), then it starts at the last IP aka program location where it left off.

Sketch of a solution:

struct
    cell field: .next
    cell field: .status
    cell field: .sp
    cell field: .rp
    cell field: .ip
    cell field: .s0
    cell field: .r0
    cell field: .handler
constant: Task

in cforth, https://github.com/MitchBradley/cforth/blob/esp32-v1.0/src/cforth/forth.c we have this restore after resume snippet line 81: rp = (token_t **)V(XRP); ip = *rp++; sp = (cell *)V(XSP); tos = *sp++; the save to suspend snippet: line 588: *--sp = tos; V(XSP) = (cell)sp; *--rp = ip; V(XRP) = (cell)rp;

looks quite similiar, right? If we have tasks instantiated in the punyforth way , we can provide a pointer to the task struct where then inner interpreter is restoring / storing it's state instead.

Detail still to be craftet, of course.

wolfgangr commented 5 years ago

Thinking of interrupt handlers implemented in forth, we think about performance, among others.

The time critical Inner loop is the execution loop of the inner interpreter with its long-long-long select-case-cascade. If this were compiled into a long cascade of compare-branches, this would be a nightmare. Preferrably this would be imlemented by a lookup table, or at least by some branching tree.

Will the C-optimizer do this favour to us? Need to have a look, at least.

wolfgangr commented 5 years ago

With any problem solved, 3 new ones pop up. I'm afraid, this might end up as a huge endeavour, consuming months of time, and getting stuck precisely where nodemcu stuck: It is hard to implement real time highl level language apps that integrate well into the ESP framework.

In another long night of futile C-makefile-boredom, I tried a mecrsp-stellaris-FORTH on a STM32-bluepill, interfaced on a ESP-Link running on a Wemos D1. So basically, the ESP is like a wifi-not-USB-serial cable.

That feels like hands on development. So maybe I drop the Idea of running forth on the ESP and focus on the load off of not-standard-tasks to a secondary processor?

Well, there are problems, too. Let's see....