punyforth integration into tasmota?

[wolfgangr]

Dear Attila,

I am looking for a scripting language for expanding the configurability of the tasmota framework https://github.com/arendst/Sonoff-Tasmota As punyforth, tasmota runs on ESP8266 platform and started on sonoff devices. It implemets a web surface, MQTT client, serial and http command, remote syslog feature...

Forth is a natural choice, and punyforth runs on the target platform - great :-)

However, tasmota is not based on RTOS, but on arduino-esp8266 and implements a simple arduino type architcture

init() { 
     do some stuff; 
}

loop() {
    repeat stuff all the time;
}

Do you think it is possible to get punyforth built and running? A first glance on https://github.com/zeroflag/punyforth/blob/master/arch/esp8266/rtos/user/user_main.c made me optimistic.

This reads like it can be called from arduino framework as well. It's just the xTaskCreate that worries me a little bit. Does this belong to RTOS? How deeply are punyforth and RTOS woven together ?

I think I willl need

• hooks to create tasks
• hooks to initiate the whole thing
• hooks for the arduino yield() which resembles the scheduler
• or may be it suffices just to call it once in every loop turn?
• hooks to redirect tasmota command processing stream to forth
• information on the memory and flash management to keep it coexistent
• some hints on the details of the multitasking implementatin
• maybe some extra hardware function wrappers

I'll try to understand the code myself, but of course, any pointer to assist me in my endeavour will be appreciated :-)

[zeroflag]

hi @wolfgangr,

The core of punyforth is independent of RTOS, but there are lots of esp specific functionalities (like WiFi controlling, GPIO, i2c, spi, netcon) which are coming from RTOS.

Most of these are defined in ext.S. The CCALL macro is used to call C functions from Punyforth.

Here is an example:

defprimitive "load",4,load,REGULAR
    DPOP a2         // block number
    CCALL forth_load
    NEXT

This will call the forth_load() C function defined in forth_io.c which will call some RTOS code.

If you want to have all these functionality (GPIO, netcon, etc) you would need to change these C functions to call the arduino equivalents of the RTOS code.

There are also some words in ext.S which are only for optimization these don't need to be changed.

It's just the xTaskCreate that worries me a little bit. Does this belong to RTOS?

Yes. The Punyforth outer interpreter is executed as an RTOS task. Regarding the init() and loop(), I'm not sure what's the right way to use them. Is code in loop() supposed to return? What happens if you put an infinite loop inside the loop? If this is not allowed the outer interpreter should be modified to evaluate one word at a time I guess, because currently that's an infinite loop.

If the init() and loop() can be avoided somehow by using the lower level arduino machinery then that would be an other option.

[wolfgangr]

@zeroflag Attila, thank you for response. Glad to see that I'm not walking alone :-)

I work on a fork of tasmota and keep documenting a journal of my considerations at the issue tracker there: https://github.com/VerboteneZone/WESPOTA/issues/15

I've selected three candidates for my implementation: punyfort, cforth and cxxforth

[wolfgangr]

punyforth is my current favorite, above all because of its existing multitasking implementation. I hope this will help me to weave it ito the task management of arduino & tasmota.

However, at he moment, https://github.com/VerboteneZone/WESPOTA/issues/15#issuecomment-455998079 I see a large problem since punyforth eats up most of RAM.

• How much of that has to be attributed to punyforth itself, how much to the RTOS and ESP core?
• Can we shift more of runtime FORTH to FLASH?
• Can we set the compiler to store the dictionary in FLASH, instead of RAM?
• Can we move/recompile dictionaries which are already compiled to FLASH?

[wolfgangr]

If you want to have all these functionality (GPIO, netcon, etc) you would need to change these C functions to call the arduino equivalents of the RTOS code.

Yes, I see. Do I find them all in arch/esp8266/rtos/user ?

I've started dissection and a preliminary appraisal of neccesary work, about here: https://github.com/VerboteneZone/WESPOTA/issues/15#issuecomment-455885325

I think there is some work to do, but I couldn't identify any major obstacle along the way, yet.

[wolfgangr]

Is code in loop() supposed to return?

Yes, of course. That's what cooperative multitasking is all about.

What happens if you put an infinite loop inside the loop?

Blocking code in loop() freezes tasmota. The loop() resembles the task handler of any arduino code.

If this is not allowed the outer interpreter should be modified to evaluate one word at a time I guess, because currently that's an infinite loop.

Yes, that might be one approach. And maybe a good one for the start and later on as a fallback for debug purposes.

So, the next question were, how can I hook into the outer interpreter?

[wolfgangr]

However, my considerations go beyond that:

What about the inner interpreter? The goal at the end is to implement drivers in forth and hook them into tasmota.

I think your task manager task.forth is a good point to start with. So I will have different forth tasks to be called in sequence in one single arduino loop() trip. The suspend/resume to be implemented then had to be linked some way into the pause mechanism, I thought.

I could not yet figure out the role of the xpause symbol. grep finds it both in task.forth and in ext.S In the last, it is written just below the readchar definition. I think it makes sense to assume (and resembles your proposal) that a task switch is called when forth is waiting for the next char on the serial line, but I haven't yet figured out the syntax of those .S files in detail.

How and when are those .S files processed?

• Is it generic assembler code? If so, could you point me to some reference guide?
• or is it one of the pyhton scripts running during build?
• or even by Forth itself?
• or whatever?

[wolfgangr]

So, the hook question from above generalizes to: How can I suspend from and resume to FORTH without screwing the C environment? I think the compiled C-code assumes some CPU-registers, C return and parameter stack und maybe some other "stuff like that" intact.

My big question for the moment (beyond the RAM issue): Can I implement a suspend from forth as a simple C-return And as resume a call to sth like forth_load such that forth resumes just where it suspended before? Ideally, forth should manage its state between such suspend - resume. (So that forth beaves as a state machine, in the context of a multitasking framework vocabulary)

If done from within a pause, should it not be sufficient just to save the task pointer, referring to his Taskstruct of the next task to run? I think, saving state of a task is what this Taskpointer is all about, right?

So the cycle goes:

• ardunino_setup() -> .... forth_load
• forth realizes it's virgin state and performs its setup work, among other, it initiates a task framework with just the REPL in and returns by a call to pause
• pause saves the next task to load and forth_load returns control to C if necessary, any C state (stack, CPU register, ... what else?) is restored before
• arduino_setup() continues after forth returns from forth_load ===== arduino starts looping =====
• arduino somewhere in its loop() calls forth_load again.
• forth now realizes that it has already been initialized -- it restores its previous state by loading the Task struct -- performs one task, or one scheduler round, or whatever is configured from priority considerations -- up to some pause condition that decides to suspend again
• forth returns control to C again the same way as described for the init
• arduino recycles its loop(),
• going on infinitely, thus switching between forth and tasmota, until some nasty blocking code triggers the watchdog

Alternatively, the suspend / resume sequence might be coded into a separate FORTH task within the task.forth framework. There might be even more than one copy of it in the task list, allowing finetuned scheduling of the task switching to tasmota&arduino core.

Can you follow me? Can you help me with my questions?

• RAM optimisation
• C-ish subroutine return from forth pause
• Forth task resume with a load_forth call

[zeroflag]

• Can we shift more of runtime FORTH to FLASH?
• Can we set the compiler to store the dictionary in FLASH, instead of RAM?
• Can we move/recompile dictionaries which are already compiled to FLASH?

Currently no. User defined words are stored in RAM. As far as I remember @holinepu did some work on compiling words to FLASH.

Do I find them all in arch/esp8266/rtos/user ?

Yes.

So, the next question were, how can I hook into the outer interpreter?

The outer interpreter is defined in: outerinterpreter.S. These .S files are assembly files and they contain either native asm code or compiled threaded forth code in a textual format (using labels and symbols instead of binary). These files contain the minimal set of words that are required to parse and compile textual forth code. These were written by hand.

For example the below code is equivalent to the "word 2dup" forth code.

    .int xt_word                           // ( a len )
    .int xt_dup2

The outer interpreter is in a loop, there are 3 places where it jumps back to its beginning.

    .int xt_branch
    lbl outer_interpreter 

The outer interpreter is invoked from punyforth.S which is invoked by the xTaskCreate in user_main.c

So if you remove the loop from the outer interpreter and call that from the arduino loop() it might work. But you also need to save the registers which are used by the outer and inner interpreter.

There might be an other problem, namely that readchar (which gets the next character of the source code or the next key entered by the user) blocks by default if there is no input available. But it has an other version which is used in multi tasking mode that doesn't block.

[holinepu]

Can we shift more of runtime FORTH to FLASH? Can we set the compiler to store the dictionary in FLASH, instead of RAM? Can we move/recompile dictionaries which are already compiled to FLASH? Currently no. User defined words are stored in RAM. As far as I remember @holinepu did some work on compiling words to FLASH. yes, I've done some work on punyForth 2 months ago. I can compile the program in RAM and then move it to flash momery. Gradually the program can be bigger and bigger. I've sent the program to Mattias and Peter Forth but no response. So, now I switch to RISC-v RV32I ，RV32M . I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth. The same above functions have been finished in my punyForth development system. I'd like to share it with Forthers it they'd like.

[holinepu]

Of course the newer added program can be compiled under cross-compiler, that way we don't have to worry about the problem of compiling in the limited RAM. Wouldn't that be great!?

[wolfgangr]

Hi @holinepu great to have a response from you.

Some findings from other forth'es:

Cforth uses a elaborated host based forth compilation machine https://github.com/MitchBradley/cforth/blob/esp32-v1.0/src/cforth/embed/BuildProcess.txt Thus it has a bunch of libraries already compiled in, which reside in flash. I thnik it should be possible, one an application i developped to maturity level, to drop this code into the build directory and flash it to the ES8266 this way, as well. Sound not really forth'ish, but will do the job.

mecrisp (on STM32 blue pill) can interactively switch between compiletoram and compiletoflash. It supports a cornerstone word to erase flash down to some marker set before. So you once flash a mecrisp core and the rest is interactively - forth'ish, I'd say. Unfortunately, mecrisp is written in assembler and not yet ported to the ESP8266 / Tensilla core.

[wolfgangr]

yes, I've done some work on punyForth 2 months ago. I can compile the program in RAM and then move it to flash momery. Gradually the program can be bigger and bigger.

Great - looks close to what I have seen in mecrisp. I assume you know mecrisp and the way it works, there? Well, you cannot move code over, you keep a source file on your workstation. Modify and compile to ram, until it settles. Switch over to flash, compile again and it is persistent.

switch to RISC-v RV32I ，RV32M

does this relate to the ESPXXX family? Does not ring a bell in my ears. Ask Google - Ah, I see, that's the VHDL path. I'm afraid that's beyond my capabilities.

Well, I think the ESP8266 is a FPGA-based SOC anyway. So yes, having a native forth engine running on it would be great. But I'm afraid the information required to implement that is not available in public.

I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth.

For the ESP-family as well? I'd prefer mecrisp because I found loads of tutorials and examples. Maybe it's not that mature. but I think that is what forth is about.

The same above functions have been finished in my punyForth development system.

I this available in public?

I'd like to share it with Forthers it they'd like.

Sure, I would like it. At least give it a thorough test.

Of course the newer added program can be compiled under cross-compiler, that way we don't have to worry about the problem of compiling in the limited RAM. Wouldn't that be great!?

To be honest, I considered that as a crutch only. It mayb be fine for large scale deployments. But for me, forth is the platform for lot sizes close to 1. This is hands on. Test on RAM - successively move to flash as components mature. Keep source files on a host, so you always can start off again. Just a bare minimum forth in flash to bootstrap. like in this tutorial: https://github.com/jeelabs/embello/tree/master/explore/1608-forth/rnw#installation

Keeping the progression steps manageable, only small dictionaries have to stay in ram. The cornerstone funcionality is essential, so that you can selectively remove some top part of the flash based dictionay as well. But this is just 4 lines of forth code.

If it works on a 64KB flash / 20 kB RAM STM32 BluePill, I think it will work on a > 500 k free flash ( 3.5 MB on a EUR 3,- NodeMCU) 128 k RAM ESP, I hope.

[holinepu]

hi, everyone       The attached files are for punyForth for esp8266. It can extend the program in RAM area to Flash rom.  Please read the punyforth550.png to know how?                                                                                                         holi                       在 2019年1月26日 星期六 上午3:22:50 [GMT+8]， wolfgangrnotifications@github.com 寫道：

yes, I've done some work on punyForth 2 months ago. I can compile the program in RAM and then move it to flash momery. Gradually the program can be bigger and bigger.

Great - looks close to what I have seen in mecrisp. I assume you know mecrisp and the way it works, there? Well, you cannot move code over, you keep a source file on your workstation. Modify and compile to ram, until it settles. Switch over to flash, compile again and it is persistent.

switch to RISC-v RV32I ，RV32M

does this relate to the ESPXXX family? Does not ring a bell in my ears. Ask Google - Ah, I see, that's the VHDL path. I'm afraid that's beyond my capabilities.

Well, I think the ESP8266 is a FPGA-based SOC anyway. So yes, having a native forth engine running on it would be great. But I'm afraid the information required to implement that is not available in public.

I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth.

For the ESP-family as well? I'd prefer mecrisp because I found loads of tutorials and examples. Maybe it's not that mature. but I think that is what forth is about.

The same above functions have been finished in my punyForth development system.

I this available in public?

I'd like to share it with Forthers it they'd like.

Sure, I would like it. At least give it a thorough test.

Of course the newer added program can be compiled under cross-compiler, that way we don't have to worry about the problem of compiling in the limited RAM. Wouldn't that be great!?

To be honest, I considered that as a crutch only. It mayb be fine for large scale deployments. But for me, forth is the platform for lot sizes close to 1. This is hands on. Test on RAM - successively move to flash as components mature. Keep source files on a host, so you always can start off again. Just a bare minimum forth in flash to bootstrap. like in this tutorial: https://github.com/jeelabs/embello/tree/master/explore/1608-forth/rnw#installation

Keeping the progression steps manageable, only small dictionaries have to stay in ram. The cornerstone funcionality is essential, so that you can selectively remove some top part of the flash based dictionay as well. But this is just 4 lines of forth code.

If it works on a 64KB flash / 20 kB RAM STM32 BluePill, I think it will work on a > 500 k free flash ( 3.5 MB on a EUR 3,- NodeMCU) 128 k RAM ESP, I hope.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[holinepu]

在 2019年1月26日 星期六 上午3:22:50 [GMT+8]， wolfgangr<notifications@github.com> 寫道：  

yes, I've done some work on punyForth 2 months ago. I can compile the program in RAM and then move it to flash momery. Gradually the program can be bigger and bigger.

Great - looks close to what I have seen in mecrisp. I assume you know mecrisp and the way it works, there? Well, you cannot move code over, you keep a source file on your workstation. Modify and compile to ram, until it settles. Switch over to flash, compile again and it is persistent.

switch to RISC-v RV32I ，RV32M

does this relate to the ESPXXX family? Does not ring a bell in my ears. Ask Google - Ah, I see, that's the VHDL path. I'm afraid that's beyond my capabilities.

Well, I think the ESP8266 is a FPGA-based SOC anyway. So yes, having a native forth engine running on it would be great. But I'm afraid the information required to implement that is not available in public.

I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth.

For the ESP-family as well? I'd prefer mecrisp because I found loads of tutorials and examples. Maybe it's not that mature. but I think that is what forth is about.

The same above functions have been finished in my punyForth development system.

I this available in public?

I'd like to share it with Forthers it they'd like.

Sure, I would like it. At least give it a thorough test.

Of course the newer added program can be compiled under cross-compiler, that way we don't have to worry about the problem of compiling in the limited RAM. Wouldn't that be great!?

To be honest, I considered that as a crutch only. It mayb be fine for large scale deployments. But for me, forth is the platform for lot sizes close to 1. This is hands on. Test on RAM - successively move to flash as components mature. Keep source files on a host, so you always can start off again. Just a bare minimum forth in flash to bootstrap. like in this tutorial: https://github.com/jeelabs/embello/tree/master/explore/1608-forth/rnw#installation

Keeping the progression steps manageable, only small dictionaries have to stay in ram. The cornerstone funcionality is essential, so that you can selectively remove some top part of the flash based dictionay as well. But this is just 4 lines of forth code.

If it works on a 64KB flash / 20 kB RAM STM32 BluePill, I think it will work on a > 500 k free flash ( 3.5 MB on a EUR 3,- NodeMCU) 128 k RAM ESP, I hope.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[holinepu]

在 2019年1月27日 星期日 下午11:34:54 [GMT+8]， chang luke<holinepu@yahoo.com.tw> 寫道：  

在 2019年1月26日 星期六 上午3:22:50 [GMT+8]， wolfgangr<notifications@github.com> 寫道：  

yes, I've done some work on punyForth 2 months ago. I can compile the program in RAM and then move it to flash momery. Gradually the program can be bigger and bigger.

Great - looks close to what I have seen in mecrisp. I assume you know mecrisp and the way it works, there? Well, you cannot move code over, you keep a source file on your workstation. Modify and compile to ram, until it settles. Switch over to flash, compile again and it is persistent.

switch to RISC-v RV32I ，RV32M

does this relate to the ESPXXX family? Does not ring a bell in my ears. Ask Google - Ah, I see, that's the VHDL path. I'm afraid that's beyond my capabilities.

Well, I think the ESP8266 is a FPGA-based SOC anyway. So yes, having a native forth engine running on it would be great. But I'm afraid the information required to implement that is not available in public.

I've completed the assemb|er，disassembler，decompiler and cross-compi|er for mecrispForth and tinyForth.

For the ESP-family as well? I'd prefer mecrisp because I found loads of tutorials and examples. Maybe it's not that mature. but I think that is what forth is about.

The same above functions have been finished in my punyForth development system.

I this available in public?

I'd like to share it with Forthers it they'd like.

Sure, I would like it. At least give it a thorough test.

Of course the newer added program can be compiled under cross-compiler, that way we don't have to worry about the problem of compiling in the limited RAM. Wouldn't that be great!?

To be honest, I considered that as a crutch only. It mayb be fine for large scale deployments. But for me, forth is the platform for lot sizes close to 1. This is hands on. Test on RAM - successively move to flash as components mature. Keep source files on a host, so you always can start off again. Just a bare minimum forth in flash to bootstrap. like in this tutorial: https://github.com/jeelabs/embello/tree/master/explore/1608-forth/rnw#installation

Keeping the progression steps manageable, only small dictionaries have to stay in ram. The cornerstone funcionality is essential, so that you can selectively remove some top part of the flash based dictionay as well. But this is just 4 lines of forth code.

If it works on a 64KB flash / 20 kB RAM STM32 BluePill, I think it will work on a > 500 k free flash ( 3.5 MB on a EUR 3,- NodeMCU) 128 k RAM ESP, I hope.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[holinepu]

hi, Wolfgangr I tried to response your request of the files relating to how to expand the program in RAM to flash rom by email, but it failed to the upper 3 comments above. I've attached the files on above, but fail to see it there. How can I attach the file to you by email. Your emil address?

[holinepu]

在 2019年3月7日 星期四 上午8:48:22 [GMT+8]， Matthias Koch<no-reply@sifive.com> 寫道：  

| | Mecrisp Matthias Koch March 7 |

Dear Holi,

actually, I cannot see anything at the link you provided without being a registered user there. Could you please put the files into a public place or send them to me via E-Mail ?

Matthias

Visit Topic to respond.

In Reply To

| | holi March 6 |

hi, Matthias I’m so glad to receive the reply from you. I can almost simulate the mecrispForth for risc-v 32IM. The cross-compiler under win32for 4.2.67x can well do the assembling and disassembling of RISC-v instruction set. It can run tF ( tinyForth) as well as mecrispForth, while tF being p… Visit Topic to respond.

To unsubscribe from these emails, click here.

[holinepu]

hi, Peter        I'm glad to receive your letter concerning Forth programming especially something connected with PunyForth for esp8266.  As you know that punyForth is confined in the RAM space to extend its program. It seems no good way to relocate the program in the RAM area to the ROM area while within the reach of punyForth itself.  I've tried to solved this problem and it seems that we can do it graciously. But on the other hand we can do it by using cross-compilerand that is what I have been doing in the past 10+ years, and I'd like to covey it to ForthWin system if possible.  The accompanying file ( ForthWin_TF3 ) is the first step toward that.  One thing I'd want to ask you how can I expand the program space to more than 30 Mbyte so that I am able to expand the cross-compiler under ForthWin. That's my request.   Thanks!      在 2019年6月29日 星期六 下午10:51:07 [UTC]， Peter Forthpeter4th2017@gmail.com 寫道：

Hi Chang,  how are you doing ?

What are your projects in Forth now ?

( I was out of Punyforth for long time, I am not using it anymore.)

But I am in other projects of Forth now...

Kind regards from Brazil ! Peter