Open enriquepinedo opened 6 years ago
enriquepinedo
commented
6 years ago This was the picture of the memory map, I can not add it as such on my message.... https://ksr-ugc.imgix.net/assets/005/420/685/7942660fd0259ae699e373b25c5c6fec_original.png?w=639&fit=max&v=1456359366&auto=format&lossless=true&s=37936c9f7731bedc214afb9b94c7d455
enriquepinedo
commented
6 years ago I continue exploring punyforth and I have tests and results ( all numbers below in decimal.)
core.forth alone gives freemem of 16.776 bytes free on an empty system here points to 1073.653.560 ( i can not find any correlation with physical memory in this large number )
tcp-repl with all it dependencies and
a bit of code for an i2c LCD gives me 2.200 bytes freemem
for programming
anything else I try to include after that Punyforth will hang. I can only add
very elementary forth operands. after a "screen" or a "block" of txt (1024 k
text code)_ it will hang.
I have been looking at flash.forth dependencies, it has only 1 needs "core.forth"
Questions : is this editor working ? It seems it saves to Flash memory, like I previously asked if there is a way to "SAVE" a system interactively and incrementally...
I will try to understand more..... apparently having Rtos and open-boot in the system those calls are causing the lack of program memory, ( even when ESP8266 has such a large ram in terms of a forth system).
Kind regards Enrique
zeroflag
commented
6 years ago I have many questions left : 1) is Uber.forth at $51000 copyed from flash to RAM before execution
No, the text is not copied into the RAM. Punyforth compiles indirect thread code from the text and fills up the dictionary.
I have left only 2K
This sounds realistic. The freemem word shows the available forth dictionary space, not the free system memory. There is an osfreemem word for getting the free memory from RTOS. Punyforth allocates about 24k dictionary space which means that your program and its dependencies consume about 22k.
When I was experimenting with larger dictionary spaces in the past, the system became unstable because not enough memory was left for RTOS. It usually crashed inside the lwip (tcp stack) code when it tried to allocate netcon buffers. So I decided to allocate only 24 forth heap, and leave the rest to RTOS.
You can experiment with larger memory settings by changing this line and recompile Punyforth
https://github.com/zeroflag/punyforth/blob/master/generic/data.S#L32
If you don't use the wifi and the tcp stack then it'll probably Ok.
But an easier way to fix this is to remove unused words from the uber.forth.
Questions : is this editor working ? It seems it saves to Flash memory, like I previously asked if there is a way to "SAVE" a system interactively and incrementally...
There is an experimental editor in flash.forth but you need to use the --block-format parameter of modules.py when generating uber.forth.
enriquepinedo
commented
6 years ago Ok excellent !
Line 32 ......... .space 25152
In your opinion if I place .space to 50000 will it work ? 30000, 35000 ? which are the limits ?
Can we expect in future a memory map of punyforth within the ESP8266 ? I would like to help in anything related to documentation to your proyect if you whish. I am preparing for my own use a Spreadsheet with all the words of punyforth, like a glossary, the old systems used to have , with stack notation and if possible or needed for more complex words an example below so anybody can learn faster. As soon I have it ready I will pass this to you. Thanks again for your time !
zeroflag
commented
6 years ago In your opinion if I place .space to 50000 will it work ? 30000, 35000 ? which are the limits ?
As far as I remember 30k should work but it will become unstable if you use the WiFi a lot.
Can we expect in future a memory map of punyforth within the ESP8266 ?
Punyforth is not a bare metal Forth (at least currently) but it runs as an RTOS task. This task is created by calling xTaskCreate, which allocates some memory dynamically. But I don't know how the memory allocation is handled internally.
I am preparing for my own use a Spreadsheet with all the words of punyforth, like a glossary
That's great. I'm sure this will be useful for others too.
enriquepinedo
commented
6 years ago Thanks Attila ! I will try this and let you know.
zeroflag
commented
6 years ago FYI I made some optimalization in variable: and constant: that reduces their memory consumption. This should buy you some additional memory.
enriquepinedo
commented
6 years ago Excelent ! THank you very much !
GerardSontag
commented
6 years ago Hello Attila, If you want to reduce the code size you can consider moving to a version with TOS in register. The a2 register is the best candidate for TOS because it is the first parameter when calling OS function. A lot of these calls when moved to this new version will end with DPOP a2 NEXT sequence which can be replace by a: B PopNext and somewhere PopNext: DPOP a2 NEXT And for a lot of primitives you will save the final DPUSH.
zeroflag
commented
6 years ago Hi Gerard,
I was thinking about TOS to register optimization earlier (mainly because of speed) but decided not to do it because some of the caveats. There are words like sp@ and sp! which would be difficult to use if TOS was in a register. Also one can write a code that treats the stack as an array which would likely break with TOS optimization. Task switching and exception handling would likely be more complicated as well. And I'm not worried about the size of the compiled primitives as much because they're not stored in the same (~24K) dictionary space like the user defined words.
Attila
GerardSontag
commented
6 years ago Hello Attila, sp@ sp! will become:
defprimitive "sp@",3,spat,REGULAR DPUSH a2 mov a2, a15 NEXT
defprimitive "sp!",3,spstore,REGULAR mov a15, a2 DPOP a2 NEXT
Gérard
zeroflag
commented
6 years ago Right. I just don't think the extra complexity would worth the effort. But if you think so feel free to send a PR with the changes required. I'm not against it, I know other Forths do this but mainly because of speed (which is not the main concern) and not memory usage.
GerardSontag
commented
6 years ago Hello Attila, I have done this work some months ago but at that time was unable to assemble the source (some weird things in ubuntu under w10) and know I am unable to merge in Github... I may be give a try again by bypassing github. BTW do you know good documentation about the ESP8266 I got the Xtensa Instruction Set Architecture but i miss some documentation about how 8266 is implementing over LX106 Gérard
zeroflag
commented
6 years ago @GerardSontag no I don't know any unfortunately. But there are some projects in github like the esp port of microphython or forthright which can be used as an example. The API of FreeRTOS is well documented (http://www.freertos.org/a00106.html).
holinepu
commented
5 years ago It seems that the flash rom is so big enough to add more compiled code into it. So if someone can design the metacompiler to append the code to it that would be great, right? I'll try, and you.
holinepu
commented
5 years ago Hi, everybody, I've developed the esp8266 assembler as well as dis-assembler under win32forth 4.2.671 in order to develop cross-compiler for the chip. For now it is almost completed with some difficult problem of dealing with create dose>. The variable header containing in flashrom causes the variable address to be in the same area. I can deal it well with some tricks to put in the ram area. But when it comes to put together with does> I was surrendered. Can anyone do the help? I can compile the program in ram as well as in flashrom area so as we can expand the program to the maximum under my cross-compiler. Or if we don't use the cross-compiler we still can copy the code from ram area to the flashrom area with the same address except the higher 4 digits changed by executing the word 3FFE>4026. Of course, you can change whatever address you like by modifying 4026 to 4027, 4028, 4029, 402A, ....402F. That can expand limited spaces for now, achieving about 240K. That's not too bad. But still you have to solve the problem of variable and does>. : 3FFE>4026 ( -- ) 3FFE88C0 ( lfa'.of.interpret? ) 402688C0 10000 88C0 - cmove' 402688C0 10000 88C0 - BOUNDS DO i 32@' FFFF0000 AND 3FFE0000 = if i 32@' ffff and 40260000 or i 32!' then 4 +LOOP 40219854 ( LFA'.OF.immediate ) 402688C0 ( lfa'.of.interpret? ) 2dup 32!' lfa'->lfa* ! \ ` 4026DAEC var-lastword ! ;
zeroflag
commented
5 years ago Hi holinepu,
Could you elaborate the problem with create does>? I'm trying to recall how it works.
create: always creates a new word with an enterdoes codeword/entrypoint instead of entercol (which is the usual entrypoint for colon definitions).
The enterdoes expects a behaviour pointer (which is an execution token of an other word) after the codeword and some arbitrary data after this. It'll push the location of the data part onto the stack and invoke the behaviour word. The default behaviour word is nop, but does> overwrites it with a user supplied word.
: create: createheader enterdoes , ['] nop cell + , ; \ default behaviour is nop, does> overwrites thisHere you can see that it creates a dictionary header and inserts the enterdoes codeword as an entry point. After that it inserts the xt of nop. The data part is not initialized.
: does> r> lastword link>body ! ;does> replaces the behaviour part with the user supplied word (from the return stack).
So a constant for example can be defined:
: constant: cerate: , does> @ ;But punyforth uses distinct codewords for variable/constants to save some space.
holinepu
commented
5 years ago Dear Attila, Thanks for the reply. I've been trying to compile the program in the rom area. It is succefull to do now. But still we have to solve the problem of variables residing in rom/ram area. The nop instruction is just to provide a ( random ) address in RAM where is a safe place. Why not just give it a fixed RAM address ( 3FFFF800, for example )? I'd like to put create: and does> in flash rom ( or ram ), that way there is no need nop accompaning to the twos.
variable: can be defined in rom area too, so long as it can provide a number which is a ram address.
: variable.base.address hex: 3FFFF800 ; \ 3FFFF800 can be changed according to need. : ram.address.replacing.nop variable.base.address : variable.counter variable.base.address 4 ; \ 0 variable.counter ! <-- how to intialize it? : variable.area variable.base.address 8 + ; : variable.area.wide hex: 400 ; \ 400 can be changed according to need. : var(ram) ( -- )
createheader hex: 40239B2C , variable.area variable.counter @ + ['], ,
4 variable.counter +! hex: 402182C8 ( exit ) , ;: create:(ram) var(ram) -4 variable.counter +! ; : allot(ram) ( -- n ) align variable.counter +! ;
var(ram) v1 var(ram) v2 var(ram) v3
see v1 see v2 see v3
\ ** ( 3FFE9098 3FFE9088 ) \ 3FFE909C 07 63 72 65 61 74 65 2F : create/ \ ------------------------------------------------------------------------------------------ ( 3FFE90A4 40239B2C ) ( docolon ) ( 3FFE90A8 40218E0C ) createheader ( 3FFE90AC 40219660 ) enterdoes ( 3FFE90B0 40218AB4 ) , ( 3FFE90B4 4021824C ) ( dolit ) 3FFE9090 ( 3FFE90BC 40218A9C ) cell ( 3FFE90C0 40218174 ) + ( 3FFE90C4 40218AB4 ) , ( 3FFE90C8 402182C8 ) exit
see create: \ ** ( 40279098 40279088 ) \ 4027909C 07 63 72 65 61 74 65 3A : create: \ ------------------------------------------------------------------------------------------ ( 402790A4 40239B2C ) ( docolon ) ( 402790A8 40218E0C ) createheader ( 402790AC 40219660 ) enterdoes ( 402790B0 40218AB4 ) , ( 402790B4 4021824C ) ( dolit ) 3FFFF800 ( 402790BC 40218A9C ) cell ( 402790C0 40218174 ) + ( 402790C4 40218AB4 ) , ( 402790C8 402182C8 ) exit
\ **
holinepu
commented
5 years ago The decompiler program compiled by punyForth in RAM can be relocated in any address of flash rom now.
So, the application program can be compiled in RAM and then transfered to flash ROM for further use.
\ lastword FFFF and 40260000 or 3FFE88C0 ! 4026DDCC 3FFE88C0 ! \ only interpret? remains left in RAM for linking purpose 3FFE88C0 var-lastword ! \ lastword change to interpret? where is in RAM
` 3FFE88E8 var-dp ! \ here can be changed to allow more RAM spaces for further use
see see see words7
: t1 8 . ;
: t2 88 . ;
see t8
zeroflag
commented
5 years ago @holinepu is the source code of this project available on github?
holinepu
commented
5 years ago hi, Attila Magyar Thank you for your interest in the source code of punyForth development system. Please bear in mind that the system is still under development so it looks not very clean. We need the help to promote the system so as most users will like it. you can find the code here:https://groups.yahoo.com/neo/groups/armForth/files/esp8266%20punyForth/ 在 2018年9月11日 星期二 下午9:14:57 [GMT+8]， Attila Magyarnotifications@github.com 寫道：
@holinepu is the source code of this project available on github?
holinepu
commented
5 years ago The program compiled by punyforth in RAM area can now be moved to any flash rom area by executing
40280000 3FFE88C0>402xxxxx under esp8266 punyforth development system. After flashing it into the chip we can execute words7 to see the org of each word. Likewise we can execute words6 to decompile the whole words.
visit --> https://groups.yahoo.com/neo/groups/armForth/files/esp8266%20punyForth/
words7
( lfa= ) 40218114 ( 00000000 ) ( cfa= 4021811C
Dear Attila,
Sory if this is explained elsewhere I did not found it
I am very confused without a memory map of Punyforth, I need to understand what is happening with my reboot problems I am experimenting with different programs. It is also almost impossible to advance with Forth without having a memory map of the full system, I feel like shooting with a machinegun blindfolded ! It is tough.
I am copying below an example of a memory Map of MicroPhyton for ESP8266 please if you can edit it and correct it for Punyforth to help everybody visualize better how is Flash and Ram used in Punyforth, and were is our free space going after loading dependencies. The example of the internal of Micropython on ESP8266 helped me understand a bit further how the Rtos and Wifi will consume memory etc...
I have many questions left : 1) is Uber.forth at $51000 copyed from flash to RAM before execution ? because it does not make sense if we have 1 MB Flash , that 96 K or 64 K ram of ESP get so fast gone ! In my application 1 LCD with i2cbus and 1 wifi Repl , it consumes all my memory, I have left only 2K ! After that I can almost not add anything more without falling into constant reboots. 2) how can this 1MB or 4MB of internal flash been better used ? Is it now a wasted space ? 3) I suppose there is no way to add an external SD card, this would not help in any sense. What else can be done ?
Mphyton Memory MAP :
----------------------- 0X4030_0000
MEMORY MAPPED FLASH ROM (1M)
----------------------- 0X4020_0000 ] ] FLASH MEMORY MAPPING CACHE #1 ] 6 16 K INSTARAM ] 4 -------------------------0X4010_C000 ] ] K FLASH MEMORY MAPPING CACHE #0 ] 16 K INSTARAM ] ] ------------------------ 0X4011_0000 ]
FAST CODE INSTARAM (32K)
------------------------ 0X4010_0000
------------------------ 0X4001_0000
------------------------ 0X4000_0000 ] ] 9 ] SYSTEM DATA RAM (16K) ] 6 ] ------------------------ 0X4001_0000 ]
] K ] USER DATA RAM (80 K) ]
] ------------------------ 0X3FFE_8000 ]
Kind regards Enrique