Compiled code takes too much memory

[yesco]

It seems the compiled code is taking a lot of memory. A simple print('hello') takes 408 bytes. Each simple function seems easily take 1-2 bytes of memory. I'd expect print('hello') when compiled to take about about 32 bytes maximum (string 8 aligned, functioncall 4, function address 4, linked list entry fo hashtable entry 8 plus some overhead). For many code files, source code maybe 3600 bytes when compiled becomes 3200 bytes but when loaded uses about 10K memory. Seems like too much.

Example:

function h() print(node.heap()) end h() 31216 a=loadstring("print('hello')") h() 30808 print(31216-30808) 408 a=null h() 31216

[creationix]

It's probably lazy loading parts of the VM and standard library. Using loadstring needs to bring up the compiler.

Also make sure to manually do full GC before measuring heap if you want to be sure about the minimum used.

[yesco]

I used loadstring to minimize the impact of other stuff. Below, h() prints node.heap, gc() does h() collectgarbage() h()

Three functions just adding two numbers with short names takes 365 bytes each... I see no reasonable explanation for this! Looking further...

function x(a,b) return a+b end gc() 11064 11064 function y(a,b) return a+b end gc() 10664 10664 function z(a,b) return a+b end gc() 10336 10336 x=null y=null z=null gc() 11432 11432 collectgarbage() h() 11432 function x(a,b) return a+b end function y(a,b) return a+b end function z(a,b) return a+b end gc() 10336 10336

--- looking at the generated code on the desktop, it seems it should be about 3 instructions per compiled function:

$ cat fun.lua function x(a,b) return a+b end $ luac -l -s fun.lua

main fun.lua:0,0 (3 instructions at 0x806bb50) 0+ params, 2 slots, 1 upvalue, 0 locals, 1 constant, 1 function 1 [1] CLOSURE 0 0 ; 0x806bc88 2 [1] SETTABUP 0 -1 0 ; _ENV "x" 3 [1] RETURN 0 1

function fun.lua:1,1 (3 instructions at 0x806bc88) 2 params, 3 slots, 0 upvalues, 2 locals, 0 constants, 0 functions 1 [1] ADD 2 0 1 2 [1] RETURN 2 2 3 [1] RETURN 0 1

[TerryE]

This sizing is all standard (e)Lua. It's not an issue with this nodeMCU implementation. You've got the Proto function header, the malloced blocks for params, slots, upvalues, locals and debug info. That's life with Lua. Try using PHP and then watch the memory vanish!!

See my Unofficial FAQ at essp8266.com for more discussion on this.

[creationix]

ELua is a pretty good value with memory use for programming flexibility. It's as powerful as about any scripting language on the market and yet can run on microcontrollers.

I'm currently designing a smaller language that will run even down to 8-bit avr microcontrollers. The language is much simpler than lua, but is extremely memory efficient with it's bytecode. It's decently fast too.

But in my language you won't get strings or tables or even function arguments. All runtime values are integers. All variables are global either indexed by global offset or push/popped on a global stack. User functions are expressions and thus return exactly one value.

If fact, it's a pretty terrible language for anything remotely complex and the lack of local variables makes for not very reusable code libraries.

But it's fast and uses very little ram. A function to add a + b is simply ADD GET a GET b which when assembled compiles down to 5 bytes.

[TerryE]

@creationix, Tim I do think that you need strings and tables (arrays) but if you are starting with embedded in mind then you could do it maybe twice as dense as Lua.

[creationix]

For simple kids stuff, I don't need the strings and tables (my goal is to teach basic programming ideas using hardware and software). But yes, you quickly reach a level where they are needed when adding any level of complexity. The closest language to mine is bitlash for arduino avr boards. They added strings and function parameters in the 2.0 version.

[yesco]

I'm was quite impressed with a working lua on this device, and initially excited, but the RAM is a real difficult issue.

@TerryE Well, I thought this lua was especially built/coded to be efficient for embedded systems. Not sure which part of that is. In any case I've read the Unoffical FAQ and it explains some.

However, comparing this "embedded" lua to the embedded javascript "espruino", that takes some shortcuts; It actually interprets/parses the source every time in order to save memory, insteado of compiling to byte code. It's "fast enogh". ( https://github.com/espruino )

Even with 10KB of ram left on nodemcu it'll easily reboot with some function calls doing some string manipulations. Maybe because no GC is being invoked during the string processing.

@creationix I assume you know FORTH. You can easily implement a parser/compiler that is interactive, implementing only 13 primitives (minimal 9?), then building the rest of the language at higher level. It's, however, as you indicate a stack based language, very minimal, but at times quite appropriate for embedded systems that need be interactive. I once implemented on on a CPM z80 machine when I was 16. Just read a short manual of the language and then started generated z80 instructions. I sketched an implementation for ATTINY once but didn't pursue it, as it was a bit too constrained.

I'm very excited about the nodemcu as it's price and capabilities (wifi, ram, flash, pins) are quite a steal, even compared to attiny, or any arduino for that matter.

Still may consider using lua as an "operating system"/loader for my own c-functions that would be doing my real work. Lua provides some nice primitives...

[TerryE]

@yesco, #538 almost doubles the RAM available and #468 (when I finish the work) decreases the size of code in RAM by 40% or so, so together these will allow you to have perhaps 3x the amount of Lua source loaded into RAM at any one time. There are new ESP8xxx chips (with a lot more RAM) planned for release in the next 12 months, so this is only a transitory issue, IMO.

[MarsTechHAN]

And we have another 8kb from espreasif's new patch(´･ᆺ･`)

[marcoskirsch]

And we have another 8kb from espreasif's new patch(´･ᆺ･`)

@MarsTechHAN do you mean there are another free 8 KB of RAM? Or it's taking another 8 KB?

[MarsTechHAN]

@macroskirsch A jack PR a package and almost double the free ram of NodeMCU, and espressif use similar way(maybe) and free up 8kb, and they make a patch for 1.2.0(which NodeMCU have already been added). So, another 8kb free memory

[yesco]

@TerryE good work... @468 has a cool discussion.

synched all up, things are improving.

print(node.heap()) 34296

@MarsTechHAN That's another 4K since few days ago.

It's funny, haven't seen the actual RAM size listed on any datasheets. I guess it's 64K currently.

[jorgempy]

shoudn't this be closed?