Rework of the lua source hierachy to support a unified apporach to ESP8266 and ESP32

[TerryE]

Background

The Lua source hierarchy is currently based on the following fork tree:

• Standard Lua 5.1.5 which is designed to compile on any POSIX toolchain and released in Feb 2012.

• eLua is based on 5.1.5 but has a number of enhancements and changes to optimise it for embedded use, including use of newlib toolchain and a set of template hardware drivers. The main enhancements to the Lua core were in the inclusion of an Emergency Garbage Collector (EGC) implementation of lightweight C functions and ROM-based tables.

• NodeMCU was again a fork of eLua by Zeroday, but because the non-OS SDK structure and execution models where quite different from the eLua assumptions, NodeMCU really only used the Lua core components. Also the SDK interface is quite different to newlib one, so this fork also introduced a lot of code changes (e.g. the string library calls were replaced by the equivalent SDK cstring ones.)

Issues

• Since the initial 1.x versions of NodeMCU, we have subsequently addressed many of the source level conflicts between newlib and the SDK libraries which in turn mean that many of the original nodeMCU changes are no longer needed. Moreover

• The ESP32 IDK is newlib-based and layered on RTOS and therefore is incompatible with these initial NodeMCU changes.

• The subsequent Lua versions have incorporated many useful components, for example 5.2 incorporates the eLua ECG and lightweight C functions, and therefore provides most of what we use from eLua. The only additional bit that we use is ROTables and we are currently investigating reimplementing this because of the performance hit on the ESP flash architecture. 5.3 supports integers as a native type alongside floating point. The eLua project itself is now pretty moribund (only 11 new threads on the eLua DL during 2016).

Whilst we have arrived at our current NodeMCU Lua code base in a set of logical steps, the reality is that this could be viewed as a dead-end from a maintenance perspective. I believe that we should now take stock and decide together how we approach a scenario where we wish to maintain a common Lua code base for both NodeMCU ESP variants.

• Do we believe that is is worth maintaining a common Lua code base for the two projects?
• Do we go back towards the newlib eLua reverting no longer needed changes?
• Do we move forward onto the 5.2 or 5.3 code?

More thought and discussion is needed, but my view is that this should be around options to maintain a common code base.

[djphoenix]

1. I really prefer to maintain as much common code as possible, because (I believe) sooner or later Espressif may release IDK for ESP8266-RTOS.
2. I vote up for update lua codebase to latest one (maybe 5.3), this have very good enchancements. Also I prefer to get it clean as possible, for ability to update it when new patches or versions released.

[TerryE]

Hummm, things get a little more complicated with 5.3.3. For a start it has proper 32-bit support as an option. Lua 5.1.5 uses a unified number type which is a 64 bit floating point and which can accurately represent integers up to 2^52, IIRC. With Lua 5.3.3 32-bit build this instead uses separate types for int32 and 32 bit FP. OK, we would lose the exact large integer support for integers > 2^31, but this a lot faster on the 8266, and a lot faster again on the ESP32 since this has H/W support for 32-bit FP.

[jmattsson]

That does sound tempting considering the hardware we're running on.

[TerryE]

The only issue is that there seem to be bleats about increased memory usage in 5.3. Still researching :(

PS. I can't see any obvious changes other than strings are now only interned if 40 bytes or shorter. In fact there are now two string types: short strings which are unique and long strings which aren't. I guess this is to avoid the cost of hashing long strings, since it is assumed that these are rarely duplicated. But note that they are still copied by reference and hence can be reused, it's just that long strings aren't put into the string hash table, so

local s = '012345678901234567890123456789'
local a = s .. s
local b = b
local c = s .. s

will now create two identical 60 character strings '012...789', one used by a and b, and a second by c.

The reasons for this are pragmatic. The hash function is O(N) and the chances of two long string being accidentally identical are small so there are strong arguments for only interning strings under a length threshold.
And a LRU cache has been added for accessing C strings to avoid hashing short Cstrings in the case of a cache hit, but a bit bizarre implementation.

[TerryE]

Having done a bit of the work, and though about this some more overnight, I feel that the next step is for me to add a rotables mod into the 5.3.3 build, so that it can be built both as a POSIX standalone (under x86) and under the xtensa toolchain with a tweaked node library so that it can also run as a firmware image on the ESP8266 (or ESP32) for evaluation and to inform discussion as to whether we should proceed with a proper integration into an SDK 2.x build. I'll set up a clean repository so that others can track this.

[TerryE]

A quick update.

I am getting to grips with Lua version 5.3.3 and the differences from 5.1.5. Perhaps the main one is that a lot of optimisations and enhancements to support embedded implementations are now in the core -- such as the EGC. The one big omission is no support for rotables. What I have done so far is to add support for RO TString constants, with some extra TS variants of get and set methods so that you can use them in the C API. So now LUA_TS(__index) in the code is a TString * to the extern in ROM which is the TString for "index". The ROstrt is in the searchlist for creating new strings so references to `indexin any compiled Lua code will pick up the same TString instead of creating a new one in RAM. Hence string comparisons use a pointer comparison, occasionally dropping down to a memcmp (since now the length of both strings are known) rather thanstrcmp(), so less entries in the RAMG[L]->strt` and in principle a lot less unaligned ROM access exceptions.

The ROstrt is generated by a preprocess of the code which detects these LUA_TS() usages and generates a c header file and an assember file which contains the static initialisers. It will take me another couple of days to get this working fully on the x86 platform. At this stage, I'll push a copy to my github repo for anyone interested to review.

[TerryE]

Another quick update. I've got Lua 5.3 working with ROstrt initialised in the .text segment with the 303 common TStrings used in the compiler and core libraries, with the extra API methods to use TString parameters directly when appropriate. Need to add the rest of the ROtable patch next and extend the test suite. Next build will be on a 32-bit Ubuntu VM with 32 ints and floats. Upwards and onwards.

[TerryE]

I've done a rough cut of my working paper documenting progress in this gist: LROR (Lua Read-Only Resources) in Lua 5.3. That's on top of fitting my new kitchen and becoming a grandfather (though the labour on this one was down to my daughter!) Any general comments then feed them back here :smile:

I am currently hammering the table implementation on an x86 build.

[jmattsson]

Two comments (other than "good job!") so far:

1. Pretty please with sugar on the top, don't #include C_HEADER_STDIO in the final stuff; Let's do the cleanup on the ESP8266 side rather than pollute the POSIX and ESP32 sides. It's largely the custom-modded eLua stuff which needed the c_stdio.h stuff in the first place, and if we're replacing it all, we really shouldn't need it.
2. Is there a version of LROR_WORD(x) than can handle strings with non-C-identifier symbols in them? LROR_WORD(x and y, with some z!) clearly wouldn't give a valid extern declaration. Or are LROR strings limited in this regard?

[TerryE]

@jmattsson, picking up these:

1. I went through some of this with the luac.cross port. The issue that we have with GCC is that the search list for system includes is configured in the compiler build, and it got very convolved. The xtensa toolchain is based on newlib libraries, which are different to the standard Linux libraries, and the non-OS SDK ones are different again. I agree that we might be able to back-port most of these SDK changes into ESP8622 toolchain newlib libraries, but there will still be some incompatibilities between this and the Linux set. At the end of the day, IMO it is more important that we have a single code-base for all three targets (ESP8266, ESP32 and Linux) and this preprocessor define trick works fine. This comes into the category of polish rather than show-stopper.
2. Yup LROR_STRING(does_not_compute, "Does not compute"). The same string can can be declared twice with different symbolic names and this will be handled, but the same symbolic name can't be used with two strings.

[jmattsson]

1. It shouldn't be too bad. As the first stage, simply renaming our c_whatever.h files to whatever.h and using -nostdinc will preference our "standard C" headers. Adding a -isystem "$(xtensa-lx106-elf-gcc -print-sysroot)/usr/include" will get the compiler to find the HAL etc (and any non-overridden headers). And yes, while in some ways it's polish, it's also about reducing the number and extent of Lua source files that have to be modified. Feel free to ping me when you're at a stage you're wanting to start integrating and I'll try to get some time to lend a hand (which I probably should for the ESP32 side anyway).
2. Nice!

[TerryE]

@jmattsson Johny, I am a little less concerned about the issue of minimising the changes. First step is t evaluate viability and benefits. And if we have a go then we review some of the implementation strategy. I did this with the LCD patch and did a master diff before the PR. I went through this diff and in a few areas reworked the modification to minimise the code change.

Once I have a clean version which passes my extended test suite, then I think that we're at the point where I can push the branch for review and evaluation.

[jmattsson]

Oh yes, you might've noticed I said "in the final stuff" in regard to point 1.

It's all sounding very promising so far though. This could be as big a RAM improvement as when we originally managed to get the C-string constants moved into flash. No pressure... ;)

[TerryE]

Just another quick update. I am still plugging away at this making steady progress. It's just that I am time-slicing it with little jobs like fitting the new kitchen in my new house and doing the plumbing. There are all sorts of subtleties involved here, so I am not rushing and doing lots of brood time so the changes don't compromise some of the nice features of 5.3 and also don't cause unacceptable performance hits.

A little example is that one of the things that you want to do for example it to set the metatable for a ROtable using the normal API. You do this if you want to extend string or table, for example. But the metatable field in standard Lua is in a field in the Table structure. You can't do this in the case of ROtables since you need this one field to be RW, so you need to store it the registry. However, you don't want to do this in general since the ability to override almost every data type in 5.3 means that this is checked a lot at low level so for RWtables, you need to keep the field in the Table record for RW variants. Hey-Ho.

Anyway, I've updated the White Paper in my Gist with current progress. As soon as I've got a standalone working compliant to the paper, I'll upload it to my repo.

[TerryE]

@flip111 I am multiplexing a lot of issues at the moment. A straight port of Lua 5.3 would be reasonably easier but would not perform as well, nor ave anywhere as near as small a RAM footprint as our currently esp8266-optimised 5.1 version. So I have to merge in these optimisation features and add a few more because the extra Lua 5.3 feature set comes with an extra RAM footprint. So slow progress.

Read the working paper linked above :)

[flip111]

Very interesting how this is evolving, good work @TerryE

[TerryE]

@jmattsson @pjsg, I am loosing the plot!! I've discussed backporting these enhancements into the current NodeMCU lua 5.1.5 implementation but I can't for the life of me remember where. Anyway, I've done the first cut to compile clean and am in the middle of testing. But on-chip testing is a bitch. Have either of you got a good primer on using the remote gdb stub. Do you use it?

Anyway, I've found the easiest way is to work on luac.cross which I can debug using regular gdb on the laptop. This has been a very useful exercise. If one of you guys can point me to the correct issue / PR then I'll give a fuller update there.

[pjsg]

The gdbstub is definitely not perfect, but there is some documentation in the gdbstub module doc. It is really good for catching crashes and poking around..... If you

#define GDBSTUB_BREAK_ON_INIT 1

then it will enter gdb fairly early on (as it is initializing the lua modules). This may be too late, in which case you probably want to move the gdbstub_init() call to somewhere rather earlier in th eboot process.

Also apologies for not being engaged for the last month or so, I've been rather under the weather...

[jmattsson]

@TerryE Are you referring to https://gist.github.com/TerryE/8afa5022042291b8add1ff3886f6c014 that you linked above?

[TerryE]

I've used my luac.cross technique to build a stripped done NodeMCU lua core VM running on the host so I can debug the core code on my laptop. Bizarre but a lot quicker and easier than using gdbstub. Onwards!!

[TerryE]

I am pretty close to having an evaluation version of the NodeMCU Lua 5.1.5 evaluation version ready. As I eluded to earlier this is based on a backport of the RO TString technology that I developed for my Lua 5.3 port, but leaving in the current ROTable implementation rather than trying to implement true RO Lua Table types.

This has been a very useful exercise because it has allowed me to appreciate some issues in concrete form.

Having a common code base

We should have a single Lua core code base that will compile to the host platform (gcc compatible *nix), esp8266 and esp32.

As I mentioned above I already have a version of the NodeMCU lua interpreter running on my dev VM on my laptop. OK, the libraries are different (e.g. io and os work, but not node or file) but we can converge on this later. Having a host-runnable means that full gdb feature are available for testing as well as the Lua test suite, etc. And minimising code variants means that we can have increased confidence that bugs that have been exercised in host testing are also removed from the esp8266 and esp32 variants.

Just to be clear this host version has ROTables, and RO TStrings; it's just that they lie between _etextand _edata rather than in some defined Flash memory address space. Code compiled with this host luac will run on the esp chips as well as code dumped from this host *lua**.

What I find really irritating about the NodeMCU code is that it is full of crap, for example chunks of source #if 0 commented out and conditional on defines that we no longer support (for example NodeMCU must be built with full ROtable support). It's probably not worth clearing this out, though to be honest when I am debugging the core code and find this crap is confusing things then I have started to remove it.

However what is very clear to me is that when we come to the Lua 5.3 port we should have a very clear set of objective which then give rise to a set of change templates or patterns, but that we should only make changes above and beyond this that are clearly documented.

Getting the peephole optimisation right

There are some aspects of our Lua implementation which cause a noticeable performance hit for no good reason and I really think that we should address these. In particular, access ROM based readonly tables and elements currently is a lot slower than RAM based ones.

Interim 5.1 update or straight to 5.3?

To be honest this 5.1 version was a pretty important step for me. It will run faster than the current version of the core code; I could extend it so that the ESP32 version could just use this rather than its own variant. It is certainly worth considering a proper NodeMCU branch for its evaluation, but is it worth promoting to dev (at some point after enough testing) or should we just plan to go direct to 5.3?

The points about 5.3 from a developer PoV is that it is not 100% code-compatible with Lua 5.1. It supports separate float and int number types and on the ESP chips, it makes sense to make these both 32 bit, especially on the ESP 32 where these are both supported in H/W . But this will lose the ability to do 52-bit integer arithmetic, so we might need to add a in64 library.

[flip111]

Better proof first that ESP8266+ESP32 can be combined into 1 firmware before adding lua 5.3 support

[TerryE]

@flip111, you misread this goal. We will never have one firmware because the two builds target different generation processors and have different underlying platform APIs. The goal here is to have a single Lua core source base and at the moment, the two branches are different. If I can get the same Lua source running on the esp8266 core and on a 64bit Intel core then I can certainly get the same source base running on esp32.

[flip111]

Seems you are confident that the 5.1 update will work, maybe go for 5.3 then.

[TerryE]

Yup, but the Lua 5.1 has advantages performance and inter-platform, and is fully source code compatible, so as long as it works, then we could promote it to dev. The move to 5.3 really merits a wider discussion and buy--in from the committers.

[jmattsson]

I'm on the fence regarding 5.1-upgrade vs straight-to-5.3. On the one hand I don't like the extra work required to get the interrim 5.1 in, but on the other hand it would likely be able to go in sooner, which would benefit everyone. As long as there's a reasoned argument, I could support either approach. I would very much like to see the Lua core being cleaned up though - as you've noticed, we've got a metric truck ton of cruft in there. Hopping straight to a 5.3 might be the least painful option.

Regarding integer types, we could perhaps do a userdata int64 with __add etc meta-methods? Would it be possible to auto-promote to such a userdata type when needed? Checking add & mul on the int type should allow for catching overflows and trigger promotion, but I don't know how feasible it would be to augment the 5.3 codebase.

[TerryE]

I am still testing the 5.1 backport on a dev build to put it through the Lua test suite clean. I've also done a full code review using meld against a vanilla Lua 5.1 and have made the following observations which swing my view of the one step vs two step path (straight to 5.3 vs 5.1 upgrade then 5.3).

• The eLua variant add support for the EGC, lightweight C functions and rotables as well as other cross support options such endian and packed vs unpacked support.
• IMO, the rotables implementation is a bit of a hack as it is very run-time inefficient (lots of linear strcmp scans of flash), and all rotables args are void * types so there is no compile-time validation of a lot of stuff that should be checked at compile-time, which makes testing changes fraught. Also the way of implementing lightweight functions doesn't follow standard Lua coding patterns.
• NodeMCU uses the eLua Lua core variant for the EGC, lightweight C functions and rotables functionality, but the xtensa architecture doesn't need the endian and packed stuff. (It also discards all of the eLua platform and driver implementations.)
• Standard Lua 5.3 has added EGC and lightweight function support as well as other changes to support embedded processors not included in eLua (separate integer and float sub-types and the ability to force the number size to 32-bits, for example). However, this is a re-implementation using standard coding patterns.
• One of the architectural changes that 5.3 implements to do this is to split the ttt type field in Lua Values and Garbage Collectables, into a 4-bit bit field and 4-bit subtype. The following types are sub-typed:
  • Numbers have separate integer and float subtypes
  • Strings have separate interned and non-interned subtypes. Strings of 40 bytes or less are interned. Long ones aren't. This is for performance: hashing is a function of length and in the majority of implementations (especially embedded) the chances of a duplicate instances of a long string is extremely small.
  • Functions have separate full and lightweight subtypes.
• Lua 5.3 also adds a new separate GC category "don't collect" which I will use to tag all RO resources.
• In eLua rotables are a separate type that is handles completely differently to normal tables, and hence this implementation leaks out into a lot of code changes in ltable.c, lvm.c, etc.
• My proposed RO table implementation makes Tables a fourth sub-type type by splitting tables into two separate RW and RO table subtypes, and the difference is only in the low level access routines, and is so is well encapsulated. The RO table implementation internally is still a vector list as with the current eLua rotables implementation, but access is through a look-aside cache so 95% of the RO table access are direct.

So my conclusion is that the changes to get from standard Lua 5.3 to a NodeMCU 5.3 are lot less than cleaning up my current 5.1 WIP version. And in my experience the smaller the change, the less bugs get introduced so the more stable the implementation. So I now propose to release a 5.3 evaluation version next.

Even so, doing the 5.1 exercise has been valuable in that it has forced me to think through the API issues in minimising the impact on the rest of the NodeMCU ecosystem, and has also cleaned up some aspects of my implementation.

[jmattsson]

:+1: Nice summary, thanks!

[pjsg]

I think that makes sense -- it is nearly always the right decision to move forward when making large changes....

[TerryE]

Getting your head around all of this isn't easy. I kept coming across fine details of the implementation and wondered why on earth is was coded that way. So I decided to go back to basics and did a

/work/esp8266/lua-5.1.5/src$ for f in *.[hc]; do test -f /tmp/n/app/lua/$f && meld {.,/tmp/n/app/lua}/$f; done

(/tmp/n is just a symlink onto my dev VMs nodemcu directory) when I realised that Bogadan's coding style and approach for his eLua changes is not very aligned to Roberto's approach to the core Lua code.

In fact 5.3 includes a cherry-pick of the best bits of eLua but in Lua style, and this really makes eLua obsolescete, IMO, given that we don't use any of the other eLua ecosystem. I see no advantages of attempting to retain any this extra eLua stuff for the 5.3 version.

Throw in some of original hacks that we made in getting our early versions up and we've currently got a bit of a mess. So what I want to do is to swap out the Lua core entirely, but adopting the pragmatic approach of keeping the NodeMCU module layering and things that work well, such as our (or Johny's to give credit where it is due) linker magic the same API or with the absolute minimum changes.

I want to keep the triple platform support : dev host, ESP8266 and ESP32, but the dev host is in a limited version: I am currently not supporting a standard Lua RTS, but instead the Lua complier luac has new -X option which allows you to not only to compile but also execute a script (on the host), but this can only use the modules in the core lua and lua/lua_cross directories. This still allows the limited use of NodeMCU Lua on the dev host for bootstrapping and gdb testing, which is all I need. (We can always revisit this later if there is a demand.)

I getting rid of all of the cstr... in the code. Unfortunately we still need to compile against some of the clib c*.h includes on the ESP8266, so I do need a minimal abstraction layer here to avoid compile-time errors.

This is all slow going since I have to re-engineer all of the Lua internals from code inspection and its implementation is very dense so this can be hard going. Still, pondering this exercises the mind when I am rebating 72 hinge sides for my 12 doors.

//Terry

[TerryE]

@jmattsson @pjsg @djphoenix @devsaurus @marcelstoer I've updated my white paper on what I am doing for the 5.3 version. Sorry that it's a long read, but it took a lot longer to write :-(

[jmattsson]

Hi Terry,

Some thoughts when reading this:

• It would seem feasible to put together a post-build tool which reads the ELF and checks all .literal* entries against the LROR strings, and reports if it finds duplicates, to help with optimising the string storage/usage.

• I'm not 100% clear on whether the LROR TString macros could also generate extern const char * entries for use within regular C. If it could, the above tool could be even more aggresive in suggesting string folding. Perhaps even automate it at some point. Main use case I'd see being inside a module where one might want to test a Lua argument against a predefined number of strings (e.g. via luaL_checkoption()).

  • I'm still not keen on C_HEADER_STDIO. I'd rather we used -isystem with a wrapping stdio.h file when dealing with the non-OS SDK. Aside from mussing with the syntax hilighting (due to missing "" or <>), it also introduces unnecessary changes to code from upstream. Both Linux/Posix and ESP32 IDF provide sane C99 compatible headers. Let's not make those suffer because of the legacy insanity of the NON-OS SDK.

  • Regarding the look-aside cache, could we avoid it by constructing the LROR data in a "better" structure, so that searching it might become O(1) or O(log(N)), e.g. by storing hashes or even plain sorting?

• I think LROR_DEFINE_GLOBAL_TABLE may be better named LROR_BEGIN_GLOBAL_TABLE, considering it needs a matching END.

• LROR_TABLE_ENTRY_NAME_CFUNC seems somewhat tedious to type. Would it make sense to shorten it? It's nicely descriptive as is, just very... long.

• The xxx_module_selected magic's purpose is to put enabled modules into the correct linker section. It may be possible to use as-is, or with a slight tweak. It effectively generates an __attribute__((section("x_or_y"))), so as long as that definition is applied to the "root" struct of a module it should cause the linker to pull in the correct bits.

  • Would use of an anonymous union save having to change to RWarray? Just trying to keep upstream code mods to a minimum.

• Heh, PL/SQL, those were the days...

Thanks for the write-up!

[TerryE]

Johny, thanks for the feedback. Much appreciated. As you no doubt noticed it is still a rough draft with the usual typo / garbage and inconsistencies. The main thing is that I want a level of comfort in the end goal from the other committers before I start shipping alpha evaluation versions.

It would seem feasible to put together a post-build tool which reads the ELF and checks all .literal* entries against the LROR strings, and reports if it finds duplicates, to help with optimising the string storage/usage.

Yes, a nice to have but remember that each LROR string has a TString header and a hash entry. OK if the string is in use in Lua then this is saving the equivalent RAM which is great, but if its not then its just a ROM overhead. So you'd need quite a few duplicates to make this worthwhile. Picking up string uses where the LROR version already exists is another issue..

I'm not 100% clear on whether the LROR TString macros could also generate extern const char * entries for use within regular C ... luaL_checkoption()).

It does this, for example:

  static const TString *const opts[] = {LROR_WORD(stop), LROR_WORD(restart), LROR_WORD(collect),
    LROR_WORD(count), LROR_WORD(step), LROR_WORD(setpause), LROR_WORD(setstepmul),
    LROR_WORD(isrunning), NULL};
  static const int optsnum[] = {LUA_GCSTOP, LUA_GCRESTART, LUA_GCCOLLECT,
     LUA_GCCOUNT, LUA_GCSTEP, LUA_GCSETPAUSE, LUA_GCSETSTEPMUL,
     LUA_GCISRUNNING};
  int o = optsnum[luaL_checkTSoption(L, 1, LROR_WORD(collect), opts)];

I'm still not keen on C_HEADER_STDIO. I'd rather ... Let's not make those suffer because of the legacy insanity of the NON-OS SDK.

Neither am I, but I think that ESP8266 part will be shipping and need S/W enhancements for years, so we do need an approach that embraces the ESP8266 build variants. What we really need to do is a full rework of the sdk / libc headers to make them newlib compatible, but this is a major chunk of work in its own right. The `C_HEADER_STDIO``apporach is a hack but it works. We only need it for the components which are compiled on all three platforms such as the Lua core. Any platform-specific modules aren't bound by this constraint.

Regarding the look-aside cache, could we avoid it by constructing the LROR data in a "better" structure, so that searching it might become O(1) or O(log(N)), e.g. by storing hashes or even plain sorting?

Yes we could and I did this in the first version. However, the main advantage of the linear list is that this enables tables to be declared using standard macros and normal conditionals work fine. (With strings the LROR macro will still be picked up by the preproessor and included in the ROstrt even if the actual inline reference is then removed by the C preprocessor). The thing that you are forgetting is that the H/W flash caching works pretty well for serial and looping access to code, but if you are walking sparse RO data structures then you are running at SDIO bandwidths = slow. The look-aside cache is RAM based so a cache-hit goes right to the correct entry.

So I see this cache as a bonus, rather than something to be avoided.

I think LROR_DEFINE_GLOBAL_TABLE may be better named LROR_BEGIN_GLOBAL_TABLE

Agreed

LROR_TABLE_ENTRY_NAME_CFUNC seems somewhat tedious to type. Would it make sense to shorten it? It's nicely descriptive as is, just very... long.

The third high level language that I learned was Cobol (at the age of 18!) and it shows :smile: OK, I will add the acronym versions eg. `LROR_TENCF()

The xxx_module_selected magic's purpose is to put enabled modules into the correct linker section. It may be possible to use as-is, or with a slight tweak...

Yup, I have retro-engineered this and understand how it works. The issue that I have is that I can now define a module table and its optional entry function purely declaratively using LROR macros, so I don't need the NODEMCU_MODULE() macro for this purpose, but I still need the module to generate the XXXX_module_selected reference so that your linker magic works, sso maybe its just easier to have another variant of the NODEMCU_MODULE type with just the module argument.

Would use of an anonymous union save having to change to RWarray?

I will try it.

Incidentally taking of avoiding changes to the source, the libc versions of the headers require c_strcmp(), etc. to compile, so the includes header does a #define strcmp c_strcmp and then the linker remaps these to the ROM ETS versions. Sad or what! Still it compiles and runs as you need.

Again, thanks for the feedback.

[TerryE]

@jmattsson, just as a quick note, I've tried the anonymous structure / union and it works fine. So this is one class of source changes avoided. Thanks.

I must admit that I've never used this construct before because my coding philosophy in general is to use the compiler to do as much checking as possible, so I always try to hard-type variables and structures (e.g. I avoid void * type whenever practical). OK, this is extra hassle in trying to get clean compiles, but I am a careless bugger and far sooner find my errors at compile time than run time.

[TerryE]

For info: I've updated my draft LROR whitepaper with my progress to date. The tl;dr is that I am working my way through all of the issues thrown up by the Lua Test suite. Progress is good, but slow.

I would appreciate constructive feedback from any of the committers or other readers. It quite a balancing act trying to keep changes to the Lua 5.3 core to a minimum yet still keeping a low RAM footprint and performance as good as possible, and better than the current NodeMCU 5.1 implementation.

[TerryE]

For info: another update to my draft LROR whitepaper with my progress to date. Key changes which reflect changes so far during testing:

• Extra details on the multi-host make system
• Extra details on the legacy module support and how this can be used to compile and run modules written for the Lua 5.1 version with no or minimal source changes.
• Slight rework of ROTable implementation to support the same.
• Explanation of the LROR Lua module support and on module rebuilding of Flash based Lua modules. This roughly doubles the RAM available to Lua applications if the developers care to use it.

I am expecting the normal zero review feedback

[NicolSpies]

@TerryE, I read the LROR update every time, I try to understand and learn, understand some and realize I cannot give any feedback. :flushed:

[TerryE]

@NicolSpies Nichol, thanks for this. Any comments gratefully received. Just read it from your perspective.

The highlightts are the we will be upgrading to a Lua 5.3 baseline with the extra benefits that this enables. Lua numbers are now always 32-bit so the per-value storage overhead is the same as with the integer build, and if you stick to integer calculations (e.g. +,-,*,//, and now native bit operators such as >>) then all of the calcuations are done using integer arithmetic. However you can also use floating point (32-bit) if you need to, so you have the size and speed advantages of an integer build but with the calculation convenience of a floating point one.

We aren't losing the advantages of the eLua features such as storing library tables in ROM, but the execution times for RO table access are a lot faster. Also as a Lua developer, if you bundle your code into Lua modules, you will now be able to store your modules in Flash and run them directly from there, so the effective available RAM will be more than double what we have on our current version.

It's just that doing this is all complicated and I am balancing finishing off the new house that we've built; doing all of the IoT based instrumentation for it and this project, so whilst progress is steady it's not as quick as I would like. Still in the great scheme of things it doesn't matter too much if this takes another couple of months to get past beta testing. :smile:

[TerryE]

We still have some modules which need minor tweaks to accommodate the Lua 5.1 to Lua 5.3 API changes. I will fix the core modules: file, net, node, uart, wifi.

However there are 16 other modules which require changes to make them compile clean: cron, gpio, http, i2c, mqtt, ow, perf, rc, rtcfifo, sjson, sntp, spi, tcs34725, u8g, websocket, ws2812. These are typically 2 or 3 lines of API changes. I will temporarily rename these files to a .c_todo extension so I can get to clean module compiles. I see no point in doing this work for the maintainers now, as this is only going to create merge issues if any other parallel maintenance is on-going.

Out of interest, here is a roll-up of the Lua 5.1 to 5.3 changes:

Changes in the Language

• The concept of environment changed. Only Lua functions have environments. To set the environment of a Lua function, use the variable _ENV or the function load.
• C functions no longer have environments. Use an upvalue with a shared table if you need to keep shared state among several C functions. (You may use luaL_setfuncs to open a C library with all functions sharing a common upvalue.) To manipulate the "environment" of a userdata (which is now called user value), use the new functions lua_getuservalue and lua_setuservalue.
• An integer subtype has been introuced for numbers. Although this change should not affect "normal" computations, some computations (mainly those that involve some kind of overflow) can give different results. You can fix these differences by forcing a number to be a float (in Lua 5.1 all numbers were float), in particular writing constants with an ending .0 or using x = x + 0.0 to convert a variable. (This recommendation is only for a quick fix for an occasional incompatibility; it is not a general guideline for good programming. For good programming, use floats where you need floats and integers where you need integers.)
• The conversion of a float to a string now adds a .0 suffix to the result if it looks like an integer. (For instance, the float 2.0 will be printed as 2.0, not as 2.) You should always use an explicit format when you need a specific format for numbers. (Formally this is not an incompatibility, because Lua does not specify how numbers are formatted as strings, but some programs assumed a specific format.)
• Equality between function values has changed, In Lua 5.1 a function definitions aways created a new value. Ths ha been optimized in Lua 5.3 and a definition can reuse a previous function value if there if the old and the new function are essentially the same.
• Lua identifiers cannot use locale-dependent letters.
• Doing a step or a full collection in the garbage collector does not restart the collector if it has been stopped.
• Weak tables with weak keys now perform like ephemeron tables.

Changes in the Libraries

• A module can no longer return a function value. It must return an array. It is easy to set up a module with regular Lua code. Modules are not expected to set global variables.
• Functions setfenv and getfenv have been removed, because of the changes in environments.
• Function math.log10 is not longer supported. Use math.log with 10 as its second argument, instead.
• Function loadstring is not longer supported. Use load instead; it now accepts string arguments and are exactly equivalent to loadstring.
• Function table.maxn is not longer supported. Write it in Lua if you really need it.
• Function unpack was moved into the table library and therefore must be called as table.unpack.
• Character class %z in patterns is not longer supported, as now patterns may contain '\0' as a regular character.
• The table package.loaders was renamed package.searchers.
• The Lua 5.1 byte code verification algorithm contained a numberof exploitable flaws and so has been dropped. All functions that load code (load and loadfile) are potentially insecure when loading untrusted binary data. When in doubt, use the mode argument of those functions to restrict them to loading textual chunks.
• The bit32 library has been deprecated. It is easy to require a compatible external library or, better yet, to replace its functions with appropriate bitwise operations.
• The Table library now respects metamethods for setting and getting elements.
• The ipairs iterator now respects metamethods and its __ipairs metamethod has been deprecated.
• The following functions were deprecated in the mathematical library: atan2, cosh, sinh, tanh, pow, frexp, and ldexp. You can replace math.pow(x,y) with x^y; you can replace math.atan2 with math.atan, which now accepts one or two parameters; you can replace math.ldexp(x,exp) with x * 2.0^exp. For the other operations, you can either use an external library or implement them in Lua.
• The call collectgarbage("count") now returns only one result. (You can compute that second result from the fractional part of the first result.)

Changes in the API

• Pseudoindex LUA_GLOBALSINDEX has been removed. You must get the global environment from the registry (see §4.5).
• Pseudoindex LUA_ENVIRONINDEX and functions lua_getfenv/lua_setfenv have been removed, as C functions no longer have environments.
• Function luaL_register is not longer supported. Use luaL_setfuncs so that your module does not create globals. (Modules are not expected to set global variables anymore.)
• The osize argument to the allocation function may not be zero when creating a new block, that is, when ptr is NULL (see lua_Alloc). Use only the test ptr == NULL to check whether the block is new.
• Finalizers (gc metamethods) for userdata are called in the reverse order that they were marked for finalization, not that they were created (see §2.5.1). (Most userdata are marked immediately after they are created.) Moreover, if the metatable does not have a gc field when set, the finalizer will not be called, even if it is set later.
• luaL_typerror was removed. Write your own version if you need it.
• Function lua_cpcall is not longer supported. You can simply push the function with lua_pushcfunction and call it with lua_pcall.
• Functions lua_equal and lua_lessthan are deprecated. Use the new lua_compare with appropriate options instead.
• Function lua_objlen was renamed lua_rawlen.
• Function lua_load has an extra parameter, mode. Pass NULL to simulate the old behavior.
• Function lua_resume has an extra parameter, from. Pass NULL or the thread doing the call.
• Continuation functions now receive as parameters what they needed to get through lua_getctx, so lua_getctx has been removed. Adapt your code accordingly.
• Function lua_dump has an extra parameter, strip. Use 0 as the value of this parameter to get the old behavior.
• Functions to inject/project unsigned integers (lua_pushunsigned, lua_tounsigned, lua_tounsignedx, luaL_checkunsigned, luaL_optunsigned) were deprecated. Use their signed equivalents with a type cast.
• Macros to project non-default integer types (luaL_checkint, luaL_optint, luaL_checklong, luaL_optlong) were deprecated. Use their equivalent over lua_Integer with a type cast (or, when possible, use lua_Integer in your code).

[szmodz]

@TerryE Is the 5.3-based version available for review? Doesn't matter if some of the modules are broken.

[TerryE]

@szmodz, I've been somewhat sidetracked by building a new house. It looks like we've now got a completion date before Christmas for the sale of our old one, so everything is panic mode at the mo.

This work is on hold until I am Inthe new house. My next priority is to get the Lua Flash Store mod into Alpha test.which I am still trying to fit in before the move.

[TerryE]

Gosh. Time passes and we haven't done this. I did do this are part of my abandoned Lua 5.3 port. I have debated the pros and cons of the putative upgrade to 5.3 myself. My main concern is that 5.3 actually buys us very little in features or performance over our current 5.1 implementation, but will introduce some backwards compatibility issues that might break stable existing applications.

If we don't adopt 5.3 then another alternative is to move the existing 5.1 code base to a common code set that will work unmodified on both the ESP8266 and ESP32 platform variants. That we we can move the various enhancements that I have implemented for the ESP8266 onto the ESP32. Thoughts?

[jmattsson]

My view is that we/the project will be better off long-term going to 5.3. Clinging to old versions often cause more pain in the end than if you were to have gotten it over and done with early. Besides, those of us who want to stick to the "tried and true" also tend to not upgrade the NodeMCU base often either, so for major upgrades like this I feel some lack of backwards compatibility is reasonable. There have also been a number of bug fixes since 5.1.

And, many of the user-visible changes we could provide shims for during a transition period if we so wish. That's my AUD$0.02 :)

[TerryE]

If we do want to have a migration path to Lua 5.3 then we need to sort out incompatibilities in our module APIs and Lua C API. Perhaps the biggest divergence is the eLua implementation for ROTables. Here a table variable wither points to a Table record in RAM or a ROTable vector in Flash. The latter vector implementation involves serially scanning the vector and this is inefficient, doubly so when the vector is in Flash and cold accesses are at least 20 times slower than RAM ones.

PR #2505 gave a big performance improvement for table-hit accesses but table miss accesses still require a full table scan. I will be raising a new issue covering the implementation and migration to a Lua 5.3 compatible RO Table implementation. I see this as an important step in preparing for any Lua 5.3 implementation.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TerryE]

Still valid

[pjsg]

Now that we have the LUA5.3 migration under our belts, I think that it is time to revisit the ESP32 situation. My reading of the tea leaves is that the ESP32-C3 is going to replace the ESP8266 in the very low end devices. It is just a more modern part with much more ram and better I/O.

I have the ESP32 branch now up and running on this new part (at least the file system is up and running as are several other modules). The network doesn't build, but the wifi module seems to do some of the right things.

However, the ESP32 code base has diverged significantly from the ESP8266 code base. We need to have a plan to migrate to a common source tree with stubs/build options so that we can build for the existing non-RTOS environment and for the RTOS environment for the ESP32 chips. One issue is that the ESP-IDF V4 uses cmake to build and the whole system has a fairly rigid file layout. I don't think that it would be that difficult to use cmake for the ESP32 chips and regular make for the ESP8266.

The ESP32C3 comes up with 230k of heap to start with and I haven't done anything to try and improve that number!

Thoughts on a path forwards?

[jmattsson]

@pjsg I did draw up a plan for the merging of the 8266 and 32 branches quite some time ago; please see internal discussion thread 9 - "Proposal for way towards merging the 8266/32 branches". Unfortunately, between my lack of time and the major 5.3 work that I didn't want to interfere with, only step 1 has been executed on. And now I don't know whether that would be compatible with the 4.0 IDF on the esp32/C3 side?

[pjsg]

@jmattsson I can't actually find that thread -- I can find https://github.com/nodemcu/nodemcu-firmware/issues/1319 

The principal differences between the esp32-c3 and the regular flavor are: 

• single core
• risc-v rather than tensilica

The ESP-IDF4:

• cmake based

Also, now are strongly discouraged from talking to devices (e.g. the uarts) directly -- should go through the RTOS. 

[pjsg]

The changes to make it build under the IDF v4 are pretty small -- though there is one thing that I don't know how to do -- I had to patch an IDF file to make it all work (the ld control file).

The good news is that the Espressif people are being very responsive with esp32c3 issues -- e.g. https://github.com/espressif/openocd-esp32/issues/144 and https://github.com/espressif/esp-idf/issues/6561