Reducing startup timing (dcc module)

[pjsg]

I'm using the dcc module (from @vsky279 ) and I'm having problems getting the nodemcu platform to boot fast enough (I think) to support service mode writes. Does it work for you?

The issue is that there is only a limited amount of time between when the service mode programmer powers up the rails before the write commands are issued and the rails powered down. [Reads work fine (mostly)]

I'm intending to try setting the cpu speed high in the user_init function, getting rid of the startup banner, and using LFS (as I suspect that is faster than loading a .lc file). With lua53, the gc mode is already set sensibly. Are there other speedups that anybody can think of?

[vsky279]

Unfortunately I don't have the experience. I use dcc module in an application for my friend so I did not have real opportunity to play with the whole dcc universe.

[nwf]

Nerfing SPIFFS (that is, setting the partition size to 0), if you can for your application, will also speed boot-up.

Is there no equivalent of "phantom power" in model rail networks to keep the ESP8266 powered between commands?

[pjsg]

There are (at least) two environments in a model railway. There is the operational track which always has power applied (sometimes quite a lot of power). The signalling is modulated on top of the power.

The service mode track (aka programming track) is only powered up when you want to program a device. This track also provides less power -- it appears that it is current limited at maybe only 200mA. The timing of all of this seems not well defined by the standard....

I'm trying to figure out really how long I have to bring up the application code far enough to be able to configure things. If you don't know, the configuration for a decoder is a few bytes up to a few hundred bytes. Each byte (called a configuration variable or CV) has an index (1 - 1023) and so programming a device involves writing a value to a particular CV. There are ways to read the CVs back, but the timing is fairly critical.

[pjsg]

I wonder if I could delay the initialization of SPIFFs until I need it.... I wonder if could be delayed from luaopen_file until maybe a file.startup() call.

[TerryE]

My current PR that I'm working on has a check on the size of the SPIFFS partition, and skips the mount if it's 0. You need to have a startup command pointing into LFS, of course. The other option is to make the FS as small as you can, because the time is size-dependent. The only other option would be to have an RCR that defers mount and allow a file.mount() command for SPIFFS.

[TerryE]

As far as LFS load is concerned, there is no latency as the Lua compiled code is already in addressable RAM.

[pjsg]

Before making any changes with a lua51 build, it took 240 ms to go from reset going high to when an init.lua drives a pin low.

All below are lua53 dev branch with changes 234ms for the lua53 build.

224ms for switching to 160MHz in user_pre_init

204ms for disabling the banner

194ms for both disabling the banner and enabling 160MHz

What is frustrating is that My pin toggling isn't working early on in the boot cycle. I think that I have the pins initialized correctly.... I was trying to reduce the rf calibration time, but I just can't make it have any effect.....

[TerryE]

@pjsg Philip, I think what we are really talking about here is how we minimise startup times for post deep sleep startup. I suggest that you put some instrumentation around the SPIFFS mount because I suspect that's the biggest component here. Having an investigation into options for getting this down might be a good start as would be dumping the use of init.lua altogether and using an init module in LFS and (if you are using Lua53) having a startup command pcall(function() LFS.init() end) which you can set in user_config.h in the LUA_INIT_STRING define or using node.startupcommand()

If this string is a "@file.lc" then this will load and run a binary LC, which is maybe 4× faster than compiling a .lua file. Maybe we should have a "!module" supported as well to do an LFS module as this will also avoid both SPIFFS access and any compilation.

Of course if you have your ESP powered, then why just not leave it in node.sleep() as the wake times are faster especially if you have preserve_mode=false and do the autoconnect off the critical path.

And another trick is that the .lua extension is a convention. The loader actually uses a file header to determine the type: source or binary so you can compile a SPIFFS init file and remain it to init.lua and startup will load this binary without compiling.

[pjsg]

I now have a set of options to enable fast application startup. These are individually controllable through a set of flags stored in an RCR. The options are:

• Disable as much banner output as possible.
• Switch to 160MHz as early as possible
• Delay mounting the SPIFFS partition until the first use of any vfs function

Together with LFS, I can now boot the platform, use LFS.init() as the startup command, and that calls LFS.mycode() which basically invokes the dcc setup -- and that all completes in around 100ms. This is a significant improvement over what I was seeing before.

I suspect that this code will be useful for other people, so I'm intending to put up a PR for the fast-start code. As an option there is code that measures the startup time by phase so that you can see where the time is spent.

Any thoughts before I rip the code out of my branch and wrote the docs?

[TerryE]

• Delay mounting the SPIFFS partition until the first use of any vfs function

I was brooding about this option myself. Using an RCR config option to do this sounds like the best way to go.

Ditto for disabling the boot message.

[TerryE]

@pjsg @nwf @HHHartmann @marcelstoer

I was brooding about this option myself. Using an RCR config option to do this sounds like the best way to go.

At the moment we have some configuration parameters in app/include/user_config.h which really makes them build options, when there are some strong arguments that it would make more sense to make them runtime (startup) options -- that is configured through RCR settings:

• These options might be Lua version specific which is confusing if they are in an app/include file and outside the app/lua or app/lua53 tree.
• Having them as a build option either complicates Cloud builder builds or means that developers using these can only have the default options
• Build options are just that: changing them requires a rebuild and reflash rather than a restart
• There is often no material code or RAM size or runtime impacts of making them boot options rather than build options.

So my thinking is that we should apply a sanity test here: any options that materially impact the size of the firmware (e.g. enabling SSL) we leave as config options, but we consider extending use of the RCR system for those options where it makes sense to do so. We need to wrap this up in a couple of node access functions, but being able to do something like:

  node.setRCR({lazySPIFFSmount=true})

seems a lot easier than having to edit app/include/user_config.h, rebuild then reflash the firmware.

If there is support then I'll raise an Issue for this.

[pjsg]

MY current approach is to add a single function:

node.startupoptions(val)

Where val is one or more of

OPTION_NO_BANNER        
OPTION_160MHZ           
OPTION_DELAY_MOUNT      

These are added (or bit or'ed) together.

[HHHartmann]

IMHO an interface to enable or disable the options individually would be better.

node.startupoption(val, bool enable)

returning the current configuration. If enable is nil only the current value is returned.

or

node.setstartupoption(val)
node.resetstartupoption(val)

We should also think about moving

adc.force_init_mode()

to this new interface.

[nwf]

IMHO an interface to enable or disable the options individually would be better.

node.startupoption(val, bool enable)

returning the current configuration.

How about a table, a la wifi.sta.config and friends? If a key's present, we update the value, if not, we leave it as is, and if the table's nil we return the current configuration (as a table suitable to be passed back in).

We should also think about moving

adc.force_init_mode()

to this new interface.

:+1:

[nwf]

Having just tripped over this myself, incidentally, you might check that your modules are both capable of and using qio flash mode. Apparently some of my qio capable boards had gotten stuck in dio mode, which can't have been doing anyone any favors.

[TerryE]

IMHO an interface to enable or disable the options individually would be better.

node.startupoption(val, bool enable)

That's why I think that it would be better to have node.startupoption(array) so that you can set multiple options in one statement and use the {} placeholder to delete options. This would also subsume the existing node.startupcommand() option. The special case node.startupoption() could return the array of existing options that have been set.

[TerryE]

I mostly use ESP12 compatible boards (Wemos D1 Mini's) and I though that these used the extra 2 pins as IOs, so only supported dio. However, looking at the schematic, I see that physical pins 19-23 are use to connect to the W25Q32FVSS flash, so it can run in QIO mode. Need to play with this. Another job to do.

[TerryE]

@pjsg you might also think of removing the else clause at app/user/user_main.c:144. This part of the normal boot diagnostics at 74,880 baud and there is little point in sending this to the UART.

In my current tweaks, I've add the the ability to use something like "!_init" as the startup command. This will execute LFS._init() directly without doing a compile whereas using "node.flashindex'_init'()" effectively does a executes loadstring("node.flashindex'_init'()")() which does invoke the compiler.

[pjsg]

If you turn off the banner, then you don't get any output from that function.

I like the !init approach -- but I'll defer that change to your PR....

[pjsg]

With regard to the interface to enable/disable features, my feeling is that this is an API that is used very infrequently -- simplicity is key. Certainly, typing node.startupoption(7) is easy in esplorer!

[TerryE]

Certainly, typing node.startupoption(7) is easy in esplorer!

Yup, but if we were to add more options, then doing this means that we will need need more and more access methods. We've already got node.startupcommand() so this makes 2 + ...

[pjsg]

How about

node.startupoption([table])

It always returns the current set of options (after any modification).

if table is missing, then it returns the current set of options. if table is {} then all options are set to their default state Otherwise, each item in the table is treated as an option. The options are listed below with their default values.

• command="@init.lua"
• mhz=80
• banner=true
• delay_mount=false

Does this make sense?

[nwf]

How about

node.startupoption([table])

It always returns the current set of options (after any modification).

That's a little unusual just in that it's perhaps a more common idiom to return the old values. But I see no harm in doing either, and I guess the old values are not really all that useful in this case.

if table is missing, then it returns the current set of options. if table is {} then all options are set to their default state Otherwise, each item in the table is treated as an option. The options are listed below with their default values.

• command="@init.lua"
• mhz=80
• banner=true
• delay_mount=false

:+1:

As a follow up, we can add @HHHartmann's suggestion, perhaps as adc_init = [boolean].

[HHHartmann]

I'm not so sure about the adc stuff anymore because technically it is similar but it isn't exactly a startup option as it does not affect strap. But if we include or here lease use something like adc.INIT_ADC or adc.INIT_VDD33 and not a boolean.

The other options might also be more speaking like show_banner. cpufreq using the labels 'node.CPU80MHZ' or 'node.CPU160MHZ' Also could name it node.startup_options

[TerryE]

Also could name it node.startup_options

Why not just node.startup()? So

node.startup{command = '!_init',
             boot_options = node.NO_BANNER + node.160MHZ + node.DELAY_MOUNT}

etc.?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.