luac.cross tidy up post SDK 3.0

[TerryE]

Suggested Changes

• The -a is now redundant since the nodemcu partitioning / provisioning tool uses the standard -f relocatable format. Anyone who wishes to have their own standalone relocator can take the code from nodemcu-partition.py. I suggest that strip out the obsolete code.

• The -f option uses uzlib to compress the image. This format is a subset of the gz format. In particular the dictionary size is limited to 16K rather than the standard 32K. This was because (with SDK 2.x) there wasn't enough RAM to use a 32K dictionary on the ESP8266. With the extra RAM available under SDK 3.0, we can now relax this limit back to 32K and so we will be compliant with the gz format.

• Whilst we are doing these changes, it makes sense to include the resource generator option as proposed by Gregor Hartmann if enabled by a user_config.h runtime switch (as I have discussed and agreed offline).

Note that the compressor uses the gz static dictionary which is not as efficient as a dynamic dictionary. If image size is at a premium upping the limit to 32K will allow repacking of the image: zcat lfs.img | gzip -c9 > lfs1.img and the repacked image will typically be 10-15% smaller than the original.

None of these items is particularly high priority so I will poll for feedback before creating the PR.

Supersedes

2691

[HHHartmann]

@TerryE Terry would it make sense to have a switch for backward compatibility with pre 3.0 firmware using only the 16K dictionary size? Will the firmware still be able to decompress sich images then or will you remove the 16K code completely. (Our will it work anyways)

[TerryE]

The 16K vs 32K is just a constant in the code. We could add a switch to expose this and then the same luac could do both or we don't bother and just update the ESP-side lflash.c to tolerate 32K dictionaries. This is just a constant change in the code. IIRC, going from a 16K to 32K dictionary with an LFS image improves compression less than 1%. The big saving is re-encoding using an optimised dictionary. With this constant change the lflash.c inflate routine would support 16K static, 32K static and the gz standard 32K dynamic.

[HHHartmann]

@TerryE Terry, as zcat seems to handle the 16K version already it does not seem to make sense to have luac.cross generate 32K variants. On the other hand it would make sense to have it generate 16K variants for backwards compatibility. So whoever wants (and has SDK 3.0 running on the ESP) can repack it and for everybody else everything works unchanged.

[TerryE]

* The `-a` is now redundant since the nodemcu partitioning / provisioning tool uses the standard `-f` relocatable format. Anyone who wishes to have their own standalone relocator can take the code from **`nodemcu-partition.py`**.  I suggest that strip out the obsolete code.

I think that we are going to have to stick with a 16Kb dictionary for LFS images owing to the RAM limitations of the ESP8266 post SDK 3.0. Also @jmattsson wants to retain the -a option to facilitate signed OTA ESP32 builds, so I will be reimplementingt this in future versions of luac.cross. Other than that, we've covered everything that I need to in this issues and will close it.