update espfs, libesphttpd, use CONFIG_ESPFS_PRESENT, rename html->espfs

[jkent]

This PR is a mess, sorry about that, I had to do a merge because I was working with an old working copy. I've renamed the html directory to espfs so it makes sense alongside of other directories like spiffs. Espfs no longer has a binary mkespfsimage, its been rewritten as a python script. This makes it possible to use cmake without mingw32. Espfs now uses a couple of yaml files to drive the use of processing tools, with custom tools possible. Yaml is optional in project, no preprocessing of files will happen in that case. Node.js tools are handled better.

For makefile under windows, need to document still:

install node.js, choose to put it in the path edit msys2_shell and uncomment: set MSYS2_PATH_TYPE=inherit

[jkent]

Oh yeah, I forgot to mention. Espfs is going to get a new data format after a very brief exchange with Sprite_tm. It is going to get a sorted hash table for lookups to limit the walking and strcmp operations (unless there is a hash collision), which should help with caching. If anyone (@phatpaul @chmorgan) cares to discuss this, I'd love it.

[phatpaul]

Nice! Thanks for working on improving the build, it looks like this should help make it easy to build on more platforms.

I'm going to need a little time to test this, as I'm currently busy with development on a different project. But I'll be coming back to my ESP32 project soon (2~3 mo) for a new feature release.

In the mean-time, anyone else using this please feel free to test and comment.

The hash table seems like a good idea. I assume the hash table will be created at compile-time. But what is the CPU cost of calculating a hash (to then lookup in the table) vs. just walking the fs and comparing strings? I'm also concerned about testing it thoroughly. Espfs is currently so simple and reliable. What kind of testing do you think is needed for the new format? (some kind of fuzzing test?)

[jkent]

The hash table is generated at compile time, yes, it should be very quick, as the hash function is hash = hash << 5 + hash + c # hash = hash * 33 + c The table is a simple sorted list of a 32 bit hash and a 32 bit offset.

I've already completed the updated mkespfsimage.py, which you can see here if you wish: https://gist.github.com/jkent/e345a46f07c744b1492f6367bc1f30e6

[jkent]

The algorithm in C would use a binary search through a sorted hash list. Of course we need to handle the case of hash collisions, but that is easy enough with a sorted list.

It should be faster than strcmp, especially if there are many files in a directory, which would create similar filename prefixes. Also the esp32 data cache should have more hits. Optionally the hash table could be stored in ram upon init,

[jkent]

Okay, I've implemented it. Now I just need a hash collision to test that. If someone can look it over along with this PR, that would be fantastic. https://github.com/jkent/esp32-espfs/tree/wip

I even wrote some documentation!

[phatpaul]

Sorry this took me so long. I had a few hiccups on building on windows legacy make w/ IDF 3.3.5. See my PR https://github.com/jkent/esphttpd-freertos/pull/2 which fixes those.

[phatpaul]

I see you updated submodule espfs from bdbe15 to b02447. Somehow that submodule didn't get updated when I pulled and tested earlier. Since upgrading to b02447 that broke my build. (again, I'm on windows7 legacy make w/ IDF 3.3.5)

error
    ERROR: Command errored out with exit status 1:
     command: C:/msys32/mingw32/bin/python.exe -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/labview/AppData/Local/Temp/pip-install-bur8gpvm/heatshrink2/setup.py'"'"'; __file__='"'"'C:/Users/labview/AppData/Local/Temp/pip-install-bur8gpvm/heatshrink2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record C:/Users/labview/AppData/Local/Temp/pip-record-mnc18g44/install-record.txt --single-version-externally-managed --compile --install-headers C:/msys32/mingw32/include/python3.8/heatshrink2
         cwd: C:/Users/labview/AppData/Local/Temp/pip-install-bur8gpvm/heatshrink2/
    Complete output (21 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.mingw-3.8
    creating build/lib.mingw-3.8/heatshrink2
    copying heatshrink2/streams.py -> build/lib.mingw-3.8/heatshrink2
    copying heatshrink2/version.py -> build/lib.mingw-3.8/heatshrink2
    copying heatshrink2/__init__.py -> build/lib.mingw-3.8/heatshrink2
    copying heatshrink2/__main__.py -> build/lib.mingw-3.8/heatshrink2
    running build_ext
    building 'heatshrink2.core' extension
    creating build/temp.mingw-3.8
    creating build/temp.mingw-3.8/heatshrink2
    creating build/temp.mingw-3.8/heatshrink2/_heatshrink
    C:\msys32\opt\xtensa-esp32-elf\bin/xtensa-esp32-elf-gcc.exe -Wno-unused-result -Wsign-compare -DNDEBUG -march=i686 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -march=i686 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -D_WIN32_WINNT=0x0601 -DNDEBUG -std=gnu99 -Os -ggdb -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -mlongcalls -nostdlib -fstack-protector -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-but-set-variable -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-old-style-declaration -mfix-esp32-psram-cache-issue -DESPFS_HEATSHRINK -DESP_PLATFORM -D IDF_VER=\"v3.3.5\" -MMD -MP -DGCC_NOT_5_2_0=0 -DWITH_POSIX -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DHAVE_CONFIG_H -DUNITY_INCLUDE_CONFIG_H -Iheatshrink2/_heatshrink -IC:/msys32/mingw32/include/python3.8 -c heatshrink2/core.c -o build/temp.mingw-3.8/heatshrink2/core.o
    xtensa-esp32-elf-gcc: error: unrecognized command line option '-march=i686'
    xtensa-esp32-elf-gcc: error: unrecognized command line option '-mtune=generic'
    xtensa-esp32-elf-gcc: error: unrecognized command line option '-march=i686'
    xtensa-esp32-elf-gcc: error: unrecognized command line option '-mtune=generic'
    error: command 'C:\\msys32\\opt\\xtensa-esp32-elf\\bin/xtensa-esp32-elf-gcc.exe' failed with exit status 1
    ----------------------------------------
CC build/main/FreeRTOS-openocd.o
CC build/main/io.o
ERROR: Command errored out with exit status 1: C:/msys32/mingw32/bin/python.exe -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:/Users/labview/AppData/Local/Temp/pip-install-bur8gpvm/heatshrink2/setup.py'"'"'; __file__='"'"'C:/Users/labview/AppData/Local/Temp/pip-install-bur8gpvm/heatshrink2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record C:/Users/labview/AppData/Local/Temp/pip-record-mnc18g44/install-record.txt --single-version-externally-managed --compile --install-headers C:/msys32/mingw32/include/python3.8/heatshrink2 Check the logs for full command output.
CC build/mqtt/esp-mqtt/mqtt_client.o
make[1]: *** [/home/labview/lumitec-dev-fw-plibox-esp32/esp-eth-libesphttpd/components/espfs/component.mk:18: requirements.stamp] Error 1
make: *** [/home/labview/lumitec-dev-fw-plibox-esp32/Tools/esp-idf/make/project.mk:552: component-espfs-build] Error 2
make: *** Waiting for unfinished jobs....

Any ideas? Should I try a later version of the submodule?

[jkent]

I forgot about this PR being open. Yes, I would try using tag v1.1.0, as it is binary compatible. Failing that, I'd try 2742c05.

[jkent]

Oh, I looked at the log closer. It looks like the heatshrink2 python package is not building. It needs to build natively. pip should do that, but you may need Visual Studio present on Windows, or GCC under mingw32. I see its trying to use the xtensa toolchain to do that which is incorrect. Using a native python instead of mingw32 python may fix this as well.

[phatpaul]

included in #37