Support for IDF 5

[marcelstoer]

Ideally we can support both IDF 3 and 4 (i.e. the NodeMCU branches dev-esp32-idf3-final and dev-esp32).

NodeMCU firmware dev-esp32 is now on IDF ~4~ 5 (as per https://github.com/nodemcu/nodemcu-firmware/commit/6798f027f3eecb2fdda4e6544c5586cdbe8802d0 on that branch).

1. https://github.com/marcelstoer/docker-nodemcu-build/blob/master/configure-esp32 does likely not need any changes.
2. https://github.com/marcelstoer/docker-nodemcu-build/blob/master/build do we still need to check for sdkconfig.defaults?
3. Adjust the build script https://github.com/marcelstoer/docker-nodemcu-build/blob/master/build-esp32

[jmattsson]

Considering the current version of IDF is 5.0, supporting IDF 3 here is probably not worth the effort?

[marcelstoer]

True, a year ago it might have been. I'll rephrase the task.

[jmattsson]

Some problems do go away when you procrastinate enough 🤣

[marcelstoer]

I haven't been using NodeMCU for quite some time. Due to the comments over at #104 I spent a little time trying to make this run - knowing that my outdated NodeMCU/ESP-IDF know-how will probably stop me half-way.

So, there's now a feat/idf-4 branch and a marcelstoer/nodemcu-build:esp32 image (<- note the tag!) with my WIP.

docker run --rm -ti -v $(pwd):/opt/nodemcu-firmware marcelstoer/nodemcu-build:esp32 configure-esp32 almost succeeds but it eventually crashes with

-- Project sdkconfig file /opt/nodemcu-firmware/sdkconfig
Traceback (most recent call last):
  File "/opt/nodemcu-firmware/sdk/esp32-esp-idf/tools/kconfig_new/confgen.py", line 33, in <module>
    import gen_kconfig_doc
  File "/opt/nodemcu-firmware/sdk/esp32-esp-idf/tools/kconfig_new/gen_kconfig_doc.py", line 27, in <module>
    import kconfiglib
ImportError: bad magic number in 'kconfiglib': b'\x03\xf3\r\n'
CMake Error at sdk/esp32-esp-idf/tools/cmake/kconfig.cmake:266 (message):
  Failed to run confgen.py
  (/root/.espressif/python_env/idf4.4_py3.8_env/bin/python;/opt/nodemcu-firmware/sdk/esp32-esp-idf/tools/kconfig_new/confgen.py;--kconfig;/opt/nodemcu-firmware/sdk/esp32-esp-idf/Kconfig;--sdkconfig-rename;/opt/nodemcu-firmware/sdk/esp32-esp-idf/sdkconfig.rename;--config;/opt/nodemcu-firmware/sdkconfig;--defaults;/opt/nodemcu-firmware/sdkconfig.defaults;--env-file;/opt/nodemcu-firmware/build/config.env).
  Error 1
Call Stack (most recent call first):
  sdk/esp32-esp-idf/tools/cmake/build.cmake:523 (__kconfig_generate_config)
  sdk/esp32-esp-idf/tools/cmake/project.cmake:384 (idf_build_process)
  CMakeLists.txt:10 (project)

-- Configuring incomplete, errors occurred!
See also "/opt/nodemcu-firmware/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1
make: *** [Makefile:21: menuconfig] Error 2

However, what's worse IMO is that I feel the whole approach of using a Docker image that way may no longer be feasible with IDF 4. We now have to call the IDF install.sh which installs loads of stuff.

On a standalone Linux installation that may be ok because you only do this once and whenever the IDF version changes. Doing this every time you create a Docker container isn't viable. We can't pre-load the Docker image with the required packages as they depend on the IDF version which is driven by the user environment rather than the build image. An obvious remedy might be to keep the container running once everything has been initialized (i.e. removing --rm from docker run). However, that doesn't really create a stable setup. An alternative might be to run install and then "Docker commit" the local changes.

Anyway, feel free to drive this forward by engaging here and - more importantly - by creating PRs for that feat/idf-4 branch.

[jmattsson]

Is it feasible to pre-load the docker image with the IDF required bits? As in, effectively run the ./install.sh for each NodeMCU/IDF version the docker image is meant to support?

That said, there was some incompatibility with some IDF 4.x versions used together with IDF 5.x, so on closer thought it might not be a good path to go down.

[ezbella]

Hi @marcelstoer, I've been using esp32 for a while in my projects, using the image provided by your site (https://nodemcu-build.com/). Now, after several months of inactiviy, I decided to resume a project and found out that your site has disabled esp32 dev option. I've tried to understand whats going on by reading these threads, but I can't get a final conclusion. I'm really sad to see that this project is falling apart. Could you please give a little light on what to do, at least on beginners like me, to carry on building the image with the needed libraries, for projects that may appear in the years to come? Or should we try to shift to another frameworks? Thanks in advance. I'm struggling to digest this notice =(

[marcelstoer]

@ezbella There are no news other than what is discussed in the comments above - sad but true.

the image provided by your site (https://nodemcu-build.com/)

Just to rule out any misunderstandings, the Docker image (this project) has nothing to do with the cloud builder. They solve a similar problem but they are technically independent.

Could you please give a little light on what to do, at least on beginners like me

It's impossible for me to tell. If you are not on Linux (I'm neither) and are not familiar with Docker then your options are very limited as far as NodeMCU for ESP32 is concerned.

[ezbella]

Thanks @marcelstoer for your quick reply, as always. I beg you please to clarify some points, to help me, and others in similar situation to decide what road to take.

1- Why is that your cloud builder can't be working with the legacy working version of ESP32 IDF3? It may be outdated but at least the working functionality at that moment could be still used. (Until further compatibility is issued) 2- Considering there is no other alternative than moving to linux to biuld the firmware. The steps in https://nodemcu.readthedocs.io/en/dev-esp32/build/#linux-build-environment are still valid? Or are there intrinsic nodemcu lua technical aspects that are blocking the creation of the images.

If at least, there is still a chance to keep building the framework via Linux, I consider it worth the time to invest on trying. It is so valuable the work you did, and the power of the framework, that I am not willing to let go so easily. Thanks in advance dear Marcel

[marcelstoer]

Why is that your cloud builder can't be working with the legacy working version of ESP32 IDF3?

That's a fair point, I don't remember. I will look into this in the coming days.

The steps in https://nodemcu.readthedocs.io/en/dev-esp32/build/#linux-build-environment are still valid?

I would assume so as the NodeMCU docs are supposed to be updated (if necessary) with every PR that changes code. However, a Linux user would need to confirm this. @jmattsson maybe?

are there intrinsic nodemcu lua technical aspects that are blocking the creation of the images

That's what this issue is all about 😄

[jmattsson]

At a quick read, yes, that still looks correct. I can't swear the IDF hasn't moved bits around in the menuconfig over time, and I haven't got the time right now to do a step-by-step to see that the precisely-as-documented procedure sticks the landing, but it should not be far off. If someone does run into issues you're welcome to tag me in an issue!

[marcelstoer]

Why is that your cloud builder can't be working with the legacy working version of ESP32 IDF3?

That's a fair point, I don't remember. I will look into this in the coming days.

I updated the frozen dev-esp32-idf3-final branch (one last time, I swear) in order to be able to build documentation for it. Then I tweaked the cloud builder to build firmware off that branch. The build passes but I didn't verify the produced binaries yet. Anything else related to the cloud builder should be discussed outside this thread to stay on topic.