Add support for ESP32 - Githubissues

marcelstoer commented 5 years ago

Over at the firmware project we have been seeing an influx of features for ESP32 in the recent weeks and months: https://github.com/nodemcu/nodemcu-firmware/pulls?q=is%3Apr+is%3Aopen+label%3AESP32 New contributors are providing fixes and new modules for that branch.

It is definitely high time we get ESP32 support for this Docker image. I took a few steps into that direction back in March but due to lack of time it was abandoned.

In theory it all sounds fairly straightforward:

one big if/else in build that detects the branch i.e. platform ESP32 vs ESP8266
invoke the build just like the CI script does https://github.com/nodemcu/nodemcu-firmware/blob/dev-esp32/.travis.yml
use srec_cat to bundle the three binaries
- 0x1000 bootloader.bin
- 0x8000 partitions_singleapp.bin
- 0x10000 NodeMCU.bin into a single binary that can be flashed to 0x00000 (padding the empty spaces with 0xFF just like for ESP8266)

marcelstoer commented 5 years ago

@devsaurus the "specification" for srec_cat is correct, isn't it?

devsaurus commented 5 years ago

Just tried to manually build and flash an image accordingly:

srec_cat -output image.bin -binary -generate 0x0000 0x1000 -constant 0xff build/bootloader/bootloader.bin -binary -offset 0x1000 -fill 0xff 0x1000 0x8000 build/partitions-2MB.bin -binary -offset 0x8000 -fill 0xff 0x8000 0x10000 build/NodeMCU.bin -binary -offset 0x10000
python sdk/esp32-esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode qio --flash_freq 40m --flash_size detect 0x0000 image.bin

I usually just do a make flash and don't care for the mechanics in the background, but it seems to work this way as well.

devsaurus commented 5 years ago

There's a pitfall with the default partition table, however. Recent additions like http or sodium bump the plain firmware beyond 1 MB, so it won't fit into the default partition table. One has to supply a custom partition table which allows for a larger firmware. E.g.

# Espressif ESP32 Partition Table
# Name,  Type, SubType, Offset,  Size
nvs,      data, nvs,     0x9000,  0x6000
phy_init, data, phy,     0xf000,  0x1000
factory,  app,  factory, 0x10000, 2M
# 0xC2 => NodeMCU, 0x0 => Spiffs
nodespiffs,  0xC2, 0x0,            , 960K

I get the impression that we should provide such partitioning in the branch and probably make it the default for menuconfig.

marcelstoer commented 5 years ago

Thanks! You went well beyond just confirming my assumptions 😄

That srec_cat command is quite a beast, isn't it.

Can't -generate -constant 0xff be simplified to -fill 0xff? That's what we do for the ESP8266.
Could we simplify the whole command by filling from 0x0000 to 0x10000 first and then placing the three .bins at their respective offset? Update: srec_cat doesn't like that: contradictory 0x00008000 value (previous = 0xFF, this one = 0xAA)

I usually just do a make flash and don't care for the mechanics in the background

If you've got the IDF that's fine. However, to cater for our largest user group a single binary for flashing to 0x0000 is essential IMO.

we should provide such partitioning in the branch and probably make it the default for menuconfig

👍

devsaurus commented 5 years ago

Can't -generate -constant 0xff be simplified to -fill 0xff?

That was my first try as well, but srec_cat's semantics don't allow this. -fill is a filter for an input, while -generate is an input on its own (i.e. it's equivalent to NodeMCU.bin for example). With ESP32 we need to generate a chunk of data out of nowhere while for ESP8266 we start with a real bin file at offset 0x0000.

Could we simplify the whole command by filling from 0x0000 to 0x10000 first and then placing the three .bins at their respective offset?

Would be more intuitive, yes. But I have no clue how to specify this logic as a srec_cat command line :smile:

BTW srec_cat allows parentheses for grouping the options. You have to escape them for the shell, though.

srec_cat -output image.bin -binary '(' -generate 0x0000 0x1000 -constant 0xff ')' '(' build/bootloader/bootloader.bin -binary -offset 0x1000 -fill 0xff 0x1000 0x8000 ')' '(' build/partitions-2MB.bin -binary -offset 0x8000 -fill 0xff 0x8000 0x10000 ')' '(' build/NodeMCU.bin -binary -offset 0x10000 ')'

marcelstoer commented 5 years ago

A huge leap forward (for me): https://github.com/marcelstoer/docker-nodemcu-build/compare/feat/esp32

Dockerfile declares the new dependencies for ESP32: https://github.com/marcelstoer/docker-nodemcu-build/compare/feat/esp32#diff-3254677a7917c6c01f55212f86c57fbfR15
the existing ESP8266 build script was moved: build -> build-esp8266
build now houses the big platform switch: https://github.com/marcelstoer/docker-nodemcu-build/blob/c4df033822c878144431c70b966d3e61ccf24dba/build
first attempt for the ESP32 build script: https://github.com/marcelstoer/docker-nodemcu-build/blob/c4df033822c878144431c70b966d3e61ccf24dba/build-esp32

Status It seems to work. I get a Lua prompt after flashing the created combined binary:

NodeMCU ESP32 build 20181123 powered by Lua 5.1.4
lua: cannot open init.lua
>

The build date there in the prompt is "wrong" i.e. not the one I set as per https://github.com/marcelstoer/docker-nodemcu-build/blob/c4df033822c878144431c70b966d3e61ccf24dba/build-esp32#L26. Not sure if this MORE_CFLAGS="-DBUILD_DATE thing is even supported. I copied it from the firmware Travis file.

Besides, the prompt is currently missing all the extra information you get when building for the ESP8266. I need to figure out how to add that:

NodeMCU 2.2.0.0 built with Docker provided by frightanic.com
    branch: master
    commit: 67027c0d05f7e8d1b97104e05a3715f6ebc8d07f
    SSL: false
    modules: adc,bit,dht,file,gpio,i2c,mqtt,net,node,ow,spi,tcs34725,tls,tmr,uart,wifi
 build created on 2018-04-08 04:13
 powered by Lua 5.1.4 on SDK 2.2.1(cfd48f3)
lua: cannot open init.lua
>

Looking for reviewers and testers!

marcelstoer commented 5 years ago

Another night session and things look very good

NodeMCU ESP32 built with Docker provided by frightanic.com
    branch: dev-esp32
    commit: 9bd8df469326bc6613ef4fbb7b87a5da862b29e1
    SSL: true
    modules: file,gpio,http,i2c,net,node,ow,sjson,sodium,tmr,wifi
 build created on 2018-11-25 08:21
 powered by Lua 5.1.4 on ESP-IDF v3.2-dev-1239-g7313e39
lua: cannot open init.lua
> =node.heap()
235052

@devsaurus / @HHHartmann can I invite you to take a look at https://github.com/marcelstoer/docker-nodemcu-build/compare/feat/esp32?

devsaurus commented 5 years ago

Checking sdkconfig is problematic. It doesn't exist yet on a fresh checkout of the dev-esp32 branch. It's not clear to me how/when the make defconfig or make menuconfig is actually triggered. Is the user supposed to do this upfront? How would he do that with docker?

marcelstoer commented 5 years ago

It's not clear to me how/when the make defconfig or make menuconfig is actually triggered.

That's indeed the essential question here, spot on. I'm not sure what the most elegant and intuitive approach is. It's clear that those operations have to be run within the container. What you would do now is a follows.

First build

Instead of running the build command in the container you start the shell (bash or /bin/bash instead of build).
You get a prompt at the mounted /opt/nodemcu-firmware inside the container with full access to the whole Ubuntu.
You run make menuconfig.
Then you either exit the container and run it again with build or you stay inside and run build from there.

Subsequent builds

You manipulate firmware sources and your sdkconfig outside the container in the host OS. Then you start the container to build.

OR

You work inside the container all/most of the time.

Do you see a way to make this more convenient?

devsaurus commented 5 years ago

You manipulate ... your sdkconfig outside the container in the host OS

Unless such manipulation is done by a native menuconfig in host OS, I'd discourage to establish this as a regular method. Mainly because of the dependencies which are modeled in Kconfigs.

Do you see a way to make this more convenient?

Well, my preference would be to just start a bash inside the container and work there like I'm used to on my Linux box. But that's probably not representative.

The actual use case are in fact quite limited:

run make menuconfig once in a while
run make to check edits
run make flash to test modifications

Which boils down to two standardized docker calls for "config" and "build". I could imagine that users are ok to first run a "config" docker call and then "build" calls. This is driven by the use model of the sdk with menuconfig. IMO there's no benefit in hiding this from the docker users. If I was a Windows user who wants to get serious with NodeMCU coding, I'd set up config_esp32.bat and build_esp32.bat scripts to wrap the complex docker commands and execute them from a CMD window.

marcelstoer commented 5 years ago

Unless such manipulation is done by a native menuconfig in host OS, I'd discourage to establish this as a regular method.

Fully agree.

Well, my preference would be to just start a bash inside the container

That's fine, there's nothing that would stop you from doing that.

Which boils down to two standardized docker calls for "config" and "build". I could imagine that users are ok to first run a "config" docker call and then "build" calls.

Right, offering that config command is a short shortcut to bash -> make menuconfig but a shortcut nonetheless. It's an improvement for anyone scripting the Docker run commands as you describe (most users I guess).

I will also check for the existence of a sdkconfig file early in build-esp32 to have the process fail fast - and with a meaningful error message.

HHHartmann commented 5 years ago

Thinking through this I think that the automatic switch for ESP8266/ESP32 is not suficcient. As a developer I want to build my own branch != dev-esp32, which is not possible here. The Doc says "It uses Git to figure out whether you checked out NodeMCU for ESP32 or ESP8266." Maybe it is possible to use the existence of a file or directory to decide which build to call.

And sorry for being late. Was a busy weekend.

marcelstoer commented 5 years ago

True that...sigh. You could always call build-esp32 but if there is an automatic switch you expect it to work reliably. I could easily check for sdkconfig.defaults to be there.

marcelstoer commented 5 years ago

@HHHartmann fixed with 19674cd

marcelstoer / docker-nodemcu-build

Add support for ESP32 #55