Compilation fails on dev-8.0 branch of esp32 port

[hmmkay]

Hi,

I have an ESP32 board and want to compile micropython using Cmake going forward. Thankfully Cmake support was just added in micropython 1.15, so now I'm wanting to add lvgl as a micropython module - which looks like it should be possible per the dev-8.0 branch of lv_micropython. However I'm having some trouble compiling this branch:

Problem 1 Checking out the dev-8.0 branch and directly executing make from the ports/esp32 directory gives a compile error which I believe I've tracked down to the lv_conf.f from commit d7f2aabfe missing a typedef for the lv_coord_t type.

I 'fixed' this by copying over the default lv_conf.h with lv_conf_template.h, which led me to Problem 2 where I'm currently stuck ...

hmmkay@hmmkay ~/repro/ $git clone --recurse-submodules -j8  https://github.com/lvgl/lv_micropython.git
hmmkay@hmmkay ~/repro/lv_micropython (master)$ git checkout dev-8.0

hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ git log --oneline -1
d7f2aabfe (HEAD -> dev-8.0, origin/dev-8.0) Merge master into dev-8.0

hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ cmake --version
cmake version 3.20.3

hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ idf.py --version
ESP-IDF v4.2-dirty

hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ xtensa-esp32-elf-gcc --version
xtensa-esp32-elf-gcc (crosstool-NG esp-2020r3) 8.4.0

hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ make
idf.py -D MICROPY_BOARD=GENERIC -B build-GENERIC  build

...

[ 73%] Generating ../../lv_png.c
[ 74%] Generating ../../lv_espidf.c.pp.filtered
[ 75%] Linking CXX static library libesp_hid.a
[ 75%] Generating ../../lv_espidf.c
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
[ 75%] Linking CXX static library libprotocomm.a
[ 75%] Built target __idf_libsodium
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
[ 75%] Built target __idf_esp_hid
[ 75%] Generating ../../lv_mp.c
GEN /home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC/genhdr/mpversion.h
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
[ 75%] Built target __idf_protocomm
make[3]: Entering directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[3]: Entering directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[3]: Entering directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[3]: Entering directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
[ 75%] Building C object esp-idf/esp_local_ctrl/CMakeFiles/__idf_esp_local_ctrl.dir/src/esp_local_ctrl.c.obj
[ 75%] Building C object esp-idf/esp_local_ctrl/CMakeFiles/__idf_esp_local_ctrl.dir/src/esp_local_ctrl_handler.c.obj
[ 75%] Building C object esp-idf/esp_local_ctrl/CMakeFiles/__idf_esp_local_ctrl.dir/proto-c/esp_local_ctrl.pb-c.c.obj
[ 75%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/wifi_config.c.obj
[ 75%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/manager.c.obj
[ 75%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/handlers.c.obj
[ 75%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/wifi_scan.c.obj
[ 75%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/scheme_softap.c.obj
[ 75%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/scheme_console.c.obj
[ 76%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/proto-c/wifi_constants.pb-c.c.obj
[ 76%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/proto-c/wifi_scan.pb-c.c.obj
[ 76%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/proto-c/wifi_config.pb-c.c.obj
[ 76%] Building C object esp-idf/wifi_provisioning/CMakeFiles/__idf_wifi_provisioning.dir/src/scheme_ble.c.obj
[ 76%] Linking CXX static library libesp_local_ctrl.a
Traceback (most recent call last):
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/gen_mpy.py", line 299, in <module>
    ast = parser.parse(s, filename='<none>')
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/c_parser.py", line 149, in parse
    return self.cparser.parse(
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/c_parser.py", line 1844, in p_error
    self._parse_error(
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/../pycparser/pycparser/plyparser.py", line 67, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: /home/hmmkay/repro/lv_micropython/lib/lv_bindings/lvgl/src/lv_hal/../lv_misc/lv_area.h:40:5: before: lv_coord_t
make[3]: *** [esp-idf/main/CMakeFiles/__idf_main.dir/build.make:75: lv_mp.c] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[2]: *** [CMakeFiles/Makefile2:4329: esp-idf/main/CMakeFiles/__idf_main.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
[ 76%] Linking CXX static library libwifi_provisioning.a
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
[ 76%] Built target __idf_esp_local_ctrl
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
[ 76%] Built target __idf_wifi_provisioning
make[2]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[1]: *** [Makefile:136: all] Error 2
make[1]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make failed with exit code 2
make: *** [Makefile:30: all] Error 2

Problem 2 - I copied the config template lv_conf_template.h over lv_conf.f (noting that the template still talks about being for 7.11.0 whereas lv_conf says its for 8.0), and set #if 1 ... hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ cp ../../lib/lv_bindings/lvgl/lv_conf_template.h ../../lib/lv_bindings/lv_conf.h

... but now I get stuck with a trickier problem that after executing make with the new lv_conf.h file, the gen_mpy.py script can't seem to lookup a key called lv_mem_buf_arr_t. This requires more understanding than I currently have to correct since I don't understand (and honestly don't want to :D) how this script does its magic. I've twiddled some of the memory settings in lv_conf.h in the hope that perhaps changing how memory management is configured that I can move past this error, but no luck so far.

[ 75%] Generating ../../lv_mp.c
Traceback (most recent call last):
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/gen_mpy.py", line 2277, in <module>
    gen_global(global_name, blobs[global_name])
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/gen_mpy.py", line 2241, in gen_global
    wrapped_type = lv_mp_type[global_type]
KeyError: 'lv_mem_buf_arr_t'
make[3]: *** [esp-idf/main/CMakeFiles/__idf_main.dir/build.make:75: lv_mp.c] Error 1
make[3]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[2]: *** [CMakeFiles/Makefile2:4329: esp-idf/main/CMakeFiles/__idf_main.dir/all] Error 2
make[2]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make[1]: *** [Makefile:136: all] Error 2
make[1]: Leaving directory '/home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC'
make failed with exit code 2
make: *** [Makefile:30: all] Error 2

Enabling -v debug for the ESP IDF in the Makefile wrapper gives me the following command as the one that fails when running make:

hmmkay@hmmkay ~/repro/lv_micropython/ports/esp32 (dev-8.0)$ cd /home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC/esp-idf/main && /home/hmmkay/.espressif/python_env/idf4.2_py3.8_env/bin/python3.8 /home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/gen_mpy.py -M lvgl -MP lv -MD /home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC/lv_mp.c.json -E /home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC/lv_mp.c.pp /home/hmmkay/repro/lv_micropython/lib/lv_bindings/lvgl/lvgl.h > /home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC/lv_mp.c || ( rm -f /home/hmmkay/repro/lv_micropython/ports/esp32/build-GENERIC/lv_mp.c && /bin/false )
Traceback (most recent call last):
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/gen_mpy.py", line 2277, in <module>
    gen_global(global_name, blobs[global_name])
  File "/home/hmmkay/repro/lv_micropython/lib/lv_bindings/gen/gen_mpy.py", line 2241, in gen_global
    wrapped_type = lv_mp_type[global_type]
KeyError: 'lv_mem_buf_arr_t'

--

If there's an easier/alternate way to get micropython >+1.14 and lvgvl >+ 7 compiled with CMake I'd be happy to be pointed in that direction. The dev-8.0 branch of lv_micropython looked to be the best path to follow for that.

Any assistance in moving forward would be appreciated! Thanks in advance.

[amirgon]

Hi @hmmkay !

Checking out the dev-8.0 branch and directly executing make from the ports/esp32 directory gives a compile error which I believe I've tracked down to the lv_conf.f from commit d7f2aabfe missing a typedef for the lv_coord_t type.

I suspect this is a local problem in your workspace.
Did you forget updating submodules recursively? (git submodule update --recursive --init)
Did you try to clean and rebuild?
Maybe some other git problem? Did you try to reproduce this by cloning a fresh repo?

I 'fixed' this by copying over the default lv_conf.h with lv_conf_template.h, which led me to Problem 2 where I'm currently stuck ...

I'm trying to keep lv_micropython in a healthy state, compiling and running, for both the unix port and ESP32 port.
I'm not familiar with any special "fixes" that are required in order to compile and run these ports.

If there's an easier/alternate way to get micropython >+1.14 and lvgvl >+ 7 compiled with CMake I'd be happy to be pointed in that direction. The dev-8.0 branch of lv_micropython looked to be the best path to follow for that.

The dev-8.0 is the right path, and it will be merged into main branch in the near future.

[hmmkay]

Thanks for the reply.

Yes, the steps I showed to reproduce Problem 1 start with a clean git clone of lv_micropython.

I am using --recurse-submodules on this initial clone, so perhaps there's some difference that I didn't realize between doing in this way and as a separate 'submodule update --recursive --init' after the fact - I'm busy taking a look to see if that's the problem.

It's good to know it works on your setup - I'll keep plugging away to see what is wrong my end then come back and close this issue with the resolution.

[amirgon]

Also - please make sure you have up-to-date CMake. I'm using cmake 3.18.4.

[hmmkay]

Yes. The version that I show in my reproduction is 3.20.3. I tried different versions of the IDF without success (but think I had different issues if I recall even though I tried to stick to those that micropython is known to work with)

[hmmkay]

Okay, umm, after running 'submodule update --recursive --init' it does in fact now compile.

I am now feeling embarassed because I suspect I've misunderstood something basic about how submodules work. I'll try reproduce where I've gone wrong and post the results back here for any future soul that follows my misguided journey :)

Thanks for the assistance.

[hmmkay]

So for the next person who wanders down this path, the issue was my misunderstanding of switching branches when git submodules are involved:

When I switched to the dev-8.0 branch of lv_micropython and also switched to the dev-8.0 branch of lib/lv_bindings (something I left out in my original Problem 1 steps) then my local repo was still pointing to the commits required by the master branch for the lvgl, pycparser and driver/png/lodepng submodules, instead of those required by the dev-8.0 branch.

ie:

hmmkay@hmmkay ~/repro2/lv_micropython/lib/lv_bindings (master)$git checkout dev-8.0 <-- I'm on the master branch of the lv_bindings submodule, switching to the dev-8.0 branch to get the CMake goodness
M       driver/png/lodepng
M       lvgl
M       pycparser
Switched to branch 'dev-8.0'
Your branch is up to date with 'origin/dev-8.0'.
hmmkay@hmmkay ~/repro2/lv_micropython/lib/lv_bindings (dev-8.0)$ git submodule <-- this tells me what submodules are downstream from this point and what commit they're currently checked out at
+2e541f53ebedf6ebae375a381ca2fd6d82b460bc driver/png/lodepng (2e541f5) <---- outdated commit (used by master branch)
+b16e7f1076963f123b36dec9ce1dd5950057ca1a lvgl (v7.11.0-34-gb16e7f10)  <--- outdated commit (used by master branch)
+e1a1d737be66308b633215fa26ac5ed30e890103 pycparser (release_v2.19-13-ge1a1d73) <--- outdated commit (used by master branch)

If I executed the 'git submodule update --recursive --init' command as amirgon suggested, then suddenly my repo pointed to the correct commits of the various submodules like so:

hmmkay@hmmkay ~/repro2/lv_micropython/lib/lv_bindings (dev-8.0)$ git submodule update  --recursive --init <-- change the submodules to the commits needed by this parent
Submodule path 'driver/png/lodepng': checked out '7fdcc96a5e5864eee72911c3ca79b1d9f0d12292'
Submodule path 'lvgl': checked out '29bfe60438a46342bcd99a7ff57939a7a508ed03'
Submodule path 'pycparser': checked out '1706a39e0116dde0b2d1c52d67078a9a0ab4dbe7'
hmmkay@hmmkay ~/repro2/lv_micropython/lib/lv_bindings (dev-8.0)$ git submodule
 7fdcc96a5e5864eee72911c3ca79b1d9f0d12292 driver/png/lodepng (heads/master) <-- different commit
 29bfe60438a46342bcd99a7ff57939a7a508ed03 lvgl (v8.0.0-3-g29bfe604) <--- now we're using the correct version of LVGL and my compilation errors go away! Yay!
 1706a39e0116dde0b2d1c52d67078a9a0ab4dbe7 pycparser (release_v2.20-27-g1706a39) <--- different commit

So today I learned with git that when you checkout different branches that involve sub-modules, the sub-modules are not automatically switched to the right commit reference the way I assumed they were - you need to manually run the submodule update command to update the sub-modules to the correct commits required by the parent.