v923z / micropython-ulab

a numpy-like fast vector module for micropython, circuitpython, and their derivatives
https://micropython-ulab.readthedocs.io/en/latest
MIT License
432 stars 117 forks source link

[BUG] Can't compile ulab with BOARD=GENERIC_S3 (GENERIC works fine) #595

Closed zachmoshe closed 1 year ago

zachmoshe commented 1 year ago

I'm trying to compile ulab into MicroPython, following this page. I have an old ESP32 and a newer ESP32-S3 which I want to try.

I've created two makefiles for BOARD=GENERIC and BOARD=GENERIC_S3. The first one works as expected, but when I try the S3 one I get this:

zmoshe@laptop ~/proj/micropython/ports/esp32 % make -f makefile_ulab_s3
idf.py -D MICROPY_BOARD=GENERIC_S3 -D MICROPY_BOARD_DIR=/Users/zmoshe/proj/micropython/ports/esp32/boards/GENERIC_S3 -B build-GENERIC_S3  -DUSER_C_MODULES=/Users/zmoshe/proj/ulab/code/micropython.cmake build || (echo -e "See \033[1;31mhttps://github.com/micropython/micropython/wiki/Build-Troubleshooting\033[0m"; false)
Executing action: all (aliases: build)
Running ninja in directory /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3
Executing "ninja all"...
[1/297] Performing build step for 'bootloader'
[1/1] cd /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/bootloader/esp-idf/esptool_py && /Users/zmoshe/.espressif/python_env/idf4.4_py3.10_env/bin/python /Users/zmoshe/proj/esp-idf/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x0 /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/bootloader/bootloader.bin
Bootloader binary size 0x48d0 bytes. 0x3730 bytes (43%) free.
[156/214] Building C object esp-idf/main/CMakeFiles/__idf_main.dir/__/__/frozen_content.c.obj
FAILED: esp-idf/main/CMakeFiles/__idf_main.dir/__/__/frozen_content.c.obj
/Users/zmoshe/.espressif/tools/xtensa-esp32s3-elf/esp-2021r2-8.4.0/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc -DFFCONF_H=\"/Users/zmoshe/proj/micropython/lib/oofatfs/ffconf.h\" -DLFS1_NO_ASSERT -DLFS1_NO_DEBUG -DLFS1_NO_ERROR -DLFS1_NO_MALLOC -DLFS1_NO_WARN -DLFS2_NO_ASSERT -DLFS2_NO_DEBUG -DLFS2_NO_ERROR -DLFS2_NO_MALLOC -DLFS2_NO_WARN -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DMICROPY_ESP_IDF_4=1 -DMICROPY_MODULE_FROZEN_MPY="(1)" -DMICROPY_PY_BTREE=1 -DMICROPY_QSTR_EXTRA_POOL=mp_qstr_frozen_const_pool -DMICROPY_VFS_FAT=1 -DMICROPY_VFS_LFS2=1 -DMODULE_ULAB_ENABLED=1 -D__DBINTERFACE_PRIVATE=1 -Dvirt_fd_t="void*" -I/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/config -I/Users/zmoshe/proj/micropython -I/Users/zmoshe/proj/micropython/lib/berkeley-db-1.xx/PORT/include -I/Users/zmoshe/proj/ulab/code -I/Users/zmoshe/proj/micropython/ports/esp32 -I/Users/zmoshe/proj/micropython/ports/esp32/boards/GENERIC_S3 -I/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3 -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble -I/Users/zmoshe/proj/esp-idf/components/newlib/platform_include -I/Users/zmoshe/proj/esp-idf/components/freertos/include -I/Users/zmoshe/proj/esp-idf/components/freertos/include/esp_additions/freertos -I/Users/zmoshe/proj/esp-idf/components/freertos/port/xtensa/include -I/Users/zmoshe/proj/esp-idf/components/freertos/include/esp_additions -I/Users/zmoshe/proj/esp-idf/components/esp_hw_support/include -I/Users/zmoshe/proj/esp-idf/components/esp_hw_support/include/soc -I/Users/zmoshe/proj/esp-idf/components/esp_hw_support/include/soc/esp32s3 -I/Users/zmoshe/proj/esp-idf/components/esp_hw_support/port/esp32s3/. -I/Users/zmoshe/proj/esp-idf/components/esp_hw_support/port/esp32s3/private_include -I/Users/zmoshe/proj/esp-idf/components/heap/include -I/Users/zmoshe/proj/esp-idf/components/log/include -I/Users/zmoshe/proj/esp-idf/components/lwip/include/apps -I/Users/zmoshe/proj/esp-idf/components/lwip/include/apps/sntp -I/Users/zmoshe/proj/esp-idf/components/lwip/lwip/src/include -I/Users/zmoshe/proj/esp-idf/components/lwip/port/esp32/include -I/Users/zmoshe/proj/esp-idf/components/lwip/port/esp32/include/arch -I/Users/zmoshe/proj/esp-idf/components/soc/include -I/Users/zmoshe/proj/esp-idf/components/soc/esp32s3/. -I/Users/zmoshe/proj/esp-idf/components/soc/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/hal/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/hal/include -I/Users/zmoshe/proj/esp-idf/components/hal/platform_port/include -I/Users/zmoshe/proj/esp-idf/components/esp_rom/include -I/Users/zmoshe/proj/esp-idf/components/esp_rom/include/esp32s3 -I/Users/zmoshe/proj/esp-idf/components/esp_rom/esp32s3 -I/Users/zmoshe/proj/esp-idf/components/esp_common/include -I/Users/zmoshe/proj/esp-idf/components/esp_system/include -I/Users/zmoshe/proj/esp-idf/components/esp_system/port/soc -I/Users/zmoshe/proj/esp-idf/components/esp_system/port/public_compat -I/Users/zmoshe/proj/esp-idf/components/xtensa/include -I/Users/zmoshe/proj/esp-idf/components/xtensa/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/driver/include -I/Users/zmoshe/proj/esp-idf/components/driver/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/esp_pm/include -I/Users/zmoshe/proj/esp-idf/components/esp_ringbuf/include -I/Users/zmoshe/proj/esp-idf/components/efuse/include -I/Users/zmoshe/proj/esp-idf/components/efuse/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/vfs/include -I/Users/zmoshe/proj/esp-idf/components/esp_wifi/include -I/Users/zmoshe/proj/esp-idf/components/esp_event/include -I/Users/zmoshe/proj/esp-idf/components/esp_netif/include -I/Users/zmoshe/proj/esp-idf/components/esp_eth/include -I/Users/zmoshe/proj/esp-idf/components/tcpip_adapter/include -I/Users/zmoshe/proj/esp-idf/components/esp_phy/include -I/Users/zmoshe/proj/esp-idf/components/esp_phy/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/esp_ipc/include -I/Users/zmoshe/proj/esp-idf/components/app_trace/include -I/Users/zmoshe/proj/esp-idf/components/esp_timer/include -I/Users/zmoshe/proj/esp-idf/components/app_update/include -I/Users/zmoshe/proj/esp-idf/components/spi_flash/include -I/Users/zmoshe/proj/esp-idf/components/bootloader_support/include -I/Users/zmoshe/proj/esp-idf/components/bt/common/osi/include -I/Users/zmoshe/proj/esp-idf/components/bt/include/esp32s3/include -I/Users/zmoshe/proj/esp-idf/components/bt/common/api/include/api -I/Users/zmoshe/proj/esp-idf/components/bt/common/btc/profile/esp/blufi/include -I/Users/zmoshe/proj/esp-idf/components/bt/common/btc/profile/esp/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/porting/nimble/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/port/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/ans/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/bas/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/dis/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/gap/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/gatt/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/ias/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/ipss/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/lls/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/tps/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/util/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/store/ram/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/nimble/host/store/config/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/nimble/porting/npl/freertos/include -I/Users/zmoshe/proj/esp-idf/components/bt/host/nimble/esp-hci/include -I/Users/zmoshe/proj/esp-idf/components/nvs_flash/include -I/Users/zmoshe/proj/esp-idf/components/mbedtls/port/include -I/Users/zmoshe/proj/esp-idf/components/mbedtls/mbedtls/include -I/Users/zmoshe/proj/esp-idf/components/mbedtls/esp_crt_bundle/include -I/Users/zmoshe/proj/esp-idf/components/esp_adc_cal/include -I/Users/zmoshe/proj/esp-idf/components/mdns/include -I/Users/zmoshe/proj/esp-idf/components/console -I/Users/zmoshe/proj/esp-idf/components/sdmmc/include -I/Users/zmoshe/proj/esp-idf/components/ulp/include -I/Users/zmoshe/proj/esp-idf/components/tinyusb/tinyusb/src -I/Users/zmoshe/proj/esp-idf/components/tinyusb/additions/include -I/Users/zmoshe/proj/esp-idf/components/freertos/include/freertos -mlongcalls -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -ggdb -O2 -fmacro-prefix-map=/Users/zmoshe/proj/micropython/ports/esp32=. -fmacro-prefix-map=/Users/zmoshe/proj/esp-idf=IDF -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu99 -Wno-old-style-declaration -D_GNU_SOURCE -DIDF_VER=\"v4.4\" -DESP_PLATFORM -DNDEBUG -D_POSIX_READER_WRITER_LOCKS -Wno-clobbered -Wno-deprecated-declarations -Wno-missing-field-initializers -MD -MT esp-idf/main/CMakeFiles/__idf_main.dir/__/__/frozen_content.c.obj -MF esp-idf/main/CMakeFiles/__idf_main.dir/__/__/frozen_content.c.obj.d -o esp-idf/main/CMakeFiles/__idf_main.dir/__/__/frozen_content.c.obj -c /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:60:5: error: redeclaration of enumerator 'MP_QSTR_w'
     MP_QSTR_w,
     ^~~~~~~~~
In file included from /Users/zmoshe/proj/micropython/py/obj.h:33,
                 from /Users/zmoshe/proj/micropython/py/objint.h:30,
                 from /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:16:
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/genhdr/qstrdefs.generated.h:1130:6: note: previous definition of 'MP_QSTR_w' was here
 QDEF(MP_QSTR_w, 46546, 1, "w")
      ^~~~~~~~~
/Users/zmoshe/proj/micropython/py/qstr.h:41:34: note: in definition of macro 'QDEF'
 #define QDEF(id, hash, len, str) id,
                                  ^~
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:73:5: error: redeclaration of enumerator 'MP_QSTR___version__'
     MP_QSTR___version__,
     ^~~~~~~~~~~~~~~~~~~
In file included from /Users/zmoshe/proj/micropython/py/obj.h:33,
                 from /Users/zmoshe/proj/micropython/py/objint.h:30,
                 from /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:16:
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/genhdr/qstrdefs.generated.h:242:6: note: previous definition of 'MP_QSTR___version__' was here
 QDEF(MP_QSTR___version__, 38975, 11, "__version__")
      ^~~~~~~~~~~~~~~~~~~
/Users/zmoshe/proj/micropython/py/qstr.h:41:34: note: in definition of macro 'QDEF'
 #define QDEF(id, hash, len, str) id,
                                  ^~
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:164:5: error: redeclaration of enumerator 'MP_QSTR_v'
     MP_QSTR_v,
     ^~~~~~~~~
In file included from /Users/zmoshe/proj/micropython/py/obj.h:33,
                 from /Users/zmoshe/proj/micropython/py/objint.h:30,
                 from /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:16:
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/genhdr/qstrdefs.generated.h:1124:6: note: previous definition of 'MP_QSTR_v' was here
 QDEF(MP_QSTR_v, 46547, 1, "v")
      ^~~~~~~~~
/Users/zmoshe/proj/micropython/py/qstr.h:41:34: note: in definition of macro 'QDEF'
 #define QDEF(id, hash, len, str) id,
                                  ^~
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:165:5: error: redeclaration of enumerator 'MP_QSTR_n'
     MP_QSTR_n,
     ^~~~~~~~~
In file included from /Users/zmoshe/proj/micropython/py/obj.h:33,
                 from /Users/zmoshe/proj/micropython/py/objint.h:30,
                 from /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:16:
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/genhdr/qstrdefs.generated.h:859:6: note: previous definition of 'MP_QSTR_n' was here
 QDEF(MP_QSTR_n, 46539, 1, "n")
      ^~~~~~~~~
/Users/zmoshe/proj/micropython/py/qstr.h:41:34: note: in definition of macro 'QDEF'
 #define QDEF(id, hash, len, str) id,
                                  ^~
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:188:5: error: redeclaration of enumerator 'MP_QSTR_delete'
     MP_QSTR_delete,
     ^~~~~~~~~~~~~~
In file included from /Users/zmoshe/proj/micropython/py/obj.h:33,
                 from /Users/zmoshe/proj/micropython/py/objint.h:30,
                 from /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:16:
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/genhdr/qstrdefs.generated.h:600:6: note: previous definition of 'MP_QSTR_delete' was here
 QDEF(MP_QSTR_delete, 59292, 6, "delete")
      ^~~~~~~~~~~~~~
/Users/zmoshe/proj/micropython/py/qstr.h:41:34: note: in definition of macro 'QDEF'
 #define QDEF(id, hash, len, str) id,
                                  ^~
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:329:5: error: redeclaration of enumerator 'MP_QSTR_diff'
     MP_QSTR_diff,
     ^~~~~~~~~~~~
In file included from /Users/zmoshe/proj/micropython/py/obj.h:33,
                 from /Users/zmoshe/proj/micropython/py/objint.h:30,
                 from /Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/frozen_content.c:16:
/Users/zmoshe/proj/micropython/ports/esp32/build-GENERIC_S3/genhdr/qstrdefs.generated.h:609:6: note: previous definition of 'MP_QSTR_diff' was here
 QDEF(MP_QSTR_diff, 64712, 4, "diff")
      ^~~~~~~~~~~~
/Users/zmoshe/proj/micropython/py/qstr.h:41:34: note: in definition of macro 'QDEF'
 #define QDEF(id, hash, len, str) id,
                                  ^~
[173/214] Building C object esp-idf/main/CMakeFiles/__idf_main.dir/Users/zmoshe/proj/ulab/code/numpy/numerical.c.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
-e See https://github.com/micropython/micropython/wiki/Build-Troubleshooting
make: *** [all] Error 1

I'm a bit clueless with C compilation errors. Looks like something is declared twice but not sure why this doesn't happen for the regular board type.. Any ideas?

v923z commented 1 year ago

What happens if you remove the artifacts, and start from scratch? It seems to me that the prepocessor doesn't reach the point, where ulab would make any difference. In other words, I believe, you would get the same error, if you left out ulab completely.

hamza-712 commented 1 year ago

I was having the same problem. I used esp idf v4.4.3 and I was able to build firmware.

hamza-712 commented 1 year ago

@v923z I need help too. I have built firmware for ESP-WROOM-32 but my ESP32 keeps resetting indefinitely. Is there a compiled binary available with latest micropython firmware?

I looked at the troubleshooting link as well. I changed CONFIG_ESPTOOLPY_FLASHMODE but it didn't work.

https://github.com/micropython/micropython/tree/master/ports/esp32#configurationtroubleshooting Configuration Troubleshooting Continuous reboots after programming: Ensure CONFIG_ESPTOOLPY_FLASHMODE is correct for your board (e.g. ESP-WROOM-32 should be DIO). Then perform a make clean, rebuild, redeploy.

zachmoshe commented 1 year ago

OK, it's a bit weird but I can confirm it works for me.. It didn't at first, then as Zoltan suggested I tried make clean and even re-cloning micropython from scratch. After a few tries I was able to compile both the GENERIC and the GENERIC_S3 boards. I'm not sure what exactly did the trick but here are the settings that work for me:

include Makefile

* I have ulab at commit `8585407` (one before the latest, don't think it will make a difference). The only change I have there is in `code/ulab.h`: 

-#define ULAB_MAX_DIMS 2 +#define ULAB_MAX_DIMS 4


* My ESP-IDF is `v4.4`.

From the `micropython/ports/esp32` I can successfully run `make -f makefile_ulab_s3` now (for both files). 
I also uploaded the result binaries to ESP32-S3 and it seems to work (was able to import numpy form the repl, didn't check further).

@hamza-712 - Hope this helps... 
hamza-712 commented 1 year ago

thanks i have successfully built firmware I use idf.py command to build firmware instead of make and it worked.

v923z commented 1 year ago

@zachmoshe @hamza-712 If either of you could reproduce these results, it would be a worthwhile addition to https://github.com/v923z/micropython-builder.

zachmoshe commented 1 year ago

@v923z - I believe this can be closed as we both were able to build the firmware and it probably wasn't a ulab problem anyway ( I guess a WONT_FIX?)

v923z commented 1 year ago

You can definitely close it, but I still feel that others could profit from an automated build. Let me know if you're interested.

zachmoshe commented 1 year ago

sure, I'll take a look and submit a PR