IDF fails to generate sections.ld due to latest version of pyparsing (3.1.0) for ESP32S3 (IDFGH-10462)

stericst commented 1 year ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.0.x

Operating System used.

Windows

How did you build your project?

Command line with CMake

If you are using Windows, please specify command line type.

CMD

What is the expected behavior?

Generating sections.ld succeeds using pyparsing

What is the actual behavior?

Generating sections.ld fails when using pyparsing 3.1.0.

pyparsing.exceptions.ParseException: Expected end of text, found 'cpu' (at char 2), (line:2, col:1) During handling of the above exception, another exception occurred: [...] pyparsing.exceptions.ParseException: Unable to parse section info file D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug/esp-idf/esp_hw_support/libesp_hw_support.a. Expected end of text (at char 0), (line:1, col:1)

Steps to reproduce.

Start with cleared cache and removed build folder CMake: Delete Cache and Reconfigure Build The last step where sections.ld is created goes wrong due to pyparsing 3.1.0. This works good when pyparsing is 3.0.9 It seems that since 19 June pyparsing 3.1.0 was released and introduced this error.

Build or installation Logs.

[main] Building folder: espressif-hands-on 
[build] Starting build
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" --build D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug --parallel 10 --target all

Additional building... (this is the not actual output, just cropped)
[...............................]

[continuation:]

[build] [445/458] Building C object esp-idf/esp_pm/CMakeFiles/__idf_esp_pm.dir/pm_impl.c.obj
[build] [446/458] Linking C static library esp-idf\mbedtls\libmbedtls.a
[build] [447/458] Linking C static library esp-idf\esp_pm\libesp_pm.a
[build] [448/458] Building C object esp-idf/driver/CMakeFiles/__idf_driver.dir/esp32s3/touch_sensor.c.obj
[build] [449/458] Linking C static library esp-idf\driver\libdriver.a
[build] [450/458] Linking C static library esp-idf\esp_timer\libesp_timer.a
[build] [451/458] Linking C static library esp-idf\efuse\libefuse.a
[build] [452/458] Linking C static library esp-idf\esp_ringbuf\libesp_ringbuf.a
[build] [453/458] Linking C static library esp-idf\xtensa\libxtensa.a
[build] [454/458] Generating ld/sections.ld
[build] FAILED: esp-idf/esp_system/ld/sections.ld D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug/esp-idf/esp_system/ld/sections.ld 
[build] cmd.exe /C "cd /D D:\gitrepos\espressif-hands-on\build\xtensa-esp32s3-debug\esp-idf\esp_system && python C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/tools/ldgen/ldgen.py --config D:/gitrepos/espressif-hands-on/sdkconfig --fragments-list C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/xtensa/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_ringbuf/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/driver/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_pm/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/spi_flash/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_system/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_system/app.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_common/common.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_common/soc.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_rom/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/hal/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/log/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/heap/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/soc/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_hw_support/linker.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/newlib/newlib.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/newlib/system_libs.lf;C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/freertos/linker.lf --input C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/components/esp_system/ld/esp32s3/sections.ld.in --output D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug/esp-idf/esp_system/ld/sections.ld --kconfig C:/temp/cpm-cache/idf/a0f8453d981df5881c83a1ea3772f1138b75de5d/IDF/Kconfig --env-file D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug/config.env --libraries-file D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug/ldgen_libraries --objdump C:/temp/cpm-cache/.espressif/tools/xtensa-esp32s3-elf/esp-2022r1-11.2.0/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-objdump"
[build] Traceback (most recent call last):
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\entity.py", line 152, in _get_infos_from_file
[build]     results = parser.parseString(info.content, parseAll=True)
[build]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\Users\esteendijk\AppData\Roaming\Python\Python311\site-packages\pyparsing\util.py", line 256, in _inner
[build]     return fn(self, *args, **kwargs)
[build]            ^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\Users\esteendijk\AppData\Roaming\Python\Python311\site-packages\pyparsing\core.py", line 1190, in parse_string
[build]     raise exc.with_traceback(None)
[build] pyparsing.exceptions.ParseException: Expected end of text, found 'cpu'  (at char 2), (line:2, col:1)
[build] 
[build] During handling of the above exception, another exception occurred:
[build] 
[build] Traceback (most recent call last):
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen.py", line 183, in <module>
[build]     main()
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen.py", line 159, in main
[build]     mapping_rules = generation_model.generate(sections_infos)
[build]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 514, in generate
[build]     root_node.insert(entity, sections, target, flags, entities)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 241, in insert
[build]     self.child_placement(entity, sections, target, flags, sections_db)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 233, in child_placement
[build]     child.insert(entity, sections, target, flags, sections_db)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 241, in insert
[build]     self.child_placement(entity, sections, target, flags, sections_db)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 233, in child_placement
[build]     child.insert(entity, sections, target, flags, sections_db)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 241, in insert
[build]     self.child_placement(entity, sections, target, flags, sections_db)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\generation.py", line 296, in child_placement
[build]     found_sections = sections_db.get_sections(self.parent.name, self.name)
[build]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\entity.py", line 197, in get_sections
[build]     obj = self._match_obj(archive, obj)
[build]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\entity.py", line 183, in _match_obj
[build]     objs = self.get_objects(archive)
[build]            ^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\entity.py", line 176, in get_objects
[build]     self._process_archive(archive)
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\entity.py", line 164, in _process_archive
[build]     parsed = self._get_infos_from_file(stored)
[build]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[build]   File "C:\temp\cpm-cache\idf\a0f8453d981df5881c83a1ea3772f1138b75de5d\IDF\tools\ldgen\ldgen\entity.py", line 154, in _get_infos_from_file
[build]     raise ParseException('Unable to parse section info file ' + info.filename + '. ' + p.msg)
[build] pyparsing.exceptions.ParseException: Unable to parse section info file D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug/esp-idf/esp_hw_support/libesp_hw_support.a. Expected end of text  (at char 0), (line:1, col:1)
[build] ninja: build stopped: subcommand failed.
[proc] The command: "C:\Program Files\CMake\bin\cmake.EXE" --build D:/gitrepos/espressif-hands-on/build/xtensa-esp32s3-debug --parallel 10 --target all exited with code: 1
[driver] Build completed: 00:01:15.099
[build] Build finished with exit code 1

More Information.

When manually reverting to pyparsing 3.0.9 the build succeeds. Using: python -m pip install pyparsing==3.0.9

IDF always installs the latest and greatest python packages using requirements.core.txt. Suggestion is to have these to be fixed versions per IDF release.

dobairoland commented 1 year ago

Hi @stericst. This shouldn't happen for v5.0.x. I assume you are using some custom approach to set up the environment. Our scripts use https://dl.espressif.com/dl/esp-idf/espidf.constraints.v5.0.txt in addition to requirements.core.txt to limit the pyparsing version. We had this limit for v5.0 and v5.1 as a precaution and added for the the master (v5.2 development) version as well when we noticed the above breaking change.

How did you set up your environment? Windows tool installer, manually install.bat & export.bat, or custom?

stericst commented 1 year ago

Thanks for your quick message. You are correct, I use a custom approach using CMake and CPM and manually invoke installing the python-env and requirements.core.txt and I was not aware of the existence of constraint file. I see now that when I use espidf.constraints.v5.0.txt it downloads 3.0.9. So this issue can be closed

dobairoland commented 1 year ago

Thanks for the notes and closing this!

Just for future reference and future readers, the updated approach can be found in the documentation: https://docs.espressif.com/projects/esp-idf/en/release-v5.0/esp32/api-guides/tools/idf-tools.html#idf-tools-py-script

espressif / esp-idf