search

espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.19k stars 7.17k forks source link

idf.py build can't find pyparsing, but pip says it is there. (Fedora 37, esp-idf5.1_ py3.11) (IDFGH-8540) #9993

Closed linuxguy123 closed 1 year ago

linuxguy123 commented 1 year ago

Answers checklist.

IDF version.

v5.1-dev

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

No response

What is the expected behavior?

I upgraded my dev machine from F36 to F37 yesterday. Today I can't build my project.

```
$idf.py build
...
Traceback (most recent call last):
  File "/home/me/Development/esp/esp-idf/tools/ldgen/ldgen.py", line 16, in <module>
    from ldgen.entity import EntityDB
  File "/home/me/Development/esp/esp-idf/tools/ldgen/ldgen/entity.py", line 12, in <module>
    from pyparsing import (Group, Literal, OneOrMore, ParseException, SkipTo, Suppress, White, Word, ZeroOrMore, alphas,
ModuleNotFoundError: No module named 'pyparsing'

$ pip install pyparsing
Requirement already satisfied: pyparsing in /home/me/.espressif/python_env/idf5.1_py3.11_env/lib64/python3.11/site-packages (3.0.9)

$ ls -l /home/me/.espressif/python_env/idf5.1_py3.11_env/lib64/python3.11/site-packages/pyparsing
total 364
-rw-r--r--. 1 me me   6426 Oct 15 16:59 actions.py
-rw-r--r--. 1 me me  12936 Oct 15 16:59 common.py
-rw-r--r--. 1 me me 213310 Oct 15 16:59 core.py
drwxr-xr-x. 1 me me     44 Oct 15 16:59 diagram
-rw-r--r--. 1 me me   9023 Oct 15 16:59 exceptions.py
-rw-r--r--. 1 me me  39129 Oct 15 16:59 helpers.py
-rw-r--r--. 1 me me   9159 Oct 15 16:59 __init__.py
drwxr-xr-x. 1 me me    454 Oct 15 16:59 __pycache__
-rw-r--r--. 1 me me      0 Oct 15 16:59 py.typed
-rw-r--r--. 1 me me  25341 Oct 15 16:59 results.py
-rw-r--r--. 1 me me  13402 Oct 15 16:59 testing.py
-rw-r--r--. 1 me me  10787 Oct 15 16:59 unicode.py
-rw-r--r--. 1 me me   6805 Oct 15 16:59 util.py

I already ran these commands, both without error.

$ idf_tools.py install-python-env  <-- to go from 3.10 to 3.11

$ source /home/me/Development/esp/esp-idf/export.sh

Also, I have pyparsing installed natively on my computer.

$ dnf list python3-pyparsing
Installed Packages
python3-pyparsing.noarch                                    3.0.9-2.fc37

Do I have to change the python env setting or a path  or ?

### What is the actual behavior?

See above.

### Steps to reproduce.

1. Upgrade to Fedora 37
2. $idf_tools.py install-python-env
3. $source /home/me/Development/esp/esp-idf/export.sh 
4. $idf.py build

### Build or installation Logs.

```plain
$ idf.py build
Executing action: all (aliases: build)
Running ninja in directory /home/me/Development/MissionControl/groundStation/build
Executing "ninja all"...
[1/8] Generating ld/sections.ldFAILED: esp-idf/esp_system/ld/sections.ld /home/me/Development/MissionControl/groundStation/build/esp-idf/esp_system/ld/sections.ld 
cd /home/me/Development/MissionControl/groundStation/build/esp-idf/esp_system && /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python /home/me/Development/esp/esp-idf/tools/ldgen/ldgen.py --config /home/me/Development/MissionControl/groundStation/sdkconfig --fragments-list "/home/me/Development/esp/esp-idf/components/xtensa/linker.lf;/home/me/Development/esp/esp-idf/components/esp_ringbuf/linker.lf;/home/me/Development/esp/esp-idf/components/driver/linker.lf;/home/me/Development/esp/esp-idf/components/esp_pm/linker.lf;/home/me/Development/esp/esp-idf/components/spi_flash/linker.lf;/home/me/Development/esp/esp-idf/components/esp_system/linker.lf;/home/me/Development/esp/esp-idf/components/esp_system/app.lf;/home/me/Development/esp/esp-idf/components/esp_rom/linker.lf;/home/me/Development/esp/esp-idf/components/hal/linker.lf;/home/me/Development/esp/esp-idf/components/log/linker.lf;/home/me/Development/esp/esp-idf/components/heap/linker.lf;/home/me/Development/esp/esp-idf/components/soc/linker.lf;/home/me/Development/esp/esp-idf/components/esp_hw_support/linker.lf;/home/me/Development/esp/esp-idf/components/freertos/linker.lf;/home/me/Development/esp/esp-idf/components/newlib/newlib.lf;/home/me/Development/esp/esp-idf/components/newlib/system_libs.lf;/home/me/Development/esp/esp-idf/components/esp_common/common.lf;/home/me/Development/esp/esp-idf/components/esp_common/soc.lf;/home/me/Development/esp/esp-idf/components/app_trace/linker.lf;/home/me/Development/esp/esp-idf/components/esp_event/linker.lf;/home/me/Development/esp/esp-idf/components/esp_phy/linker.lf;/home/me/Development/esp/esp-idf/components/lwip/linker.lf;/home/me/Development/esp/esp-idf/components/esp_netif/linker.lf;/home/me/Development/esp/esp-idf/components/esp_wifi/linker.lf;/home/me/Development/esp/esp-idf/components/bt/linker.lf;/home/me/Development/esp/esp-idf/components/esp_adc/linker.lf;/home/me/Development/esp/esp-idf/components/esp_gdbstub/linker.lf;/home/me/Development/esp/esp-idf/components/esp_lcd/linker.lf;/home/me/Development/esp/esp-idf/components/esp_psram/linker.lf;/home/me/Development/esp/esp-idf/components/espcoredump/linker.lf" --input /home/me/Development/esp/esp-idf/components/esp_system/ld/esp32/sections.ld.in --output /home/me/Development/MissionControl/groundStation/build/esp-idf/esp_system/ld/sections.ld --kconfig /home/me/Development/esp/esp-idf/Kconfig --env-file /home/me/Development/MissionControl/groundStation/build/config.env --libraries-file /home/me/Development/MissionControl/groundStation/build/ldgen_libraries --objdump /home/me/.espressif/tools/xtensa-esp32-elf/esp-2022r1-RC1-11.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-objdump
Traceback (most recent call last):
  File "/home/me/Development/esp/esp-idf/tools/ldgen/ldgen.py", line 16, in <module>
    from ldgen.entity import EntityDB
  File "/home/me/Development/esp/esp-idf/tools/ldgen/ldgen/entity.py", line 12, in <module>
    from pyparsing import (Group, Literal, OneOrMore, ParseException, SkipTo, Suppress, White, Word, ZeroOrMore, alphas,
ModuleNotFoundError: No module named 'pyparsing'
[1/1] cd /home/me/Development/MissionControl/groundStation/build/bootlo...evelopment/MissionControl/groundStation/build/bootloader/bootloader.binBootloader binary size 0x66c0 bytes. 0x940 bytes (8%) free.
[3/6] cd /home/me/Development/MissionControl/groundStation/build/esp-id...age-size=256 --obj-name-len=32 --meta-len=4 --use-magic --use-magic-lenninja: build stopped: subcommand failed.
ninja failed with exit code 1, output of the command is in the /home/me/Development/MissionControl/groundStation/build/log/idf_py_stderr_output_129390 and /home/me/Development/MissionControl/groundStation/build/log/idf_py_stdout_output_129390

More Information.

No response

linuxguy123 commented 1 year ago

BTW, upgrading from F36 to F37 added idf5.1_py3.11_env to .espressif_env after I reran install-python-env.

I was running idf5.1_py3.10_env with F36.

dobairoland commented 1 year ago

Hi @linuxguy123. Could you please remove /home/me/.espressif/python_env and rerun the install and export scripts again? Does this solve the issue?

Please use install.sh and . export.sh instead of using idf_tools.py directly.

dobairoland commented 1 year ago

Please also share the output of the install and export scripts if the issue is not solved.

linuxguy123 commented 1 year ago

Thanks for the reply. I'll do this in a bit and get back to you.

linuxguy123 commented 1 year ago

No joy. Output as requested.

$ ./install.sh Detecting the Python interpreter Checking "python3" ... Python 3.11.0rc2 "python3" has been detected Checking Python compatibility Installing ESP-IDF tools Current system platform: linux-amd64 Selected targets are: esp32, esp32h2, esp32s2, esp32s3, esp32c2, esp32c3 Installing tools: xtensa-esp-elf-gdb, riscv32-esp-elf-gdb, xtensa-esp32-elf, xtensa-esp32s2-elf, xtensa-esp32s3-elf, riscv32-esp-elf, esp32ulp-elf, esp32s2ulp-elf, openocd-esp32 Skipping xtensa-esp-elf-gdb@11.2_20220729 (already installed) Skipping riscv32-esp-elf-gdb@11.2_20220729 (already installed) Skipping xtensa-esp32-elf@esp-2022r1-RC1-11.2.0 (already installed) Skipping xtensa-esp32s2-elf@esp-2022r1-RC1-11.2.0 (already installed) Skipping xtensa-esp32s3-elf@esp-2022r1-RC1-11.2.0 (already installed) Skipping riscv32-esp-elf@esp-2022r1-RC1-11.2.0 (already installed) Skipping esp32ulp-elf@2.28.51-esp-20191205 (already installed) Skipping esp32s2ulp-elf@2.28.51-esp-20191205 (already installed) Skipping openocd-esp32@v0.11.0-esp32-20220706 (already installed) Installing Python environment and packages Creating a new Python environment in /home/me/.espressif/python_env/idf5.1_py3.11_env Requirement already satisfied: pip in /home/me/.espressif/python_env/idf5.1_py3.11_env/lib64/python3.11/site-packages (22.2.2) Collecting pip Using cached pip-22.3-py3-none-any.whl (2.1 MB) Requirement already satisfied: setuptools in /home/me/.espressif/python_env/idf5.1_py3.11_env/lib64/python3.11/site-packages (62.6.0) Collecting setuptools Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB) Installing collected packages: setuptools, pip Attempting uninstall: setuptools Found existing installation: setuptools 62.6.0 Uninstalling setuptools-62.6.0: Successfully uninstalled setuptools-62.6.0 Attempting uninstall: pip Found existing installation: pip 22.2.2 Uninstalling pip-22.2.2: Successfully uninstalled pip-22.2.2 Successfully installed pip-22.3 setuptools-65.5.0 Downloading https://dl.espressif.com/dl/esp-idf/espidf.constraints.v5.1.txt Destination: /home/me/.espressif/espidf.constraints.v5.1.txt.tmp Done Installing Python packages Constraint file: /home/me/.espressif/espidf.constraints.v5.1.txt Requirement files:

. ./export.sh Setting IDF_PATH to '/home/me/Development/esp/esp-idf' Detecting the Python interpreter Checking "python3" ... Python 3.11.0rc2 "python3" has been detected Checking Python compatibility Checking other ESP-IDF version. Adding ESP-IDF tools to PATH... Using Python interpreter in /home/me/.espressif/python_env/idf5.1_py3.11_env/bin/python Checking if Python packages are up to date... Skipping the download of /home/me/.espressif/espidf.constraints.v5.1.txt because it was downloaded recently. Constraint file: /home/me/.espressif/espidf.constraints.v5.1.txt Requirement files:

When I go to my project directory, I have to run export.sh there. This is what happens.

$ source /home/me/Development/esp/esp-idf/export.sh Setting IDF_PATH to '/home/me/Development/esp/esp-idf' Detecting the Python interpreter Checking "python3" ... Python 3.11.0rc2 "python3" has been detected Checking Python compatibility Checking other ESP-IDF version. Adding ESP-IDF tools to PATH... Using Python interpreter in /home/me/.espressif/python_env/idf5.1_py3.11_env/bin/python Checking if Python packages are up to date... Skipping the download of /home/me/.espressif/espidf.constraints.v5.1.txt because it was downloaded recently. Constraint file: /home/me/.espressif/espidf.constraints.v5.1.txt Requirement files:

$ idf.py build Executing action: all (aliases: build) Running ninja in directory /home/me/Development/MissionControl/groundStation/build Executing "ninja all"... [1/8] cd /home/me/Development/MissionControl/groundStation/build/esp-id...age-size=256 --obj-name-len=32 --meta-len=4 --use-magic --use-magic-lenFAILED: esp-idf/main/CMakeFiles/spiffs_spiffs_bin /home/me/Development/MissionControl/groundStation/build/esp-idf/main/CMakeFiles/spiffs_spiffs_bin cd /home/me/Development/MissionControl/groundStation/build/esp-idf/main && /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python /home/me/Development/esp/esp-idf/components/spiffs/spiffsgen.py 0x100000 /home/me/Development/MissionControl/groundStation/spiffs /home/me/Development/MissionControl/groundStation/build/spiffs.bin --page-size=256 --obj-name-len=32 --meta-len=4 --use-magic --use-magic-len /bin/sh: line 1: /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python: No such file or directory [2/8] Generating ld/sections.ldFAILED: esp-idf/esp_system/ld/sections.ld /home/me/Development/MissionControl/groundStation/build/esp-idf/esp_system/ld/sections.ld cd /home/me/Development/MissionControl/groundStation/build/esp-idf/esp_system && /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python /home/me/Development/esp/esp-idf/tools/ldgen/ldgen.py --config /home/me/Development/MissionControl/groundStation/sdkconfig --fragments-list "/home/me/Development/esp/esp-idf/components/xtensa/linker.lf;/home/me/Development/esp/esp-idf/components/esp_ringbuf/linker.lf;/home/me/Development/esp/esp-idf/components/driver/linker.lf;/home/me/Development/esp/esp-idf/components/esp_pm/linker.lf;/home/me/Development/esp/esp-idf/components/spi_flash/linker.lf;/home/me/Development/esp/esp-idf/components/esp_system/linker.lf;/home/me/Development/esp/esp-idf/components/esp_system/app.lf;/home/me/Development/esp/esp-idf/components/esp_rom/linker.lf;/home/me/Development/esp/esp-idf/components/hal/linker.lf;/home/me/Development/esp/esp-idf/components/log/linker.lf;/home/me/Development/esp/esp-idf/components/heap/linker.lf;/home/me/Development/esp/esp-idf/components/soc/linker.lf;/home/me/Development/esp/esp-idf/components/esp_hw_support/linker.lf;/home/me/Development/esp/esp-idf/components/freertos/linker.lf;/home/me/Development/esp/esp-idf/components/newlib/newlib.lf;/home/me/Development/esp/esp-idf/components/newlib/system_libs.lf;/home/me/Development/esp/esp-idf/components/esp_common/common.lf;/home/me/Development/esp/esp-idf/components/esp_common/soc.lf;/home/me/Development/esp/esp-idf/components/app_trace/linker.lf;/home/me/Development/esp/esp-idf/components/esp_event/linker.lf;/home/me/Development/esp/esp-idf/components/esp_phy/linker.lf;/home/me/Development/esp/esp-idf/components/lwip/linker.lf;/home/me/Development/esp/esp-idf/components/esp_netif/linker.lf;/home/me/Development/esp/esp-idf/components/esp_wifi/linker.lf;/home/me/Development/esp/esp-idf/components/bt/linker.lf;/home/me/Development/esp/esp-idf/components/esp_adc/linker.lf;/home/me/Development/esp/esp-idf/components/esp_gdbstub/linker.lf;/home/me/Development/esp/esp-idf/components/esp_lcd/linker.lf;/home/me/Development/esp/esp-idf/components/esp_psram/linker.lf;/home/me/Development/esp/esp-idf/components/espcoredump/linker.lf" --input /home/me/Development/esp/esp-idf/components/esp_system/ld/esp32/sections.ld.in --output /home/me/Development/MissionControl/groundStation/build/esp-idf/esp_system/ld/sections.ld --kconfig /home/me/Development/esp/esp-idf/Kconfig --env-file /home/me/Development/MissionControl/groundStation/build/config.env --libraries-file /home/me/Development/MissionControl/groundStation/build/ldgen_libraries --objdump /home/me/.espressif/tools/xtensa-esp32-elf/esp-2022r1-RC1-11.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-objdump /bin/sh: line 1: /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python: No such file or directory [3/8] Performing build step for 'bootloader'FAILED: bootloader-prefix/src/bootloader-stamp/bootloader-build bootloader/bootloader.elf bootloader/bootloader.bin bootloader/bootloader.map /home/me/Development/MissionControl/groundStation/build/bootloader-prefix/src/bootloader-stamp/bootloader-build /home/me/Development/MissionControl/groundStation/build/bootloader/bootloader.elf /home/me/Development/MissionControl/groundStation/build/bootloader/bootloader.bin /home/me/Development/MissionControl/groundStation/build/bootloader/bootloader.map cd /home/me/Development/MissionControl/groundStation/build/bootloader && /usr/bin/cmake --build . [1/1] cd /home/me/Development/MissionControl/groundStation/build/bootlo...evelopment/MissionControl/groundStation/build/bootloader/bootloader.binFAILED: esp-idf/esptool_py/CMakeFiles/bootloader_check_size /home/me/Development/MissionControl/groundStation/build/bootloader/esp-idf/esptool_py/CMakeFiles/bootloader_check_size cd /home/me/Development/MissionControl/groundStation/build/bootloader/esp-idf/esptool_py && /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python /home/me/Development/esp/esp-idf/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x1000 /home/me/Development/MissionControl/groundStation/build/bootloader/bootloader.bin /bin/sh: line 1: /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python: No such file or directory ninja: build stopped: subcommand failed. ninja: build stopped: subcommand failed. ninja failed with exit code 1, output of the command is in the /home/me/Development/MissionControl/groundStation/build/log/idf_py_stderr_output_20925 and /home/me/Development/MissionControl/groundStation/build/log/idf_py_stdout_output_20925

dobairoland commented 1 year ago

What I've noticed that /home/me/.espressif/python_env/idf5.1_py3.11_env/bin/python is set up properly (Notice 3.11 in the path). However, /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python is used in the output of idf.py build (notice 3.10 instead).

So I guess the upgrade of the Python system had some issue.

What is the output of which python and which python3? Do you have any aliases set to your previous 3.10? Have you set up an alias with update-alternatives (probably the output of "which python" will tell)?

dobairoland commented 1 year ago

Please also run idf.py fullclean before idf.py build (just to make sure the old Python path is not used from the cache).

linuxguy123 commented 1 year ago

What I've noticed that /home/me/.espressif/python_env/idf5.1_py3.11_env/bin/python is set up properly (Notice 3.11 in the path). However, /home/me/.espressif/python_env/idf5.1_py3.10_env/bin/python is used in the output of idf.py build (notice 3.10 instead).

It did after following your instructions. If you look at my original post, was trying to use 3.11

So I guess the upgrade of the Python system had some issue.

What is the output of which python and which python3?

$ which python /home/me/.espressif/python_env/idf5.1_py3.11_env/bin/python

$ which python3 /home/me/.espressif/python_env/idf5.1_py3.11_env/bin/python3

Do you have any aliases set to your previous 3.10?

I don't believe so.

Have you set up an alias with update-alternatives (probably the output of "which python" will tell)?

Nope

linuxguy123 commented 1 year ago

Please also run idf.py fullclean before idf.py build (just to make sure the old Python path is not used from the cache).

BINGO. This fixed it. It works now. It built successfully.

Thank you for your help.

Alvin1Zhang commented 1 year ago

Thanks for sharing the updates, feel free to reopen.