Menuconfig does not run with python 3.10.6, citing lack of curses python module (IDFGH-10387)

hierophect commented 1 year ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.2-dev-1010-g213504238f

Operating System used.

Ubuntu 22.04

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

running idf.py menuconfig should open menuconfig

What is the actual behavior?

Executing action: menuconfig
Running ninja in directory /home/hierophect/repos/hello_world/build
Executing "ninja menuconfig"...
[0/1] Re-running CMake...
-- git rev-parse returned 'fatal: not a git repository (or any of the parent directories): .git'
-- Could not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32s2
-- Project sdkconfig file /home/hierophect/repos/hello_world/sdkconfig
-- Compiler supported targets: xtensa-esp32s2-elf
-- App "hello_world" version: 1
-- Adding linker script /home/hierophect/repos/hello_world/build/esp-idf/esp_system/ld/memory.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_system/ld/esp32s2/sections.ld.in
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.api.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.libgcc.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.newlib-funcs.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.newlib-data.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/esp_rom/esp32s2/ld/esp32s2.rom.spiflash.ld
-- Adding linker script /home/hierophect/repos/esp-idf/components/soc/esp32s2/ld/esp32s2.peripherals.ld
-- Components: [long]
-- Component paths: [long]
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hierophect/repos/hello_world/build
[0/1] cd /home/hierophect/repos/hello_world/build...nfig /home/hierophect/repos/hello_world/sdkconfig
TERM environment variable is set to "xterm-256color"
Traceback (most recent call last):
  File "/home/hierophect/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/hierophect/.pyenv/versions/3.10.6/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/hierophect/.espressif/python_env/idf5.2_py3.10_env/lib/python3.10/site-packages/menuconfig.py", line 194, in <module>
    import curses
  File "/home/hierophect/.pyenv/versions/3.10.6/lib/python3.10/curses/__init__.py", line 13, in <module>
    from _curses import *
ModuleNotFoundError: No module named '_curses'
FAILED: CMakeFiles/menuconfig /home/hierophect/repos/hello_world/build/CMakeFiles/menuconfig 
cd /home/hierophect/repos/hello_world/build && /home/hierophect/.espressif/python_env/idf5.2_py3.10_env/bin/python /home/hierophect/repos/esp-idf/tools/kconfig_new/prepare_kconfig_files.py --list-separator=semicolon --env-file /home/hierophect/repos/hello_world/build/config.env && /home/hierophect/.espressif/python_env/idf5.2_py3.10_env/bin/python -m kconfgen --list-separator=semicolon --kconfig /home/hierophect/repos/esp-idf/Kconfig --sdkconfig-rename /home/hierophect/repos/esp-idf/sdkconfig.rename --config /home/hierophect/repos/hello_world/sdkconfig --env-file /home/hierophect/repos/hello_world/build/config.env --env IDF_TARGET=esp32s2 --env IDF_TOOLCHAIN=gcc --env IDF_ENV_FPGA= --dont-write-deprecated --output config /home/hierophect/repos/hello_world/sdkconfig && /home/hierophect/.espressif/python_env/idf5.2_py3.10_env/bin/python /home/hierophect/repos/esp-idf/tools/check_term.py && /usr/bin/cmake -E env COMPONENT_KCONFIGS_SOURCE_FILE=/home/hierophect/repos/hello_world/build/kconfigs.in COMPONENT_KCONFIGS_PROJBUILD_SOURCE_FILE=/home/hierophect/repos/hello_world/build/kconfigs_projbuild.in KCONFIG_CONFIG=/home/hierophect/repos/hello_world/sdkconfig IDF_TARGET=esp32s2 IDF_TOOLCHAIN=gcc IDF_ENV_FPGA= /home/hierophect/.espressif/python_env/idf5.2_py3.10_env/bin/python -m menuconfig /home/hierophect/repos/esp-idf/Kconfig && /home/hierophect/.espressif/python_env/idf5.2_py3.10_env/bin/python -m kconfgen --list-separator=semicolon --kconfig /home/hierophect/repos/esp-idf/Kconfig --sdkconfig-rename /home/hierophect/repos/esp-idf/sdkconfig.rename --config /home/hierophect/repos/hello_world/sdkconfig --env-file /home/hierophect/repos/hello_world/build/config.env --env IDF_TARGET=esp32s2 --env IDF_TOOLCHAIN=gcc --env IDF_ENV_FPGA= --output config /home/hierophect/repos/hello_world/sdkconfig
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

Steps to reproduce.

1) Install the IDF based on the latest 5.2 documentation 2) export idf tools 3) attempt to run menuconfig on any project

Build or installation Logs.

No response

More Information.

I've been setting up the idf on a new ubuntu 22.04 computer and ran into this issue while installing. It was a very conventional install and this is almost the first thing I've done on the machine. Reverting manually to 3.9.4 (manually or with pyenv) resolved the issue. There are a couple things I'm confused by here:

While researching this issue, I was under the impression that idf versions later than 4.2 were not using curses at all? Why is it a prerequisite for 5.2?
I'm not exactly a python guru so I'm not sure why the curses module fails with 3.10.6. However, it was the default version installed when installing the prerequisites. Should users be warned to manually install a specific python version?
Alternatively, while I'm not confident on this from my research, it seems possible that pyenv and the native installation of python3 simply broke while installing, somehow. I'm not sure the mechanics of how this would happen, especially with a fresh install on a new machine, so I'm also curious if this might be worth some kind of warning.

823984418 commented 1 year ago

Is this the same problem I encountered?

PS D:\code\esp32_demo> idf.py menuconfig           
Executing action: menuconfig
Running ninja in directory D:\code\esp32_demo\build
Executing "ninja menuconfig"...
[0/1] Re-running CMake...
-- ccache will be used for faster recompilation
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
-- Project sdkconfig file D:/code/esp32_demo/sdkconfig
Compiler supported targets: xtensa-esp32-elf

-- App "esp32_demo" version: 1
-- Adding linker script D:/code/esp32_demo/build/esp-idf/esp_system/ld/memory.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_system/ld/esp32/sections.ld.in
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom/esp32/ld/esp32.rom.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom/esp32/ld/esp32.rom.api.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld
-- Adding linker script E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/soc/esp32/ld/esp32.peripherals.ld
-- Components: app_trace app_update bootloader bootloader_support bt cmock console cxx driver efuse esp-tls esp_adc esp_app_format esp_coex esp_common esp_eth esp_event esp_gdbstub esp
_hid esp_http_client esp_http_server esp_https_ota esp_https_server esp_hw_support esp_lcd esp_local_ctrl esp_mm esp_netif esp_netif_stack esp_partition esp_phy esp_pm esp_psram esp_ri
ngbuf esp_rom esp_system esp_timer esp_wifi espcoredump esptool_py fatfs freertos hal heap http_parser idf_test ieee802154 json log lwip main mbedtls mqtt newlib nvs_flash openthread partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport ulp unity usb vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/app_trace E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/app_update E:/eps32_env/espressif_5.
1/esp-idf/release-v5.1/components/bootloader E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/bootloader_support E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/bt
 E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/cmock E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/console E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/com
ponents/cxx E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/driver E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/efuse E:/eps32_env/espressif_5.1/esp-idf/releas
e-v5.1/components/esp-tls E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_adc E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_app_format E:/eps32_env/espr
essif_5.1/esp-idf/release-v5.1/components/esp_coex E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_common E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_
eth E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_event E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_gdbstub E:/eps32_env/espressif_5.1/esp-idf/relea
se-v5.1/components/esp_hid E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_http_client E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_http_server E:/eps3
2_env/espressif_5.1/esp-idf/release-v5.1/components/esp_https_ota E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_https_server E:/eps32_env/espressif_5.1/esp-idf/release
-v5.1/components/esp_hw_support E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_lcd E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_local_ctrl E:/eps32_en
v/espressif_5.1/esp-idf/release-v5.1/components/esp_mm E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_netif E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/e
sp_netif_stack E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_partition E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_phy E:/eps32_env/espressif_5.1/es
p-idf/release-v5.1/components/esp_pm E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_psram E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_ringbuf E:/eps3
2_env/espressif_5.1/esp-idf/release-v5.1/components/esp_rom E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_system E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/compon
ents/esp_timer E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/esp_wifi E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/espcoredump E:/eps32_env/espressif_5.1/esp
-idf/release-v5.1/components/esptool_py E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/fatfs E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/freertos E:/eps32_en
v/espressif_5.1/esp-idf/release-v5.1/components/hal E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/heap E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/http_pars
er E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/idf_test E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/ieee802154 E:/eps32_env/espressif_5.1/esp-idf/release-
v5.1/components/json E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/log E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/lwip D:/code/esp32_demo/main E:/eps32_env
/espressif_5.1/esp-idf/release-v5.1/components/mbedtls E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/mqtt E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/newlib
 E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/nvs_flash E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/openthread E:/eps32_env/espressif_5.1/esp-idf/release-v
5.1/components/partition_table E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/perfmon E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/protobuf-c E:/eps32_env/esp
ressif_5.1/esp-idf/release-v5.1/components/protocomm E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/pthread E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/sdmmc
 E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/soc E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/spi_flash E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/com
ponents/spiffs E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/tcp_transport E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/ulp E:/eps32_env/espressif_5.1/esp-id
f/release-v5.1/components/unity E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/usb E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/vfs E:/eps32_env/espressif_5.1
/esp-idf/release-v5.1/components/wear_levelling E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/wifi_provisioning E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/wpa_supplicant E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/components/xtensa
-- Configuring done
-- Generating done
-- Build files have been written to: D:/code/esp32_demo/build
[0/1] cmd.exe /C "cd /D D:\code\esp32_demo\build && E:\home\.espressif\python_env\idf5.1_p...v --env IDF_TARGET=esp32 --env IDF_ENV_FPGA= --output config D:/code/esp32_demo/sdkconfig" 
menuconfig failed to import the standard Python 'curses' library. Try
installing a package like windows-curses
(https://github.com/zephyrproject-rtos/windows-curses) by running this command
in cmd.exe:

    pip install windows-curses

Starting with Kconfiglib 13.0.0, windows-curses is no longer automatically
installed when installing Kconfiglib via pip on Windows (because it breaks
installation on MSYS2).

Exception:
ModuleNotFoundError: No module named '_curses'
FAILED: CMakeFiles/menuconfig D:/code/esp32_demo/build/CMakeFiles/menuconfig
cmd.exe /C "cd /D D:\code\esp32_demo\build && E:\home\.espressif\python_env\idf5.1_py3.7_env\Scripts\python.exe E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/tools/kconfig_new/prepar
e_kconfig_files.py --list-separator=semicolon --env-file D:/code/esp32_demo/build/config.env && E:\home\.espressif\python_env\idf5.1_py3.7_env\Scripts\python.exe -m kconfgen --list-sep
arator=semicolon --kconfig E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/Kconfig --sdkconfig-rename E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/sdkconfig.rename --config D:/code/
esp32_demo/sdkconfig --env-file D:/code/esp32_demo/build/config.env --env IDF_TARGET=esp32 --env IDF_ENV_FPGA= --dont-write-deprecated --output config D:/code/esp32_demo/sdkconfig && E
:\home\.espressif\python_env\idf5.1_py3.7_env\Scripts\python.exe E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/tools/check_term.py && E:\home\.espressif\tools\cmake\3.24.0\bin\cmake.
exe -E env COMPONENT_KCONFIGS_SOURCE_FILE=D:/code/esp32_demo/build/kconfigs.in COMPONENT_KCONFIGS_PROJBUILD_SOURCE_FILE=D:/code/esp32_demo/build/kconfigs_projbuild.in KCONFIG_CONFIG=D:
/code/esp32_demo/sdkconfig IDF_TARGET=esp32 IDF_ENV_FPGA= E:/home/.espressif/python_env/idf5.1_py3.7_env/Scripts/python.exe -m menuconfig E:/eps32_env/espressif_5.1/esp-idf/release-v5.
1/Kconfig && E:\home\.espressif\python_env\idf5.1_py3.7_env\Scripts\python.exe -m kconfgen --list-separator=semicolon --kconfig E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/Kconfig 
--sdkconfig-rename E:/eps32_env/espressif_5.1/esp-idf/release-v5.1/sdkconfig.rename --config D:/code/esp32_demo/sdkconfig --env-file D:/code/esp32_demo/build/config.env --env IDF_TARGET=esp32 --env IDF_ENV_FPGA= --output config D:/code/esp32_demo/sdkconfig"
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

I eventually solved it by executing pip in the environment.

dobairoland commented 1 year ago

@823984418 No, it is not the same. That package is required additionally on Windows. This issue will be handled by the next version of Windows Tools Installer.

dobairoland commented 1 year ago

@hierophect

While researching this issue, I was under the impression that idf versions later than 4.2 were not using curses at all? Why is it a prerequisite for 5.2?

It is not. The curses module is integrated into Python on Linux systems. I guess the selected Python version from pyenv is broken.

I'm not exactly a python guru so I'm not sure why the curses module fails with 3.10.6. However, it was the default version installed when installing the prerequisites. Should users be warned to manually install a specific python version?

Looking at your logs, the broken 3.10.6 also comes from pyenv. The default python3 on Ubuntu 22.04 is 3.10.4 (https://packages.ubuntu.com/jammy/python3). I don't think there is an issue with that. Otherwise we would have received other reports like this over the year.

Alternatively, while I'm not confident on this from my research, it seems possible that pyenv and the native installation of python3 simply broke while installing, somehow. I'm not sure the mechanics of how this would happen, especially with a fresh install on a new machine, so I'm also curious if this might be worth some kind of warning.

Officially we support the native python of the OS. https://packages.ubuntu.com/jammy/python3 in your case. However, pyenv should work as well - but it is not tested by us. Also, some versions included in pyenv could have issues. I'm sorry but we don't have control over that.

hierophect commented 1 year ago

Ok, I've figured this out. My core confusion was that this did appear to still be occurring on the system version of python3, which I had installed while following the ESP-IDF installation steps (this automatically installed version was 3.10.6, not 3.10.4, it's why I'd selected 3.10.6 for pyenv). When pyenv was completely disabled by removing the shims, which returned /usr/bin/python3 and ./install.sh was run again, the problem would still occur.

However, it looks like this was because install.sh does not remove or overwrite the existing python virtual environments, so after running it once with any version of 3.10 on your system, that version will be used by idf.py forever, regardless of what python version is currently active or available in the shell. It'll still say your local version was "detected", but it won't use it. I was able to resolve the issue by manually deleting the environment from .espressif.

I apologize for assuming this issue was broader than it ended up being, I'd thought same issue was occurring across two different installations of 3.10.6. Is there a good way to reset the python environments when running the IDF installation step?

dobairoland commented 1 year ago

It actually cleans the Python environment if any issue is detected. Your issue happened much later so it was not detected.

But I think we could add an import curses to test the environment on non-Windows systems. So lets keep this open until we do so.

hierophect commented 1 year ago

Great, I'm glad this issue ended up being of some use. Thanks for your insight.

espressif / esp-idf