search

espressif / idf-component-manager

Tool for installing ESP-IDF components
https://components.espressif.com/
Apache License 2.0
42 stars 15 forks source link

No internet, fails idf.py build (PACMAN-612) #27

Closed ligantx closed 1 year ago

ligantx commented 1 year ago

The Component Manager version

v1.2.2

ESP-IDF Version

ESP-IDF v5.1-dev-3797-gaf805df3cb

python Version

3.8.10

Operating System

docker esp-idf container

Browser (for https://components.espressif.com Issues)

wget

Description

I want to use a remote machine for building using esp-idf docker image. In my local machine (with internet access): idf.py set-target esp32s3 and then i transfer all my files except build folder to the docker container. The docker machine is isolated from internet access, so when i try to idf.py build, i get the error:

Are you connected to the internet?

Call Stack (most recent call first):
  /opt/esp/idf/tools/cmake/project.cmake:453 (idf_build_process)   
  CMakeLists.txt:6 (project)

-- Configuring incomplete, errors occurred!

Then if inside the container i run export IDF_COMPONENT_MANAGER=0 and rerun idf.py build (blink example) i got the error 

No such file or directory 
   14 | #include "led_strip.h"
      |          ^~~~~~~~~~~~~

although, managed_components include led_strip code downloaded and transferred from my local machine.

Is there an easy way to include all managed_components code if there is no internet access?

This works (with export IDF_COMPONENT_MANAGER=0):

dependencies:
  espressif/led_strip: 
    version: "^2.0.0"
    path: /project/managed_components/espressif__led_strip/

but i have to modify my idf_component.yml file every time i want to build and put the path of every dependency separately. Can i just add all the managed_components file with IDF_COMPONENT_MANAGER=0 variable?

To Reproduce

idf.py set-target esp32s3
Cut internet connection
idf.py build

Expected behaviour

use a special flag, so if there is no internet connection the local managed_components folder will be used

Additional info

No response

I have checked existing issues and online Documentation

kumekay commented 1 year ago

Hi @ligantx Thank you for opening the issue. Once components are solved and dependencies are downloaded component manager should not try to make network connections, so this behavior looks incorrect. So let me double-check a couple of details:

ligantx commented 1 year ago

Thanks for the quick response. I rechecked and here are the steps i followed:

  1. set-target locally. works
  2. don't touch IDF_COMPONENET_MANAGER or idf_component.yml file
  3. send all files except build folder to the docker container
  4. run idf.py from the container: get the "Are you connected to the internet?" issue
  5. idf version locally: ESP-IDF v5.0.1. In the server ESP-IDF v5.1-dev-3797-gaf805df3cb
kumekay commented 1 year ago

@ligantx The problem is you have different versions of IDF. idf_component.yml may contain rules to include or exclude dependencies based on the current ESP-IDF version or target. This is why it tries to solve dependencies again in the container.

If you'll check dependencies.lock file, it states the IDF version:

  idf:
    ...
    version: 5.0.1`

So the simplest way is to use the same version of ESP-IDF.

What are workarounds if you cannot use the same ESP-IDF versions:

  1. You can have multiple lock files in IDF master since https://github.com/espressif/esp-idf/commit/17224f325450a001c4744064470cb6813b1c8514. It's possible to add DEPENDENCIES_LOCK in CMakeLists.txt and generate lock files for every component version
  2. Going with IDF_COMPONENT_MANAGER=0 is possible without modification of manifest files. In this case you need to copy content of managed_components to your components directory in container.
ligantx commented 1 year ago

"So the simplest way is to use the same version of ESP-IDF." I see, although i guess one solution is to modify the idf version to: >=5.0.0 in dependencies.lock as stated here

"Going with IDF_COMPONENT_MANAGER=0 is possible without modification of manifest files. In this case you need to copy content of managed_components to your components directory in container." I tried: 

mkdir -p components
cp -r managed_components/* components/.
ls components/

returns espressif__led_strip component folder and IDF_COMPONENT_MANAGER=0 But, still idf.py build doesn't include the component.

A solution i found is to edit CMakeLists.txt adding set(EXTRA_COMPONENT_DIRS managed_components) that worked, but a more clean solution -if exists- would be better.

kumekay commented 1 year ago

The lock file dependencies.lock is automatically generated, and it is not supposed to be modified by a user. The manifest file idf_component.yml may contain version ranges, but dependencies.lock only contains exact versions.

Unfortunately, I wasn't able to reproduce your problem with IDF_COMPONENT_MANAGER=0, it worked just fine when espressif__led_strip is in the components. But going with EXTRA_COMPONENT_DIRS should be usable too.

To help fix such situations in the future, we will add a message to the solver with the exact reason why did it decide to solve components.

ligantx commented 1 year ago

My idf_component.yml file contains:

dependencies:
  espressif/led_strip: 
    version: "^2.0.0"

Even changed like:

dependencies:
  idf: ">=4.0"
  espressif/led_strip: 
    version: "^2.0.0"

it couldnt work with:

  1. locally idf.py set-target esp32s3
  2. transfer everything except build folder
  3. idf.py build inside container created with IDF_COMPONENT_MANAGER=0 and without it
  4. Tried renaming mv managed_components/ components/ but no changes.

The only solution i found is IDF_COMPONENT_MANAGER=0 and set(EXTRA_COMPONENT_DIRS managed_components) in CMakeLists.txt

To help fix such situations in the future, we will add a message to the solver with the exact reason why did it decide to solve components.

Yes, this is a good idea. Also, maybe let user download "manually" the packages from https://components.espressif.com/ to a custom named folder, so the manager can find locally instead of download.

ligantx commented 1 year ago

@kumekay I have some updates on the issue: I managed to give access to the remote machine (and docker container also) to https://components.espressif.com/ and https://api.components.espressif.com/. wget https://components.espressif.com/ and wget https://api.components.espressif.com/ from inside the container works fine. I create my container (with the default IDF_COMPONENT_MANAGER value) like this: docker run -dit -v /mylocalfolder/project:/project -w /project --name test espressif/idf:latest"

So if from inside container i do: idf.py set-target esp32s3 i get:

Adding "set-target"'s dependency "fullclean" to list of commands with default set of options.
Executing action: fullclean
Build directory '/project/build' not found. Nothing to clean.
Executing action: set-target
Set Target to: esp32s3, new sdkconfig created. Existing sdkconfig renamed to sdkconfig.old.
Running cmake in directory /project/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/opt/esp/python_env/idf5.1_py3.8_env/bin/python -DESP_PLATFORM=1 -DIDF_TARGET=esp32s3 -DCCACHE_ENABLE=1 /project"...
-- Found Git: /usr/bin/git (found version "2.25.1")
-- ccache will be used for faster recompilation
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32s3
Solving dependencies requirements
CMake Error at /opt/esp/idf/tools/cmake/build.cmake:542 (message):
  ERROR: Cannot establish a connection to the component registry.  Are you
  connected to the internet?

Call Stack (most recent call first):
  /opt/esp/idf/tools/cmake/project.cmake:453 (idf_build_process)
  CMakeLists.txt:6 (project)

-- Configuring incomplete, errors occurred!
See also "/project/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1, output of the command is in the /project/build/log/idf_py_stderr_output_204 and /project/build/log/idf_py_stdout_output_204

and with tail -100 /project/build/CMakeFiles/CMakeOutput.log i get:

Parsed CXX implicit link information from above output:
  link line regex: [^( *|.*[/\])(xtensa-esp32s3-elf-ld|CMAKE_LINK_STARTFILE-NOTFOUND|([^/\]+-)?ld|collect2)[^/\]*( |$)]
  ignore line: [Change Dir: /project/build/CMakeFiles/CMakeTmp]
  ignore line: []
  ignore line: [Run Build Command(s):/usr/bin/ninja cmTC_a92c7 && [1/2] Building CXX object CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj]
  ignore line: [Using built-in specs.]
  ignore line: [COLLECT_GCC=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-g++]
  ignore line: [Target: xtensa-esp32s3-elf]
  ignore line: [Configured with: /builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/src/gcc/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=xtensa-esp32s3-elf --prefix=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf --exec_prefix=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf --with-local-prefix=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf/xtensa-esp32s3-elf --with-headers=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf/xtensa-esp32s3-elf/include --with-newlib --enable-threads=no --disable-shared --with-pkgversion='crosstool-NG esp-12.2.0_20230208' --disable-__cxa_atexit --enable-cxx-flags=-ffunction-sections --disable-libgomp --disable-libmudflap --disable-libmpx --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libstdcxx-verbose --with-gmp=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --with-mpfr=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --with-mpc=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --with-isl=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --enable-lto --enable-target-optspace --without-long-double-128 --disable-nls --enable-multiarch --enable-languages=c c++ --disable-libstdcxx-verbose --enable-threads=posix --enable-gcov-custom-rtio --enable-libstdcxx-time=yes]
  ignore line: [Thread model: posix]
  ignore line: [Supported LTO compression algorithms: zlib]
  ignore line: [gcc version 12.2.0 (crosstool-NG esp-12.2.0_20230208) ]
  ignore line: [COLLECT_GCC_OPTIONS='-mlongcalls' '-v' '-o' 'CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj' '-c' '-dumpdir' 'CMakeFiles/cmTC_a92c7.dir/']
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/cc1plus -quiet -v -iprefix /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/ /opt/esp/tools/cmake/3.24.0/share/cmake-3.24/Modules/CMakeCXXCompilerABI.cpp -quiet -dumpdir CMakeFiles/cmTC_a92c7.dir/ -dumpbase CMakeCXXCompilerABI.cpp.cpp -dumpbase-ext .cpp -mlongcalls -version -o /tmp/ccpECh3H.s]
  ignore line: [GNU C++17 (crosstool-NG esp-12.2.0_20230208) version 12.2.0 (xtensa-esp32s3-elf)]
  ignore line: [        compiled by GNU C version 4.9.2  GMP version 6.2.1  MPFR version 4.1.0  MPC version 1.2.1  isl version isl-0.24-GMP]  ignore line: []
  ignore line: [GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include/c++/12.2.0"]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include/c++/12.2.0/xtensa-esp32s3-elf"]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include/c++/12.2.0/backward"]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/include"]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/include-fixed"]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/sys-include"]
  ignore line: [ignoring duplicate directory "/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/../../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include"]
  ignore line: [#include "..." search starts here:]
  ignore line: [#include <...> search starts here:]
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include/c++/12.2.0]
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include/c++/12.2.0/xtensa-esp32s3-elf]
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include/c++/12.2.0/backward]
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/include] 
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/include-fixed]
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/sys-include]
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/include]
  ignore line: [End of search list.]
  ignore line: [GNU C++17 (crosstool-NG esp-12.2.0_20230208) version 12.2.0 (xtensa-esp32s3-elf)]
  ignore line: [        compiled by GNU C version 4.9.2  GMP version 6.2.1  MPFR version 4.1.0  MPC version 1.2.1  isl version isl-0.24-GMP]  ignore line: []
  ignore line: [GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072]
  ignore line: [Compiler executable checksum: 79a1e6018e5d3fdbef5e02b3475aa2a9]
  ignore line: [COLLECT_GCC_OPTIONS='-mlongcalls' '-v' '-o' 'CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj' '-c' '-dumpdir' 'CMakeFiles/cmTC_a92c7.dir/']
  ignore line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/bin/as --traditional-format --longcalls -o CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj /tmp/ccpECh3H.s]     
  ignore line: [COMPILER_PATH=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/bin/]
  ignore line: [LIBRARY_PATH=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib/]
  ignore line: [COLLECT_GCC_OPTIONS='-mlongcalls' '-v' '-o' 'CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj' '-c' '-dumpdir' 'CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.']
  ignore line: [[2/2] Linking CXX executable cmTC_a92c7]
  ignore line: [Using built-in specs.]
  ignore line: [COLLECT_GCC=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-g++]
  ignore line: [COLLECT_LTO_WRAPPER=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/lto-wrapper]
  ignore line: [Target: xtensa-esp32s3-elf]
  ignore line: [Configured with: /builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/src/gcc/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=xtensa-esp32s3-elf --prefix=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf --exec_prefix=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf --with-local-prefix=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf/xtensa-esp32s3-elf --with-headers=/builds/idf/crosstool-NG/builds/xtensa-esp32s3-elf/xtensa-esp32s3-elf/include --with-newlib --enable-threads=no --disable-shared --with-pkgversion='crosstool-NG esp-12.2.0_20230208' --disable-__cxa_atexit --enable-cxx-flags=-ffunction-sections --disable-libgomp --disable-libmudflap --disable-libmpx --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libstdcxx-verbose --with-gmp=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --with-mpfr=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --with-mpc=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --with-isl=/builds/idf/crosstool-NG/.build/xtensa-esp32s3-elf/buildtools --enable-lto --enable-target-optspace --without-long-double-128 --disable-nls --enable-multiarch --enable-languages=c c++ --disable-libstdcxx-verbose --enable-threads=posix --enable-gcov-custom-rtio --enable-libstdcxx-time=yes]
  ignore line: [Thread model: posix]
  ignore line: [Supported LTO compression algorithms: zlib]
  ignore line: [gcc version 12.2.0 (crosstool-NG esp-12.2.0_20230208) ]
  ignore line: [COMPILER_PATH=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/bin/]
  ignore line: [LIBRARY_PATH=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/:/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib/]
  ignore line: [COLLECT_GCC_OPTIONS='-mlongcalls' '-v' '-o' 'cmTC_a92c7' '-dumpdir' 'cmTC_a92c7.']
  link line: [ /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/collect2 -plugin /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/liblto_plugin.so -plugin-opt=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccPL7glY.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lnosys -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -o cmTC_a92c7 /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib/crt0.o /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crti.o /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtbegin.o -L/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0 -L/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc -L/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj -lstdc++ -lm -lgcc -lc -lnosys -lc -lgcc /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtend.o /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtn.o]
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/collect2] ==> ignore
    arg [-plugin] ==> ignore
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/liblto_plugin.so] ==> ignore
    arg [-plugin-opt=/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../libexec/gcc/xtensa-esp32s3-elf/12.2.0/lto-wrapper] ==> ignore
    arg [-plugin-opt=-fresolution=/tmp/ccPL7glY.res] ==> ignore
    arg [-plugin-opt=-pass-through=-lgcc] ==> ignore
    arg [-plugin-opt=-pass-through=-lc] ==> ignore
    arg [-plugin-opt=-pass-through=-lnosys] ==> ignore
    arg [-plugin-opt=-pass-through=-lc] ==> ignore
    arg [-plugin-opt=-pass-through=-lgcc] ==> ignore
    arg [-o] ==> ignore
    arg [cmTC_a92c7] ==> ignore
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib/crt0.o] ==> obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib/crt0.o]
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crti.o] ==> obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crti.o]
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtbegin.o] ==> obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtbegin.o]
    arg [-L/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0] ==> dir [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0]
    arg [-L/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc] ==> dir [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc]
    arg [-L/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib] ==> dir [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib]
    arg [CMakeFiles/cmTC_a92c7.dir/CMakeCXXCompilerABI.cpp.obj] ==> ignore
    arg [-lstdc++] ==> lib [stdc++]
    arg [-lm] ==> lib [m]
    arg [-lgcc] ==> lib [gcc]
    arg [-lc] ==> lib [c]
    arg [-lnosys] ==> lib [nosys]
    arg [-lc] ==> lib [c]
    arg [-lgcc] ==> lib [gcc]
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtend.o] ==> obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtend.o]
    arg [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtn.o] ==> obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtn.o]
  collapse obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib/crt0.o] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/xtensa-esp32s3-elf/lib/crt0.o]
  collapse obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crti.o] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crti.o]
  collapse obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtbegin.o] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crtbegin.o]
  collapse obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtend.o] 
==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crtend.o]
  collapse obj [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/crtn.o] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crtn.o]
  collapse library dir [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0]
  collapse library dir [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc]
  collapse library dir [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/lib] ==> [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/xtensa-esp32s3-elf/lib]     
  implicit libs: [stdc++;m;gcc;c;nosys;c;gcc]
  implicit objs: [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/xtensa-esp32s3-elf/lib/crt0.o;/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crti.o;/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crtbegin.o;/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crtend.o;/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0/crtn.o]
  implicit dirs: [/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc/xtensa-esp32s3-elf/12.2.0;/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/lib/gcc;/opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/xtensa-esp32s3-elf/lib]
  implicit fwks: []

/opt/esp/idf/tools/cmake/build.cmake line 542 is: image and 453: image

Any ideas? Should i whitelist more resources other than https://components.espressif.com/ and https://api.components.espressif.com/?

kumekay commented 1 year ago

@ligantx

Yes, the component archives and metadata during dependencies resolution are downloaded from CDN https://components-file.espressif.com/, while https://components.espressif.com/api is mainly used for modifying data in the registry.

kumekay commented 1 year ago

https://api.components.espressif.com/ is a legacy URL, recent versions of the component manager don't try accessing it.

kumekay commented 1 year ago

To help fix such situations in the future, we will add a message to the solver with the exact reason why did it decide to solve components.

The reason for the solver to run was added here https://github.com/espressif/idf-component-manager/commit/a52d55db50bcfd69f4cd913fc85619ff3babae44 and will be included in 1.3.0 release

ligantx commented 1 year ago

@ligantx

Yes, the component archives and metadata during dependencies resolution are downloaded from CDN https://components-file.espressif.com/, while https://components.espressif.com/api is mainly used for modifying data in the registry.

I'm not sure I understand. From my log files does it look like the manager cannot achieve connection to the online resources? The server has already https://components.espressif.com/ and https://api.components.espressif.com/ whitelisted. Should I also add https://components-file.espressif.com/?

kumekay commented 1 year ago

@ligantx Yes, component archives are always downloaded from https://components-file.espressif.com/, so it should be allowed.

ligantx commented 1 year ago

To help fix such situations in the future, we will add a message to the solver with the exact reason why did it decide to solve components.

The reason for the solver to run was added here a52d55d and will be included in 1.3.0 release

@kumekay How i can update to this commit? I cannot find idf-component-manager .git folder.

I've run pip install idf-component-manager --upgrade and i'm in the latest version.

Also, i tried to delete dependencies.py file and wget https://raw.githubusercontent.com/espressif/idf-component-manager/a52d55db50bcfd69f4cd913fc85619ff3babae44/idf_component_manager/dependencies.py but i get "Please use idf.py only in an ESP-IDF shell environment." when i run idf.py set-target esp32s3 even if i run the /opt/esp/entrypoint.sh script first. (with the default dependencies.py file, idf.py commands run fine).

kumekay commented 1 year ago

@ligantx

I added instructions on how to install a development version here: https://github.com/espressif/idf-component-manager#installing-a-development-version-of-the-component-manager

ligantx commented 1 year ago

@kumekay Thank you for the simple instructions.

So, now i get the following:

Manifest hash changed, solving dependencies.
CMake Error at /opt/esp/idf/tools/cmake/build.cmake:542 (message):
  ERROR: Solver failed processing dependency "espressif/led_strip" from the
  manifest file "/project/main/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/led_strip.json (Caused by
  SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
  certificate verify failed: unable to get local issuer certificate
  (_ssl.c:1131)')))

Call Stack (most recent call first):
  /opt/esp/idf/tools/cmake/project.cmake:453 (idf_build_process)
  CMakeLists.txt:7 (project)

-- Configuring incomplete, errors occurred!
See also "/project/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1, output of the command is in the /project/build/log/idf_py_stderr_output_736 and /project/build/log/idf_py_stdout_output_736

I guess this caused by components-file.espressif.com being blacklisted?

kumekay commented 1 year ago

I guess this caused by components-file.espressif.com being blacklisted?

@ligantx Yes, it looks so

ligantx commented 1 year ago

@kumekay After whitelisting components-file.espressif.com everything works fine.. Thanks