Full Clone of ESP Matter SDK takes lot of time to pull all submodules and consumes GB's of memory in laptop (CON-383)

[aggaddam]

Hi,

I downloaded the latest ESP Matter SDK version (using the shallow clone and without any install commands). I uploaded the matter SDK as it is into our git repository hosted in bitbucket. When I tried to download the Esp Matter SDK from our bitbucket repo and tried to install matter tools by running ./install.sh in the esp-matter folder, I got the error saying [not a valid git repository](fatal: not a git repository: /esp-matter/connectedhomeip/connectedhomeip/../../.git/modules/connectedhomeip/connectedhomeip). If I clone ESP Matter SDK manually from the git hub, it works fine. Not sure What am I missing here. Any thoughts?

git clone --depth 1 https://github.com/espressif/esp-matter.git
cd esp-matter
git submodule update --init --depth 1
cd connectedhomeip/connectedhomeip
./scripts/checkout_submodules.py --platform esp32 --shallow

I tried with the full clone of ESP matter SDK which takes almost 14 GB of disk memory and almost takes 20-30 minutes git clone --recursive https://github.com/espressif/esp-matter.git

Is there any way we can clone the light version of ESP matter SDK by stripping off unnecessary modules/third-party libraries I want to upload this light version of ESP matter SDK in to bitbucket repository.

[wqx6]

I didn't meet the error [not a valid git repository](fatal: not a git repository: /esp-matter/connectedhomeip/connectedhomeip/../../.git/modules/connectedhomeip/connectedhomeip), Currently we have an issue for install.sh, which is that we does not checkout submodule libwebsockets required by chip-tool.

I am not sure why you have the error. Here are my steps:

git clone --depth 1 https://github.com/espressif/esp-matter.git
cd esp-matter
git submodule update --init --depth 1
cd connectedhomeip/connectedhomeip
./scripts/checkout_submodules.py --platform esp32 --shallow
git submodule update --init third_party/libwebsockets/repo
cd ../..
./install.sh

[dhrishi]

I downloaded the latest ESP Matter SDK version (using the shallow clone and without any install commands).

How did you download this? If you download it as a zip, then it won't have git and none of the git commands would work. Hence the error is expected. To confirm, please check for .git folder inside esp-matter.

Edit: If you didn't download a zip, please mention the exact commands used so that we can try out

[aggaddam]

@dhrishi Here are the steps

1. Clone the latest ESP-Matter SDK by running the below commands 'git clone --depth 1 https://github.com/espressif/esp-matter.git cd esp-matter git submodule update --init --depth 1 cd connectedhomeip/connectedhomeip ./scripts/checkout_submodules.py --platform esp32 --shallow`

2. I can see.git folder inside the esp-matter folder after the clone. Now, I am pushing this ESP-Matter SDK to the bitbucket repo for our development

3. Now, clone the ESP-Matter SDK from the bitbucket repo and try to run ./install.sh. It shows failure. (I also noticed the .git folder is also not created when cloned from the bitbucket repo)

(Note: I can install all the tools and run the examples if cloned directly from GitHub i.e. as I mentioned in step 1)

[shubhamdp]

Files starting with . are hidden files, and all the git config are inside .git, I guess when you ran git add command it failed to add those files and thats why you are seeing it as that error.

Why don't you add the esp-matter as a submodule? It would save space as well as easy to maintain.

[aggaddam]

@shubhamdp can you provide steps for linking specific version/commit of esp-matter SDK as submodule?

Currently, I am running shell script to clone the ESP-matter SDK

[shubhamdp]

You can find the detailed documentation here https://git-scm.com/book/en/v2/Git-Tools-Submodules.

git submodule add https://github.com/espressif/esp-matter.git
# check git status, there will be two files staged to commit, `.gitmodules` and esp-matter (submodule)
# commit them 

[aggaddam]

@shubhamdp Thanks for the details.

We are using a shallow clone for ESP32 platform so that required modules will be pulled out from github. otherwise, it takes a lot of time to clone the full repo of the ESP matter SDK

Here is the script we are using right now for shallow clone

` git clone --depth 1 https://github.com/espressif/esp-matter.git cd esp-matter git submodule update --init --depth 1 cd connectedhomeip/connectedhomeip ./scripts/checkout_submodules.py --platform esp32 --shallow git submodule update --init third_party/libwebsockets/repo

`

[shubhamdp]

This is what we have been suggesting in our programming guide. What is the problem with above steps? And I cannot think a way to optimize this further right now.

One think that you can do is, since .git folder will be big in size you can download a zip for source code and then unpack it and upload that, but tracking the changes will be upto you. If you use the submodule, it will increase the size but updating the esp-matter code in your repo is pretty simple and straight forward.

[aggaddam]

@shubhamdp I tried downloading the zip version of IDF and ESP-Matter SDK and extracting the files and trying to run the example code inside esp-idf folder. But I am getting below error while setting target to esp32c3

Adding "set-target"'s dependency "fullclean" to list of commands with default set of options.
Executing action: fullclean
Build directory '/home/agaddam/Documents/espidfrelease_v51/examples/get-started/blink/build' not found. Nothing to clean.
Executing action: set-target
Set Target to: esp32c3, new sdkconfig will be created.
Running cmake in directory /home/agaddam/Documents/espidfrelease_v51/examples/get-started/blink/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/home/agaddam/.espressif/python_env/idf5.1_py3.10_env/bin/python -DESP_PLATFORM=1 -DIDF_TARGET=esp32c3 -DCCACHE_ENABLE=0 /home/agaddam/Documents/espidfrelease_v51/examples/get-started/blink"...
-- Found Git: /usr/bin/git (found version "2.34.1") 
fatal: not a git repository (or any of the parent directories): .git
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /home/agaddam/.espressif/tools/riscv32-esp-elf/esp-12.2.0_20230208/riscv32-esp-elf/bin/riscv32-esp-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/agaddam/.espressif/tools/riscv32-esp-elf/esp-12.2.0_20230208/riscv32-esp-elf/bin/riscv32-esp-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/agaddam/.espressif/tools/riscv32-esp-elf/esp-12.2.0_20230208/riscv32-esp-elf/bin/riscv32-esp-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32c3
Solving dependencies requirements
......Updating lock file at /home/agaddam/Documents/espidfrelease_v51/examples/get-started/blink/dependencies.lock
Processing 2 dependencies:
[1/2] espressif/led_strip (2.3.0)
[2/2] idf (5.1.0)
CMake Error at /home/agaddam/Documents/espidfrelease_v51/tools/cmake/component.cmake:224 (message):
  CMake Warning (dev) at build_properties.temp.cmake:8:

    Syntax Warning in cmake code at column 47

    Argument not separated from preceding token by whitespace.

  Call Stack (most recent call first):

    /home/agaddam/Documents/espidfrelease_v51/tools/cmake/scripts/component_get_requirements.cmake:3 (include)

  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Error at
  /home/agaddam/Documents/espidfrelease_v51/components/mqtt/CMakeLists.txt:1
  (include):

    include could not find requested file:

      /home/agaddam/Documents/espidfrelease_v51/components/mqtt/esp-mqtt/CMakeLists.txt

  Call Stack (most recent call first):

    /home/agaddam/Documents/espidfrelease_v51/tools/cmake/scripts/component_get_requirements.cmake:106 (include)
    /home/agaddam/Documents/espidfrelease_v51/tools/cmake/scripts/component_get_requirements.cmake:124 (__component_get_requirements)

Call Stack (most recent call first):
  /home/agaddam/Documents/espidfrelease_v51/tools/cmake/build.cmake:573 (__component_get_requirements)
  /home/agaddam/Documents/espidfrelease_v51/tools/cmake/project.cmake:464 (idf_build_process)
  CMakeLists.txt:6 (project)

-- Configuring incomplete, errors occurred!

[dhrishi]

@aggaddam Were you able to solve this issue?

[aggaddam]

@dhrishi It was not solved. I raised a separate issue in esd-idf repository as it is more related to esp-idf SDK.

Here is the link https://github.com/espressif/esp-idf/issues/11152

[dhrishi]

Okay. Thanks! If it is being tracked there, would you mind closing it here?

[aggaddam]

Please close the issue here.