idf.py fullclean shows ERROR in remove_managed_components (.gitignore support for hash calculations) (IDFGH-9024) (PACMAN-544)

AxelLin commented 1 year ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.1-dev-2509-gcfef24863f

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32C3DevKit-M1

Power Supply used.

USB

What is the expected behavior?

"idf.py fullclean" should work without any error.

What is the actual behavior?

After upgrade to v5.1-dev-2509-gcfef24863f, now everytime executing "idf.py fullclean" shows below error: I didn't change anythin in managed_components folder, and this error does not happen before upgrading esp-idf.

$ idf.py fullclean Executing action: fullclean Executing action: remove_managed_components ERROR: Some components (joltwallet__littlefs) in the "managed_components" directory were modified on the disk since the last run of the CMake. Content of this directory is managed automatically.

    If you want to keep the changes, you can move the directory with the component to the "components"         directory of your project.

    I.E. for "joltwallet__littlefs" run:
    mv /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs /home/axel/esp/esp-idf-dev/apps/myproj/components/joltwallet__littlefs

    Or, if you want to discard the changes remove the ".component_hash" file from the component's directory.

    I.E. for "joltwallet__littlefs" run:
    rm /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs/.component_hash

Steps to reproduce.

idf.py build ; idf.py fullclean

Debug Logs.

No response

More Information.

No response

kumekay commented 1 year ago

Hi @AxelLin could you please provide more details about your environment to help us reproduce this problem:

Linux Distro with version
File system type
Content of dependencies.lock

Do you experience the problem only with this component or with other too?

AxelLin commented 1 year ago

$ lsb_release -a
LSB Version:    core-11.1.0ubuntu4-noarch:printing-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:    22.04
Codename:   jammy

filesystem: ext4

$ cat dependencies.lock 
dependencies:
  idf:
    component_hash: null
    source:
      type: idf
    version: 5.1.0
  joltwallet/littlefs:
    component_hash: 2fa0ef8823451b00cec4eec1a18f810e9267b1f588513a64526be456c8a9d59a
    source:
      service_url: https://api.components.espressif.com/
      type: service
    version: 1.5.1
manifest_hash: c14ee59777e03272dc787e6a72f0e5ccf67cf0c3820f182730fbe2e28d652138
target: esp32c3
version: 1.0.0

AxelLin commented 1 year ago

Do you experience the problem only with this component or with other too?

I only use littlefs, so I'm not sure if other components have issue or not.

$ cat main/idf_component.yml
## IDF Component Manager Manifest File
dependencies:
  joltwallet/littlefs: "^1.4.0"
  ## Required IDF version
  idf:
    version: ">=4.1.0"
  # # Put list of dependencies here
  # # For components maintained by Espressif:
  # component: "~1.0.0"
  # # For 3rd party components:
  # username/component: ">=1.0.0,<2.0.0"
  # username2/component2:
  #   version: "~1.0.0"
  #   # For transient dependencies `public` flag can be set.
  #   # `public` flag doesn't have an effect dependencies of the `main` component.
  #   # All dependencies of `main` are public by default.
  #   public: true

kumekay commented 1 year ago

@AxelLin thank you so much,

Could you please also share the content of managed_components/joltwallet__littlefs/.component_hash file?

AxelLin commented 1 year ago

$ cat managed_components/joltwallet__littlefs/.component_hash 2fa0ef8823451b00cec4eec1a18f810e9267b1f588513a64526be456c8a9d59a

BTW, everytime I execute "idf.py menuconfig" it will show message to ask deleting managed_components/joltwallet__littlefs/.component_hash.

I delete it, then execute "idf.py menuconfig". It will generate exactly the same content again: $ cat managed_components/joltwallet__littlefs/.component_hash 2fa0ef8823451b00cec4eec1a18f810e9267b1f588513a64526be456c8a9d59a

AxelLin commented 1 year ago

It was working with v5.1-dev-2356-gedd815af2e, and fails after upgrade to v5.1-dev-2495-ge5926d1b1b. BTW, ./install.sh updates below packages, I'm not sure if it's related or not. just FYI.

Downloading https://dl.espressif.com/pypi/idf-component-manager/idf_component_manager-1.2.1-py3-none-any.whl (116 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.1/116.1 kB 2.7 MB/s eta 0:00:00 Downloading https://dl.espressif.com/pypi/esptool/esptool-4.5.dev0-py3-none-any.whl (310 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 310.2/310.2 kB 4.8 MB/s eta 0:00:00
Downloading https://dl.espressif.com/pypi/packaging/packaging-22.0-py3-none-any.whl (42 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 4.8 MB/s eta 0:00:00

kumekay commented 1 year ago

@AxelLin Thank you so much. We were able to reproduce the problem.

While a proper solution is on its way, could you please stick to an older version of the component manager by running

pip install idf-component-manager==1.1.4

in your ESP-IDF environment (after . $IDF_PATH/export.sh)?

We will move this issue to http://github.com/espressif/idf-component-manager project.

AxelLin commented 1 year ago

Hi @kumekay https://github.com/espressif/idf-component-manager/issues/15#issuecomment-1364633783 does not work, I got another error: ''' -- Building ESP-IDF components for target esp32c3 CMake Error at /home/axel/esp/esp-idf/tools/cmake/build.cmake:542 (message): usage: main.py [-h] [--project_dir PROJECT_DIR]

                 [--interface_version {0,1}]
                 {prepare_dependencies,inject_requirements,inject_requrements}
                 ...

main.py: error: argument --interface_version: invalid choice: 2 (choose from 0, 1)

Call Stack (most recent call first): /home/axel/esp/esp-idf/tools/cmake/project.cmake:447 (idf_build_process) CMakeLists.txt:26 (project)

-- Configuring incomplete, errors occurred! ''' I give up using master for now (hope to get it fix soon).

kumekay commented 1 year ago

@AxelLin Yes, unfortunately, after https://github.com/espressif/esp-idf/commit/17224f325450a001c4744064470cb6813b1c8514 master branch requires component-manager 1.2+

You can use the commit of ESP-IDF before this.

The solution to the problem is not trivial and takes time. We will release it by next week.

kumekay commented 1 year ago

A problem in https://github.com/espressif/idf-flash-vendor-patches/issues/1#issuecomment-1252470782 is probably related to this one, so I'm mentioning the comment.

AxelLin commented 1 year ago

@AxelLin Yes, unfortunately, after espressif/esp-idf@17224f3 master branch requires component-manager 1.2+

You can use the commit of ESP-IDF before this.

The solution to the problem is not trivial and takes time. We will release it by next week.

@kumekay Please don't block the users due to this issue. If it really takes time to fix the issue properly, please release a temporary fix first.

kumekay commented 1 year ago

@AxelLin, I apologize for the delay in addressing this issue. We are committed to releasing a solution by tomorrow.

Update: 1.2.2 release is planned for Friday, 13.01

kumekay commented 1 year ago

@AxelLin

Before we release, would you mind trying the version of the component manager from the branch bugfix/default_excludes to confirm that it resolves the issue for you?

You can install it with this command:

pip install git+https://github.com/espressif/idf-component-manager.git@bugfix/default_excludes -U

kumekay commented 1 year ago

@AxelLin

I apologize for not mentioning it in my previous comment, but before installing from git, it is necessary to remove the previous version from PyPI.

However, I am pleased to inform you that a new bugfix release has been made available on PyPI. You can update to v1.2.2 either by running ./install.sh from the ESP-IDF directory or by running pip install -U idf-component-manager within the ESP-IDF environment.

AxelLin commented 1 year ago

Hi @kumekay It works now, thanks a lot.

AxelLin commented 10 months ago

@kumekay

I'm using v5.1.1-1-gd3c99ed3b8 to build for esp32c3.

$ idf.py fullclean build Executing action: fullclean Executing action: remove_managed_components ERROR: Some components (joltwalletlittlefs) in the "managed_components" directory were modified on the disk since the last run of the CMake. Content of this directory is managed automatically. If you want to keep the changes, you can move the directory with the component to the "components"directory of your project. I.E. for "joltwallet__littlefs" run: mv /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwalletlittlefs /home/axel/esp/esp-idf-dev/apps/myproj/components/joltwalletlittlefs Or, if you want to discard the changes remove the ".component_hash" file from the component's directory. I.E. for "joltwallet__littlefs" run: rm /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwalletlittlefs/.component_hash axel@alpha:~/esp/esp-idf-dev/apps/myproj$ cat /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs/.component_hash 2e392b279f0bc282a9dfb66608de4b5351d5ab4249254cbb5ebae09099c045ec

But I didn't touch anything in managed_components folder. Note: v5.2-dev-2383-g82cceabc6e also has the same issue.

kumekay commented 10 months ago

Hello @AxelLin

It looks like it's the same problem as in https://github.com/espressif/idf-component-manager/issues/40

For now, you can use IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS=1 as a quick workaround.

AxelLin commented 10 months ago

@kumekay When would you fix this?

kumekay commented 10 months ago

Hello @AxelLin

We are still looking for the best approach to dealing with this problem maintaining guarantees on reproducibility that component manager promises.

We will address it in the coming release

kumekay commented 9 months ago

Hello @AxelLin the files that littlefs generates during build are already gitignored and can be excluded to resolve the problem.

Unfortunately, we can only do it for coming ESP-IDF version 5.2. For existing versions of IDF, we can exclude filters to the idf_component.yml of the littlefs. So for this particular component, will do our best to fix it next week.

AxelLin commented 6 months ago

Hi @kumekay

This is probably off-topic from the origianl issue, I'm wondering if you can help to answer it: (I asked the question in ep32 forum https://www.esp32.com/viewtopic.php?f=13&t=37005, but unforunately no response there.)

While debugging, sometimes I would like to add some debug code in the managed component files, but it seems the idf.py build always build with a clean cloned managed component files. Is it possible to make some local changes in the managed component files and build with the local changes? This would be helpful to debug issues in the managed component files.

kumekay commented 6 months ago

Hi @AxelLin You can move a component from managed_components directory of your project to components directory and modify it there. Then IDF will use the component from components directory and component manager won't make a copy in the managed_components

espressif / idf-component-manager