Recreate a build from dependencies.lock file (PACMAN-623)

ljden commented 1 year ago

Related area

repeatable builds

Is your feature request related to a problem?

I need to keep a copy of the managed_components directory to recreate a previous build. If I build with an old dependencies.lock and no managed_components the idf manager will pull the latest components and update the lock file. There should be some way to recreate a build from a lock file without having the lock file be updated

Describe the solution you'd like

I would like to be able to recreate a previous build using only the dependencies.lock file, and not need to keep a copy of the managed_components directory

Describe alternatives you've considered

No response

Additional context

I couldn't see there being an option for this, if it is there and I have missed it let me know

I have checked existing list of Feature requests and I have provided enough information.

[X] I confirm I have checked existing list of Feature requests and I have provided enough information.

kumekay commented 1 year ago

@ljden Thank you for the issue. You are right component manager must not solve dependencies again if the lock file is up to date.

However, solving may be automatically triggered in a few cases when the environment is changed. Because solver results may vary on different ESP-IDF versions, it may be triggered automatically if you run it against a different ESP-IDF version than written in the lock file.

Starting commit https://github.com/espressif/idf-component-manager/commit/a52d55db50bcfd69f4cd913fc85619ff3babae44 component manager prints the reason why solver is running. However, a new component manager version with this version is not released yet. You may try installing component manager from the main branch and check the reason why solver runs in your case.

ljden commented 1 year ago

I would presume the solver is rerunning as the managed_components have been removed/not stored. The stated reason is "Manifest hash changed, solving dependencies."

Personally I think that you should be able to recreate a build without the managed_components as they can be re-pulled from their remote sources.

kumekay commented 1 year ago

@ljden

You are right in stating that dependencies.lock should be enough to reproduce the build, and it's an expected behavior.

I did an experiment to reproduce your situation. However, I wasn't able to reproduce the issue. When I move the project directory, the manifest hash stays the same.

Could you please check that problem exists in your case for a minimal project?

Here are my steps.

Create a project:

cd ~/t
idf.py create-project test
cd test
idf.py add-dependency example/cmp
idf.py reconfigure

Output:

Dependencies lock doesn't exist, solving dependencies.
.........Updating lock file at /Users/ku/t/test/dependencies.lock
Processing 2 dependencies:
[1/2] example/cmp (3.3.8)
[2/2] idf (5.1.0)

And the lock file:

dependencies:
  example/cmp:
    component_hash: 8c5891c21ecdebd1f545252144a5bf541071dce316f3e12eb288a5bcb506badf
    source:
      service_url: https://api.components.espressif.com/
      type: service
    version: 3.3.8
  idf:
    component_hash: null
    source:
      type: idf
    version: 5.1.0
manifest_hash: ff7214c6404c4f2e8356e01390590a4f0e4021c877b84ae0c41a743026e919f7
target: esp32
version: 1.0.0

Then move the project:

mv ~/t/test ~/t/test2
cd ~/t/test2
rm -rf build managed_components
idf.py reconfigure

But the solver is not triggered:

-- Building ESP-IDF components for target esp32
Processing 2 dependencies:
[1/2] example/cmp (3.3.8)
[2/2] idf (5.1.0)

The lock file is untouched.

manifest_hash: ff7214c6404c4f2e8356e01390590a4f0e4021c877b84ae0c41a743026e919f7

Few more questions about your environment:

Does the IDF version stay the same?
Do you have dependencies in your manifest file comping from a git repo?

ljden commented 1 year ago

The issue is more so when a component dependency has been updated.

Ie:

add git dependency
build
add new commit to remote git dependency
remove managed_components
rebuild

I would expect the component manager to try and pull the same commit hash as in the lock file, rather than pulling the latest master commit.

Predominantly our dependencies come from git sources (internally developed) and idf version stays the same

kumekay commented 1 year ago

@ljden Thank you for the details, I was able to reproduce it, and it's of course a bug. We will fix it in the next release. I'll update this issue as soon as is it will be merged into the main branch.

kumekay commented 1 year ago

@ljden this should be fixed in https://github.com/espressif/idf-component-manager/releases/tag/v1.3.0

https://docs.espressif.com/projects/idf-component-manager/en/latest/guides/updating_component_manager.html

espressif / idf-component-manager