Dependency resolution (PACMAN-568)

[ljden]

The Component Manager version

v1.1.4

ESP-IDF Version

v5.0

python Version

3.10.6

Operating System

Linux

Browser (for https://components.espressif.com Issues)

No response

Description

project has dependencies A and B Component A has dependency B

B requires modification so project idf_component.yml is updated from git to local path

Building project results in dependency resolution of: [1/3] B (version number) [2/3] B (hash) [3/3] A (hash)

dependencies.lock has B->source->type: git

It is unclear as to why the component manager decided that the git version of B was the necessary component to satisfy the requirement. Given the project requires the local copy I would expect that either the build should fail with conflicting dependencies or the resolution ends with the local copy being used.

To Reproduce

can provide an example if required

Expected behaviour

see above

Additional info

No response

I have checked existing issues and online Documentation

• [X] I confirm I have checked existing issues and online Documentation.

[kumekay]

Hello @ljden

Could you please send idf_component.yml of your project's main and component A?

[ljden]

The actual project is more complex than that but that was the simplification. Here's the relevant excerpts:

$ cat proj/main/idf_component.yml
dependencies:
  idf: "5.0.*"
  component_base:
    path: /home/vbox/component_base
  event_sim:
    version: master
    git: ssh://git@gitlab/path/to/event_sim.git
...
$ cat managed_components/event_sim/idf_component.yml 
version: "0.0.1"
description: event simulation
dependencies:
  idf: "5.0.*"
  component_base:
    version: master
    git: ssh://git@gitlab/path/to/component_base.git
$ idf.py build
...
Solving dependencies requirements
Updating lock file at /home/vbox/proj/dependencies.lock
Processing 10 dependencies:
...
[2/10] component_base (0.0.1)
[3/10] component_base (bf6205ccb2d90871d9318ef88289ed28c1b42896)
...
[5/10] event_sim (5e7e39bc058f6179b5a1c487f64bc65f213fdbee)
...

dependencies.lock contains only 9 dependencies, as there is only a single component_base, and is the git source not local

  component_base:
    component_hash: 866e7dd7cecb26b0b812afde25ed0d8671ce016873254e51dc534de10889a173
    source:
      git: ssh://git@gitlab/path/to/component_base.git
      path: .
      type: git
    version: bf6205ccb2d90871d9318ef88289ed28c1b42896

[kumekay]

@ljden You are right. It makes sense to expect local dependencies to be processed with higher priority. We will take a look.

[ljden]

Right, so I've run a few tests and have some interesting observations. To explain the scenario, our system has a component used as a common dependency for multiple other components. ie A, B, C, D, proj all depend on Common. If you have multiple of the same dependency in an idf_component.yml file, and they don't clash, it takes the last one. eg this resolves to git

vbox@vbox:~/proj$ cat main/idf_component.yml
dependencies:
  component_base:
    path: /home/vbox/component_base
  component_base:
    version: master
    git: ssh://git@gitlab/path/to/component_base.git
...

eg this resolves to local

vbox@vbox:~/proj$ cat main/idf_component.yml
dependencies:
  component_base:
    version: master
    git: ssh://git@gitlab/path/to/component_base.git
  component_base:
    path: /home/vbox/component_base
...

Now, it appears that the idf component manager will take the last unique dependency it sees as it's chosen dependency, ie:

vbox@vbox:~/proj$ cat main/idf_component.yml
dependencies:
  component_base:
    version: master
    git: ssh://git@gitlab/path/to/component_base.git
  event_sim:
    version: master
    git: ssh://git@gitlab/path/to/event_sim.git
  hello_world:
    version: master
    git: ssh://git@gitlab/path/to/hello_world.git
...
vbox@vbox:~/proj$ cat managed_components/event_sim/idf_component.yml
dependencies:
  component_base:
    path: /home/vbox/component_base
vbox@vbox:~/proj$ cat managed_components/hello_world/idf_component.yml
dependencies:
  component_base:
    version: master
    git: ssh://git@gitlab/path/to/component_base.git

will resolve to:

...
[2/10] component_base (bf6205ccb2d90871d9318ef88289ed28c1b42896)
[3/10] component_base (0.0.1)
...

which resolves to local in dependencies.lock. This seems to be as it finds git version in proj idf_component.yml then finds the local version in event_sim idf_component.yml then finds and ignores the git version in hello_world as it matches to a previously found git version in proj.

Swapping the source in proj idf_component.yml to local means local is found first, then local found again and ignored in event_sim, then git found in hello_world resolving to the last unique source which is git.

...
[2/10] component_base (0.0.1)
[3/10] component_base (bf6205ccb2d90871d9318ef88289ed28c1b42896)
...

If the common dependency is not in proj, it is unclear how the dependency order is resolved - it doesn't seem to be processed in the order found in the proj idf_component.yml nor alphabetical order.

[ljden]

@kumekay I have one further observation that was unexpected: proj depends on git version of component_base and local version of A A depends on local version of component_base idf component manager picks up both dependencies and (as expected) resolves to the 2nd unique one, local

...
[2/10] component_base (bf6205ccb2d90871d9318ef88289ed28c1b42896)
[3/10] component_base (0.0.1)
...

dependencies.lock also confirms that local was the resolved dependency. However, component_base has still been downloaded as part of the dependency resolution procedure and is located in managed_components. Building the proj results in a build error as the includes for component_base are sourced from the managed_component directory

... -I/home/vbox/proj/managed_components/component_base/include ...

No idea how or why this occurred, I can't reliably make this happen

[kumekay]

@ljden Thank you for the details, I think this new problem will be solved along with the original one