[bug] Dependency chains are not respected in `moon ci`

[kikones34]

Describe the bug

If you have a dependency chain task1 -> task2 -> task3, when task1 is touched, moon ci will only execute task2 and it will not propagate to task3.

Steps to reproduce

Configure three tasks as follows:

  dependency:
    command: echo "I'm the dependency"
    inputs:
      - dependency-input.txt
    options:
      runInCI: true

  dependency-checker:
    deps:
      - dependency
    command: echo "I'm the dependency checker"
    inputs:
      - dependency-checker-input.txt
    options:
      runInCI: true

  dependent:
    deps:
      - dependency-checker
    command: echo "I'm the dependent"
    inputs:
      - dependent-input.txt
    options:
      runInCI: true

Commit everything to the default git branch. moon ci shouldn't execute anything. Now, modify the file dependency-input.txt and execute moon ci. The dependency-checker task is run, but the dependent is not:

▪▪▪▪ Gathering touched files
  services/trio-cloud-search/dependency-input.txt
▪▪▪▪ Gathering potential targets
  ...
  trio-cloud-search:dependency
  trio-cloud-search:dependency-checker
  trio-cloud-search:dependent
  ...
▪▪▪▪ Generating action graph
Target count: 335
Action count: 12
▪▪▪▪ Running targets
▪▪▪▪ trio-cloud-search:dependency (faf9f199)
        trio-cloud-search:dependency | I'm the dependency
▪▪▪▪ trio-cloud-search:dependency (faf9f199)
▪▪▪▪ trio-cloud-search:dependency-checker (7f147af2)
trio-cloud-search:dependency-checker | I'm the dependency checker
▪▪▪▪ trio-cloud-search:dependency-checker (7f147af2)

Expected behavior

The task dependent should also be executed since it depends on dependency-checker.

Additional context

Note that these dependency chains behave correctly regarding the cache logic (so if dependency is dirty, it will propagate the dirty state up to dependent). The issue only occurs in moon ci.

[milesj]

@kikones34 This is by design at the moment. The "run dependents" logic only applies to direct dependents of affected tasks, and not the entire tree. It was built this way as a simple regression testing mechanism and ensuring a change in a project didn't affect it's immediate consumers.

However, we can probably add an option to control the depth.

[kikones34]

@milesj Oh, I see. The problem is that this breaks the way we're managing inter-project dependencies. In the real-world use case, what we actually have is:

• A check-source-changed task on each project which gets dirty when the source code of the project changes.
• A check-deps-source-changed task on each project, which has dependencies such as proj1:check-source-changed, proj2:check-source-changed, etc. It gets dirty whenever one of those project's source is changed.
• Tasks that depend on check-deps-source-changed. For example, test only gets dirty if the project source or the source of the dependencies has changed.

This has worked perfectly as far as caching goes, but I'm finding it impossible to replicate with the CI feature.

[milesj]

@kikones34 Yah that makes sense. Are you able to use moon run in CI now to replicate what you want?

[kikones34]

Yeah for now we're using moon run, but managing the cache is tricky due to the pipelines being run by Bamboo agents, different PRs open on different branches at the same time, etc. For now it's mostly working, although we are limiting the cache to be per-branch, otherwise many issues arise. That's why we were looking into leveraging the moon ci capabilities so that we can always run just the minimum necessary tasks given by the touched files, without having to worry so much about the cache state.

[milesj]

I'll look into supporting deep dependents in the next release.