Wait the end of all pipelines?

[kdisneur]

We have a monorepo and want to move our CI to drone. We tried your plugin and it works really well.

We just have one remaining issue: we would like to have a pipeline to start our end-to-end test when all our pipelines have successfully finished.

But, because we now have a separate drone.yaml for each project in the monorepository, we don't know how to wait that all pipelines have finished (because we don't know to which pipelines this last one should depend)

Have you already encounter this need? Do you have an idea of how we can achieve this?

[foosinn]

Hey @kdisneur,

have you tried https://docs.drone.io/configure/pipeline/multiple/#graph-execution

Sound like it should work.

[kdisneur]

We tried this but we might have misunderstood something because we weren't able to make it work in the monorepo context.

When we installed drone-tree-config we added a .drone.yaml by project:

monorepo/
  project-a/
    .drone.yaml # pipeline: project-a
    ...
  project-b/
    .drone.yaml # pipeline: project-b
    ...
  project-c/
    .drone.yaml # pipeline: project-c
    ...
  ....

drone-tree-config concatenates all the projects' drone.yaml files and drone builds each pipelines properly (i.e. project-a, project-b, and project-c).

We then tried to add a drone.yaml file at the root of the repository which depends of all possible pipelines:

depends_on: 
  - project-a
  - project-b
  - project-c

It worked if we changed all projects (i.e. project-a, project-b, and project-c) but didn't work if we only changed some projects (e.g. project-a and project-b) because it depends_on a non existing pipeline (project-c).

We don't know how to tell drone to run the pipeline when, in this example, project-a and project-b finished (or another combination depending of what changed in the monorepo).

To summarize: we would like the pipeline to wait for all the "project's pipelines" to successfully finish before starting but we don't know how to dynamically define the "project pipelines" we need to wait for.

[foosinn]

did you name the pipeline steps?

[kdisneur]

Yes I think so. You can find an example repository here: https://github.com/kdisneur/monodroneci

If I change something in project-a it works, it builds only project-a If I change something in project-b it works, it builds only project-b If I change something in both it works, it builds both

I then tried to add, at the root of the repository, another drone.yaml (https://github.com/kdisneur/monodroneci/commit/5713be5c3e47384ba8c3849d67d7583458c043c7). It works if I change both projects but fails if I change only one of them because the other pipeline doesn’t exist.

The error makes sense, I just don’t know how to get around it.

[foosinn]

Hmm, thats a good point. I am not sure how to fix this.

One way would be that drone allows non-existing steps to be gracefully skipped.

Another one would be that drone-tree-config parses all files and removes non existing steps, which means a lot more complexity.

You are welcome to contribute the second option, right now we have neigher a requirement nor the time to implement this.

[kdisneur]

OK. Thanks for your help :)

To be honest we don't really need this last pipeline. What we really need is a way to know Drone-CI finished to build the projects but we didn't find any webhook configuration in Drone-CI.

I will try to dig a little bit more the webhook solution with the Drone-CI community. If I don't find anything, I will try to suggest something here :)

Thanks again 🙇

[yekibud]

Hi @kdisneur - we could really use a solution to this, as well, did you figure anything out? @foosinn 's option to modify drone (or woodpecker?) to gracefully ignore non-existent pipeline dependencies seems like the best solution to me. Don't think I'm going to be able to find the time to learn Go and look into it soon, though, unfortunately.