microsoft / azure-pipelines-yaml

Azure Pipelines YAML examples, templates, and community interaction
MIT License
1.19k stars 924 forks source link

Examples for resource pipeline and source inputs are not clear. #594

Open PramodKumarYadav opened 1 year ago

PramodKumarYadav commented 1 year ago

This issue is with reference to examples listed here: https://github.com/microsoft/azure-pipelines-yaml/blob/master/design/pipeline-resources.md#examples

For example, it is not clear what the distinction between input at "pipeline: SmartHotel" and "source: SmartHotel-CI" from a source side is. If you could given some examples on how to get these names and from where with an example, that would help a lot. Currently its a bit fuzzy and making it difficult to make this part work.

resources: pipelines:

DmitriiBobreshev commented 1 year ago

Hi @PramodKumarYadav, thank you for the suggestion! The main goal of the docs is to support collaborative contributions to the design process. For the actual and relevant information you could check the Azure Devops Docs, speaking about the pipeline resources, you could check detailed documentation here. We'll also change the example for the file we will have enough capacity, but we also very welcome any PRs to make the documentation in the repository more helpful.

PramodKumarYadav commented 1 year ago

Thanks for the response Dmitrii.

We'll also change the example for the file we will have enough capacity, but we also very welcome any PRs to make the documentation in the repository more helpful.

On your above suggestion I would have made a PR but its not clear to me how & what values for these fields should be based on an upstream jobs pipeline. I went through the docs more than a few times and tried some combinations but all of them resulted in error saying "Pipeline Resource trigger-tests-on-merge-to-main Input Must be Valid."

Now it could be because of some simple thing missing and I tried a lot of trial and error combinations but since there is no one-to-one reference on the values to be filled in downstream job vs upstream job pipeline source, its not clear to me what is getting missed here and why it is not working. If you can help give me an example and if it works, I will be glad to create a PR to update the docs.

Thanks again for looking into it and appreciate your support on this 🙌

DmitriiBobreshev commented 1 year ago

@PramodKumarYadav Sure, If I understood you correctly, you want to run one pipeline from another, you can check this detailed example as a reference. If you already saw it, could you please write in detail what you want to do, maybe I could help you with it?

btw, maybe the problem is that you need to put branch into the trigger input as in the example

# app-ci YAML pipeline
resources:
  pipelines:
  - pipeline: securitylib
    source: security-lib-ci
    trigger: 
      branches:
        include: 
        - releases/*
        exclude:
        - releases/old*
PramodKumarYadav commented 1 year ago

Hello @DmitriiBobreshev , my use case is to use option number 5 (Pipeline completion triggers) here:

I have a "test repository" which contains a pipeline that can run tests. What I want is when an "application" repository pipeline finishes (say for deployment), then I can immediately run my tests in the "test" repository post the application pipeline completion.

btw, maybe the problem is that you need to put branch into the trigger input as in the example

P.S: I did include branch name in the pipeline I setup but it didnt work. So maybe a working example that shows how the downsteam (test) repo should refer to resource names in an upstream (app) repo that would be great!

DmitriiBobreshev commented 1 year ago

@PramodKumarYadav, Have you tried to enable build completion triggers using GUI? In the "application's" pipeline, you need to select triggers image and set your"test repository's" pipeline image

here you can find the description: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops

PramodKumarYadav commented 1 year ago

@DmitriiBobreshev : Classic build triggers are not allowed in our company. So I have to use pipeline triggers and I keep getting this error.

on-internal-trigger.yml (Line: 7, Col: 15): Pipeline Resource securitylib Input Must be Valid.

even when I use exactly the same trigger example as specified here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops

P.S: This has been so much more simpler to setup when I worked with Github Actions. I wish it wasn't so difficult here :). Maybe an example public test repo that shows all these pipelines working examples would be great.

Step1: Pipeline to run first: security-lib-ci.yml file Result: pipeline succeds

# security-lib-ci YAML pipeline
steps:
- bash: echo "The security-lib-ci pipeline runs first"

Step2: Pipeline to run second: on-internal-trigger.yml Result: fails with error on-internal-trigger.yml (Line: 7, Col: 15): Pipeline Resource securitylib Input Must be Valid.

# app-ci YAML pipeline
# We are setting up a pipeline resource that references the security-lib-ci
# pipeline and setting up a pipeline completion trigger so that our app-ci
# pipeline runs when a run of the security-lib-ci pipeline completes
resources:
  pipelines:
  - pipeline: securitylib # Name of the pipeline resource.
    source: security-lib-ci # The name of the pipeline referenced by this pipeline resource.
    # project: app # Required only if the source pipeline is in another project
    trigger: true # Run app-ci pipeline when any run of security-lib-ci completes

steps:
- bash: echo "app-ci runs after security-lib-ci completes"

Location of files (both in root repository)

Screenshot 2023-06-12 at 18 54 45
PramodKumarYadav commented 1 year ago

Okay, I finally got to make it work but it seems the documentation here would need some updating if others dont go through the same issues. In short the name of pipeline is not the nae of the file but the name of pipeline as seen in the GUI. source: security-lib-ci # The name of the pipeline referenced by this pipeline resource. So for example the name of the first pipeline to finish is:

Now this again is not enough. After user add the correct pipeline name in the source, they would also need to add a pipeline via GUI for these secondary pipeline triggers. Unless we do that the the yaml files in themselves are not enough to trigger the runs (something which works automatically in tools like github action). So unless you add this instruction also in the read me file, it would not work.

As a last step, If this trigger should really run "after" the first pipeline is finished then we need to add a trigger: none on the top so that our second pipeline only run one time "after" the first pipeline is finished. If we dont add this our pipeline will run two times. First in parallel to the first line and then again after first pipeline is finished.

P.S: If you could ping me the location to update documentation, I will be more than happy to create a PR with detailed instructions on how to do this properly. Hopefully no other user has to face this issue again ever.

PramodKumarYadav commented 1 year ago

My bad. I see this is the repo where all the documentation is :). I will create a PR right away with detailed examples.

Edit: Spoken too soon. I dont see the files for this page in this repo: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops

If you could ping me the repo where this documentation is updated, I will clone and create a PR there.

DmitriiBobreshev commented 1 year ago

Hi @PramodKumarYadav, I'm glad that it finally working! The documentation for ms learn is placed here. You could open most of the ms learn sources by pressing the button image