microsoft / azure-pipelines-yaml

Azure Pipelines YAML examples, templates, and community interaction
MIT License
1.19k stars 924 forks source link

How to have multiple foreach iteration azurepipeline yaml? #563

Closed vivuu1989 closed 2 years ago

vivuu1989 commented 2 years ago

in azuredevops yaml pipeline, can we have nested forloop ? or multiple for loop.

My requirements here is to iterate the pipeline through the mapped variable values and this should be done across environments.

Eg: In my ADO repo i have 10 folders for each apps and in each apps i have 4 different variabletemplate files for each environment ,namely "vars-development.yaml, vars-staging.yaml, vars-prod.yaml, vars-dr.yaml"

What i need to have is in each of this environment specific variable template, i need to have differet alert rule inputs (there will be multiple alerts per environment ). variables.development.yaml

    variables:
      alert1:
         environment: development
         subscription: 'mysubscription'
         resourceGroup: myrg
         AlertRuleName: myalertrule
         query:myquery
         timeWindowInMinutes: 1    
         severity: 1
         operator: GreaterThanOrEqual
         threshold: 20

      alert2:
         environment: development
         subscription: 'mysubscription'
         resourceGroup: myrg
         AlertRuleName: myalertrule
         query:myquery
         timeWindowInMinutes: 1    
         severity: 1
         operator: GreaterThanOrEqual
         threshold: 20
         autoMitigate: false

    etc.............

      alertn: 

azure-pipeline.yaml

name: $(Build.SourceBranchName)-$(Build.BuildId)
trigger: none

parameters:
- name: appname
  type: string
  values:
    - app1
    - app2
    - app3
    - app4
    - app5
    - app5
stages:
  - template: loganalytic-alertrulet.yaml
    parameters:
      rootModuleFolder: ${{ parameters.appname }}
      envList:
        - development
        - staging
        - dr
        - production

loganalytic-alertrulet.yaml

parameters:
- name: appname
  type: string
  default: ""
- name: envList
  type: object
- name: rootModuleFolder
  type: string

stages:
- stage: BuilD_ARM_LogAnalytic_Artifact
  displayName: 'Build and validate Query Based Alert for aggregator Count'
  jobs:
    - ${{ each environment in parameters.envList }}:
      - job: BuilD_ARM_LogAnalytic_Artifact_${{ environment }}
        variables:
          - group: ${{ environment }}-alert-config
          - template: ../${{ parameters.rootModuleFolder }}/vars-${{ environment }}.yaml      
        workspace:
          clean: all
        pool:
          ${{ if eq(environment, 'development')}}:
            name: devpool
          ${{ if eq(environment, 'staging')}}:
            name: stagepool
          ${{ if eq(environment, 'production')}}:
            name: prodpool
          ${{ if eq(environment, 'dr')}}:
            name: drpool               
        steps:
          - ${{ each alert in variable.alertList }}:        
            - task: qetza.replacetokens.replacetokens-task.replacetokens@3
              inputs:
                rootDirectory: '$(System.DefaultWorkingDirectory)/'
                targetFiles: '$(System.DefaultWorkingDirectory)/logsearch-numberofresult/*.json'
                encoding: 'auto'
                writeBOM: true
                actionOnMissing: 'warn'
                keepToken: false
                tokenPrefix: '#{'
                tokenSuffix: '}#'
                useLegacyPattern: false
                enableTelemetry: true
            - task: AzureCLI@2
              displayName: "validate the templates"
              inputs:
                azureSubscription: ${{ variables.subscription }}
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                inlineScript: 'az group deployment validate --resource-group ${{ variables.resourceGroup }} --template-file $(System.DefaultWorkingDirectory)/logsearch-numberofresult/numofresult_template.json --parameters $(System.DefaultWorkingDirectory)/ogsearch-numberofresult/numofresulparam.json'
            - task: AzureCLI@2
              displayName: "verify the change result"
              inputs:
                azureSubscription: ${{ variables.subscription }}
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                inlineScript: 'az deployment group what-if --resource-group ${{ variables.resourceGroup }} --template-file $(System.DefaultWorkingDirectory)/logsearch-numberofresult/numofresult_template.json --parameters $(System.DefaultWorkingDirectory)/logsearch-numberofresult/numofresulparam.json'          
            - task: PublishBuildArtifacts@1
              inputs:
                PathtoPublish: '$(System.DefaultWorkingDirectory)/05-logsearch-numberofresult/'
                ArtifactName: 'arm-source-${{ variables.environment }}'
                publishLocation: 'Container'
              name: "Publish_arm_code"
              displayName: "Publish arm code as build artifact"

- ${{ each environment in parameters.envList }}:         
    - stage: Create_LogSearch__Alert_${{ environment }}
      displayName: 'Create LogSearch Count Aggregation alert Alert ${{ environment }}'
      condition: succeeded()
      variables:
        - group: ${{ environment }}-alert-config
        - template: ../${{ parameters.rootModuleFolder }}/vars-${{ environment }}.yaml
      jobs:     
      - deployment: deploy_Alert_${{ environment }}
        ${{ if eq(variables.subscription, 'NonProd') }}:
          environment: DEV        
        ${{ if eq(variables.subscription, 'Test') }}:    
          environment: DEV
        ${{ if eq(variables.subscription, 'PROD') }}:
          environment: PROD
        ${{ if eq(variables.subscription, 'DR') }}:    
          environment: DR              
        displayName: "Creating ALert in ${{ environment }}"
        pool:
          ${{ if eq(variables.subscription, 'NonProd') }}:          
            name: devpool
          ${{ if eq(variables.subscription, 'Test') }}:
            name: stagepool
          ${{ if eq(variables.subscription, 'PROD') }}:          
            name: prodpool
          ${{ if eq(variables.subscription, 'DR') }}:
            name: drpool                
        strategy:
          runOnce:
            deploy:
              steps:
              - task: DownloadBuildArtifacts@0
                inputs:
                  artifactName: 'arm-source-${{ environment }}'
                  downloadPath: $(System.ArtifactsDirectory)              
              - task: CopyFiles@2
                inputs:
                  sourceFolder: $(System.ArtifactsDirectory)/arm-source-${{ environment }}
                  contents: '**'
                  targetFolder: $(System.DefaultWorkingDirectory)/arm-source-${{ environment }}
                  cleanTargetFolder: true                   
              - task: AzureResourceGroupDeployment@2
                ${{ if eq(environment, 'development') }}:              
                  inputs:
                    azureSubscription: 'NonProd'
                    action: 'Create Or Update Resource Group'
                    resourceGroupName: ${{ variables.resourcegroup }}
                    location: 'US'
                    templateLocation: 'Linked artifact'
                    csmFile: '$(System.DefaultWorkingDirectory)/arm-source-development/numofresult_template.json '
                    csmParametersFile: '$(System.DefaultWorkingDirectory)/arm-source-development/numofresulparam.json'
                ${{ if eq(environment, 'staging') }}:   
                  inputs:
                    azureSubscription: 'NonProd'
                    action: 'Create Or Update Resource Group'
                    resourceGroupName: ${{ variables.resourcegroup }}
                    location: 'US'
                    templateLocation: 'Linked artifact'
                    csmFile: '$(System.DefaultWorkingDirectory)/arm-source-staging/numofresult_template.json '
                    csmParametersFile: '$(System.DefaultWorkingDirectory)/arm-source-staging/numofresulparam.json'
                ${{ if eq(environment, 'production') }}:   
                  inputs:
                    azureSubscription: 'PROD'
                    action: 'Create Or Update Resource Group'
                    resourceGroupName: ${{ variables.resourcegroup }}
                    location: 'West Europe'
                    templateLocation: 'Linked artifact'
                    csmFile: '$(System.DefaultWorkingDirectory)/arm-source-production/numofresult_template.json '
                    csmParametersFile: '$(System.DefaultWorkingDirectory)/arm-source-production/numofresulparam.json'
                ${{ if eq(environment, 'dr') }}:   
                  inputs:
                    azureSubscription: 'DR'
                    action: 'Create Or Update Resource Group'
                    resourceGroupName: ${{ variables.resourcegroup }}
                    location: 'US'
                    templateLocation: 'Linked artifact'
                    csmFile: '$(System.DefaultWorkingDirectory)/arm-source-dr/numofresult_template.json '
                    csmParametersFile: '$(System.DefaultWorkingDirectory)/arm-source-dr/numofresulparam.json'                                           

so here I am using arm template to create the alert and need to generate these alert rules across the 4 environments. in Azurepipeline yaml, How i can use the 2 level foreach iteration for alert rules as well as environment widse.

anatolybolshakov commented 2 years ago

Hi @vivuu1989 not really sure if there is some capability to do this... I would suggest to open a ticket on https://developercommunity.visualstudio.com/search?space=21 to get right eyes on it, since this repo is mostly for questions about templates located in this repo.