search

scolladon / sfdx-git-delta

Generate the sfdx content in source format from two git commits
Other
438 stars 115 forks source link

salesforce delta deployment CICD using azure devops yaml #573

Closed bharuk closed 1 year ago

bharuk commented 1 year ago

Is your proposal related to a problem?

Hi we are planning to salesforce delta using azure devops CICD i want help from yaml script to build salesforce delta deployment

trigger: none

resources:
    webhooks:
    - webhook: testspa_new          ### Webhook alias
      connection: testspa_new
      filters:
        - path: pull_request.merged  
          value: true
        - path: pull_request.base.ref  
          value: "develop"

    repositories:
    - repository: Mytest  
      type: githubenterprise
      name: karu-bhar/Mytest
      ref: develop
      endpoint: myrepo1

      extends:
      template: keyfiles.yml

variables:
    - name: devops_dir
      value: "/home/vsts/work/1/scripts-deploy"  
    - name: global_dir
      value: "/home/vsts/work/1/s/Mytest"
    #- group: varhashicorp
    #- group: vargroup-sfdx

stages:
 - stage: UnlockedPackage_Creation
   jobs:
      - job: Build 
        timeoutInMinutes: 360
        pool:
         vmImage: ubuntu-latest

        steps:
        - checkout: self
          path: scripts-deploy
        - checkout: Mytest
        - task: PowerShell@2
          displayName: 'List files in Git Directory'
          inputs:
            targetType: 'inline'
            script: 'Get-ChildItem -Path "$(System.DefaultWorkingDirectory)" -recurse'    
            workingDirectory: '$(System.DefaultWorkingDirectory)'
        - task: Bash@3
          inputs:
            targetType: 'inline'
            script: |
              cd $(System.DefaultWorkingDirectory)/Mytest
            #cat /home/vsts/work/1/s/Mytest/describe.log
        - task: Bash@3
          displayName: 'Set git config email & name'
          inputs:
            targetType: 'inline'
            script: |
             git config --global user.email "kar.bhar@xx.com"
             git config --global user.name "kar-bhar"
        - task: UsePythonVersion@0
          inputs:
           versionSpec: '3.7'
        - script: |
           python -m pip install --upgrade pip
        - bash:
           #python --version 
           pip install wheel --upgrade
           pip install dohq-artifactory
           pip install hvac
           pip install -r /home/vsts/work/1/scripts-deploy/folder/requirements.txt --no-cache-dir
          # pip install -r /home/vsts/work/1/s/scripts-deploy/folder/requirements.txt  

        #- script: pip install -r requirements.txt
        #  displayName: 'Install requirements'
        - bash:
           npm install -g npm
          displayName: Install Latest version of npm  

             #INSTALL SFDX_CLI
        - bash: 
            npm install sfdx-cli --global
          displayName: Install SFDX
           echo y | node_modules/sfdx-cli/bin/run plugins:install sfdx-git-delta

        #INSTALL VLOCITY BUILD TOOL
        - bash:
            npm install --global vlocity
          displayName: Install Vlocity
        - bash: |
                 cd $(devops_dir)
                 sfdx force:auth:jwt:grant --clientid "3MVG" --jwtkeyfile "./buildfiles/server.key" --username "bharu@vois.sandbox" --instanceurl "https://login.salesforce.com"

        - task: Bash@3
          inputs:
           targetType: 'inline'
           script: |
            $logFile= sfdx force:mdapi:describemetadata -u "@vois.sandbox" -f $global_dir/describe.log --apiversion "56.0"`

Describe a solution you propose

Describe alternatives you've considered

Additional context

scolladon commented 1 year ago

Hi,

I'm not very skilled at azure devops pipeline, but I'll ask experts around me and try to come with more precise advice.

The guidelines when using SGD is to git fetch locally in the CI the references you are going to compare so SGD can access those.

So the checkout step should set fetchDepth: 0.

And then you can use SGD:

- bash:
            mkdir output
            sfdx sgd:source:delta --to "HEAD" --from "<target-branch>" --output output -d

Then you can use the generated package.xml in output/package/package.xml to deploy or the source folder generated in the output

Hope it helps

laithal commented 1 year ago

Hi,

The best way to build YAML pipelines is by trying, failing, and trying again :)

  1. I'd stick to either bash (preferred) or PowerShell. Combining them increases complexities.
  2. Do not store your secrets in the yaml file! Please consider using Library of assets A library is a collection of build and release assets for an Azure DevOps project. Assets defined in a library can be used in multiple build and release pipelines of the project. The Library tab can be accessed directly in Azure Pipelines. You can use it to e.g. store your Salesforce Connected Apps secrets, certifications, creds, etc. e.g. in your Yaml file you then simply reference to the library by e ${{ parameters.clientID }}
  3. Add fetchDepth: 0 as line 43 image

Here is a sample YAML - please test and validate in a testing environment before using in Prod.

` parameters:

bharuk commented 1 year ago

I am getting error destructive changes should i create a directory in salesforce code repository? Warning: We plan to deprecate this command in the future. Try using the "project deploy start" command instead. Warning: The "--targetusername" flag has been deprecated. Use "--target-org | -o" instead. Error (1): Deploy directory destructive does not exist or is not a directory /home/vsts/work/_temp/cd3913f9-c8a4-405b-afdf-a1cf72dda31a.sh: line 2: unexpected EOF while looking for matching ``' /home/vsts/work/_temp/cd3913f9-c8a4-405b-afdf-a1cf72dda31a.sh: line 3: syntax error: unexpected end of file

[debug]Exit code 2 received from tool '/usr/bin/bash'

scolladon commented 1 year ago

The sample uses a destructive folder. In this case it should be the <output>/destructiveChanges, where output is the folder passed to --output / -o parameter of the sgd command.

jobs:
  - deployment: DeployPackage
    environment: ${{ parameters.environment }}
    container: salesforce/salesforcedx:latest-full
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
            fetchDepth: 0
          - task: DownloadPipelineArtifact@2
            inputs:
              artifact: drop
              path: '$(Pipeline.Workspace)'
          - task: DownloadSecureFile@1
            name: jwtkeyfile
            displayName: 'Download JWT key file'
            inputs:
              secureFile: 'JWT-${{ parameters.library }}'
          - script: |
              echo y | sfdx plugins:install sfdx-git-delta
              sfdx plugins
            displayName: 'Installing SGD'
          - script: |
              sfdx force:auth:jwt:grant --clientid $(consumerkey) \
              --jwtkeyfile $(jwtkeyfile.secureFilePath) --username $(username) \
              --setalias ${{ parameters.library }} -r ${{ parameters.instanceurl }}
              sfdx force:auth:list
            displayName: 'Authenticate'
          -script: |
              mkdir output
              sfdx sgd:source:delta --to "HEAD" --from "<target-branch>" --output output -d
            displayName: 'Generate incremental package'
          - script: |
              sfdx force:source:deploy -p output/force-app -w 60 --targetusername $(username) --noprompt       
            displayName: 'Deploy package'
          # Following Script will remove Salesforce components specified in the destructive/destructiveChangesPost.xml file in target Orgs.
          - script: |
              sfdx force:mdapi:deploy -d output/destructiveChanges -w -1 --targetusername $(username) --ignorewarnings
            displayName: 'Destructive Changes'`

You need to replace <target branch> with the name of the branch the pull request is targetting, with its remote name. Per example if your branch target the main branch and the remote is named origin, the target branch is origin/main

mehdicherf commented 1 year ago

@bharuk If that's ok with you I'm going to close this issue as the error does not seem related to the plugin itself, and @laithal and @scolladon have provided sample working Azure pipeline. Of course feel free to reopen an issue if you face an error related to the plugin.

bharuk commented 1 year ago

Hi I am using above code I am getting unknown error

`trigger: none

resources: webhooks:

variables:

branch :origin/develop feature: feature-version1

Error:

Generating script. ========================== Starting Command Output =========================== /usr/bin/bash --noprofile --norc /home/vsts/work/_temp/0b5e8411-b898-4f3e-a7da-c2d1a8ee14ba.sh (node:2104) Warning: Deprecated config name: apiVersion. Please use org-api-version instead. (Use node --trace-warnings ... to show where the warning was created) { "error": "--from is not a valid sha pointer: 'origin/develop'", "output": "output", "success": false, "warnings": [] }

[error]Bash exited with code '1'.

Finishing: Generate incremental package

'

scolladon commented 1 year ago

Hi @bharuk !

It seems the checkout steo does not specifyfetchDepth: 0attribute. Then the ref 'origin/develop' is not available in the execution context of the pipeline.

Could you try using the fetchDepth attribute in the checkout steps?

bharuk commented 1 year ago

It is working fine ` steps:

we created already output directory why it is not found

Error:

`/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/f8493e5e-19d5-43c9-b685-373f0d0efbfd.sh Warning: We plan to deprecate this command in the future. Try using the "project deploy start" command instead. Warning: The "--targetusername" flag has been deprecated. Use "--target-org | -u" instead. Error (1): output/force-app: File or folder not found

[debug]Exit code 1 received from tool '/usr/bin/bash'

[debug]STDIO streams have closed for tool '/usr/bin/bash'

[error]Bash exited with code '1'.

`

scolladon commented 1 year ago

@bharuk it could be because there is nothing to deploy incrementally (no changes). In this case the force-app folder is not created. You could check if the output/package/package.xml contains element by using grep: grep -q '<types>' output/package/package.xml cf this comment

bharuk commented 1 year ago

` What is the name of the 'type'. I already have the Salesforce code available inside that force-app directory. mandatory to create output directoy? what is the use of output directory. In which line to pass the grep grep -q 'types' output/package/package.xml'

please explain i am totally confusing.

scolladon commented 1 year ago

What I understand from the issue is that the deploy command try to deploy the content of output/force-app folder without success because it seems the output/force-app folder does not exist. From what I read, the output folder exist as it is created just before. So I was thinking maybe the force-app folder does not exist under the output folder. It happens when sgd does not detect any change (even if -d parameter flag is set). When there are no sources to deploy, it does not copy any. In order to detect if there are changes to deploy I was suggesting reading the package.xml. If it contains <types> tag then it means it contains elements. If you used the -d parameter then those elements should have been copied and you will be pretty sure it is deployable using force:source:deploy.

So, after having executed sgd to setup the incremental source content, and before deploying it using force:source:deploy, you could check if there is anything to deploy by checking if the package.xml contains any <types> tag (using grep).

bharuk commented 1 year ago

Any alternative method for delta deployment. still I am facing the issues. any python scripts for this delta deployment. why because we are planning multiple stages in azure devops yaml dev sit at uat preprod and prod. still i am getting intial build i am unable to deploy the package. when i am trying grep is no luck.

trigger: none

resources:
    webhooks:
    - webhook: gspdeltawebhook          ### Webhook alias
      connection: gspdeltawebhook
      filters:
        - path: pull_request.merged  
          value: true
        - path: pull_request.base.ref  
          value: "develop"              ##branch

    repositories:
    - repository: b2b-gsp-slfdelta  
      type: githubenterprise
      name: VFGroup-TaaS-Tenants/b2b-gsp-slfdelta
      ref: develop
      endpoint: Vodafone_GSP_Delta

variables:
    - name: devops_dir
      value: "/home/vsts/work/1/b2b-gsp-deltadevops"  
    - name: global_dir
      value: "/home/vsts/work/1/s/b2b-gsp-slfdelta"
    - group: vargroup-sfdx
stages:
  - stage: Creation
    jobs:
      - job: Build 
        timeoutInMinutes: 360
        pool:
         vmImage: ubuntu-latest

        steps:
          - checkout: self
            path: b2b-gsp-deltadevops
          - checkout: b2b-gsp-slfdelta
            fetchDepth: 0
          - task: DownloadSecureFile@1
            name: DevserverkeyFile
            displayName: download rsa key
            inputs:
             secureFile: 'Devserver.key'        
          - script: |
             echo installing the $(DevserverkeyFile.secureFilePath) to the trusted directory
          - bash:
              npm install -g npm
            displayName: Install Latest version of npm
          - bash: 
              npm install sfdx-cli --global
            displayName: Install SFDX
          - script: |
              echo y | sfdx plugins:install sfdx-git-delta
              sfdx plugins
            displayName: 'Installing SGD'
        #INSTALL VLOCITY BUILD TOOL
          - bash:
              npm install --global vlocity
            displayName: Install Vlocity
          - bash: |
                 cd $(devops_dir)
                 sfdx force:auth:jwt:grant --clientid $(DevhubclientId) --jwtkeyfile $(DevserverkeyFile.secureFilePath) --username $(DevhubUserName) --instanceurl $(DevhubInstanceURL)
            displayName: 'Authenticate'
          - script: |
              cd $(global_dir)
               #mkdir output
              sfdx sgd:source:delta --to "HEAD" --from "origin/develop" --output
            displayName: 'Generate incremental package'
          - script: |
              cd $(global_dir)
                #grep -q  output/package/package.xml && sfdx force:source:deploy -x output/package/package.xml
                sfdx force:source:deploy -x package/package.xml
              #sfdx force:source:deploy -p output/force-app -w 60 --targetusername $(DevhubUserName)
            displayName: 'Deploy package'
scolladon commented 1 year ago

Hi!

I see multiple potential root cause here:

  1. For "displayName: 'Generate incremental package'" step: --output parameter does not have value. It should be: 
    mkdir output
sfdx sgd:source:delta --to "HEAD" --from "origin/develop" --output output
  2. For "displayName: 'Deploy package'" step: deployment uses a package/package.xml. It should use output/package/package.xml 
    sfdx force:source:deploy -x output/package/package.xml

I hope it helps

bharuk commented 1 year ago

2023-05-11T12:57:17.8740582Z ##[debug]/usr/bin/bash arg: /home/vsts/work/_temp/ebb9c158-9009-4f48-a258-00afe10ca9a9.sh 2023-05-11T12:57:17.8743881Z ##[debug]exec tool: /usr/bin/bash 2023-05-11T12:57:17.8744261Z ##[debug]arguments: 2023-05-11T12:57:17.8745658Z ##[debug] --noprofile 2023-05-11T12:57:17.8746510Z ##[debug] --norc 2023-05-11T12:57:17.8746860Z ##[debug] /home/vsts/work/_temp/ebb9c158-9009-4f48-a258-00afe10ca9a9.sh 2023-05-11T12:57:17.8749822Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/ebb9c158-9009-4f48-a258-00afe10ca9a9.sh 2023-05-11T12:57:19.4465795Z Warning: We plan to deprecate this command in the future. Try using the "project deploy start" command instead. 2023-05-11T12:57:19.5008330Z Error (1): No authorization information found for harvesttest. 2023-05-11T12:57:19.5256693Z ##[debug]Exit code 1 received from tool '/usr/bin/bash' 2023-05-11T12:57:19.5260955Z ##[debug]STDIO streams have closed for tool '/usr/bin/bash' 2023-05-11T12:57:19.5297977Z ##[error]Bash exited with code '1'. 2023-05-11T12:57:19.5303975Z ##[debug]Processed: ##vso[task.issue type=error;]Bash exited with code '1'. 2023-05-11T12:57:19.5304992Z ##[debug]task result: Failed 2023-05-11T12:57:19.5306760Z ##[debug]Processed: ##vso[task.complete result=Failed;done=true;] 2023-05-11T12:57:19.5328666Z ##[section]Finishing: CmdLine

above error i am getting no authorization information found

scolladon commented 1 year ago

Hi @bharuk

I would say the authentication step failed here, no ? Have you made changes in the pipeline file you shared ?

What I can see in there is it seems to authenticate to a devhub but it does not create a scratch org The authentication does not seems to set a default alias and the deploy does not specify alias to target the deployment.

I hope it helps