HodorNV / ALOps

ALOps
57 stars 24 forks source link

How do you handle storing the new version of your appsource after release to appsource in regards to use it to check for breaking changes. #716

Open MortenRa opened 8 months ago

MortenRa commented 8 months ago

After a release you new app to appsource, how do you handle updating the new app and appsourcecop.json in your CI/CD setup using alops or extending this with maybe some powershell and how do your developer use that app to check for breaking changes.

Looking for some inspiration for a multi developer environment setup and CI/CD to be updated automatically after release to appsource.

Currently we upload this to a previous folder and update the appsourcecop.json after every release by creating a new branch and merge. but looking into maybe put this as a step into the Release Pipeline.

So what do you do:

How do you update appsource.json after a Release to appsource Do you store the new app in the repo or another storage How do your developers use the a app to make sure not to do breaking changes to you handle this in vs code or in CI/CD (could require some iteration).

Can ALops offer any Task in Release Pipeline to use the artifact from the Release to be able to create a branch and update appsourceCop and .previous folder with new app file to be ready for Merge.

waldo1001 commented 8 months ago

Yeah, that's not easy.

what we do: Step 1 - AppSource Validation this is pretty much the "ALOps App Validation" step.

Step 2 - Upload to AppSource This is the "ALOps AppSource" step

Step 3 - Update master This will update the master branch with the new apps, AND update the AppSourceCop.json THis is the script:

$CollectionURI = "$(System.CollectionUri)"
$ProjectID = "$(System.TeamProjectId)"
$RootDir = "$(System.ArtifactsDirectory)"

$Authorization = "Basic $([Convert]::ToBase64String([System.Text.Encoding]::Default.GetBytes(":$(System.AccessToken)")))"

## Do not run when not enabled
if ("$(UpdateMasterBranches)" -ne "true")
{
  break
}

#####################
# Find all apps
#####################
$AppDir = Join-Path -Path $RootDir -ChildPath "Distri Apps"
$PreviousDir = Join-Path $RootDir -ChildPath "AppSourcePrevious"

$SourceFiles = Get-ChildItem -Path $AppDir -Filter "*.app" -Recurse
$PreviousFiles = Get-ChildItem -Path $PreviousDir -Filter "*.app" -Recurse

function Get-TFSFile()
{
    param(
        [string] $Path,
        [string] $RepoID
    )

    try
    {
        $TargetUri = "$($CollectionURI)/$($ProjectID)/_apis/git/repositories/($($RepoID))/items?path=$($Path)&api-version=5.1"
        Write-Debug "Target Url: [$($TargetUri)]"
        return [System.Text.Encoding]::UTF8.GetString((Invoke-WebRequest -UseBasicParsing -Method Get -Uri $TargetUri -ContentType "application/json" -Headers @{ "Authorization" = $Authorization }).Content)
    }
    catch
    {}

    return $null
}

function Get-LastCommitID()
{
    param(
        [string] $RepoID
    )

    $TargetUri = "$($CollectionURI)/$($ProjectID)/_apis/git/repositories/($($RepoID))/stats/branches?api-version=5.1"
    Write-Debug "Target Url: [$($TargetUri)]"
    $BranchInfo = (Invoke-RestMethod -Uri $TargetUri -Method Get -Headers @{ "Authorization" = $Authorization }).value
    $LastCommitID = ($BranchInfo | Where-Object { $_.name -eq 'master' -or $_.name -eq 'main'}).commit.commitID
    if ([String]::IsNullOrEmpty($LastCommitID)) {
        $LastCommitID = "0000000000000000000000000000000000000000"
    }

    return $LastCommitID
}

#####################
# Get Repo's
#####################
$TargetUri = "$($CollectionURI)/$($ProjectID)/_apis/git/repositories?api-version=5.1"
Write-Debug "Target Url: [$($TargetUri)]"
$Repos = (Invoke-RestMethod -Method Get -Uri $TargetUri -Headers @{ "Authorization" = $Authorization }).value

#####################
# Fetch Repo App Info
#####################
$RepoInfo = @()
foreach ($Repo in $Repos) {
    $RepoInfo += [PSCustomObject]@{
        Name = $Repo.name
        Found = $false
        AppName = ""  
        TFSObject = $Repo
    }
}

foreach ($item in $RepoInfo) {
    $AppJson = Get-TFSFile -Path "App/app.json" -RepoID $item.TFSObject.id

    if ($null -ne $AppJson)
    {
        $item.Found = $true
        $item.AppName = ($AppJson | ConvertFrom-Json).name
    }
}

#####################
# Link all app files to Repo
#####################
$FindObjects = @()
Foreach ($File in $SourceFiles)
{
    $FileInfo = $File.Name.Split('_')
    $PreviousFile = $PreviousFiles | Where-Object { $_.Name.Contains("_$($FileInfo[1])_") }

    $PreviousFileInfo = @()
    if ($null -ne $PreviousFile)
    {
        $PreviousFileInfo = $PreviousFile.Name.Split('_')
    }
    $FindObject = [PSCustomObject]@{
                        Name = $FileInfo[1];
                        Version = $FileInfo[2];
                        PreviousVersion = $PreviousFileInfo[2];
                        Path = $File.FullName;
                        FileName = $File.Name
                        PreviousPath = $PreviousFile.FullName;
                        PreviousFileName = $PreviousFile.Name
                        Repo = ($RepoInfo | Where-Object {$_.AppName -eq $FileInfo[1]}).TFSObject
                    }

    $FindObjects += $FindObject
}

$FindObjects

Foreach ($File in $FindObjects)
{
    Write-Host "*** [$($File.Name)]"
    $AppsourceJson = Get-TFSFile -Path "App/AppSourceCop.json" -RepoID $File.Repo.id

    if ($null -ne $AppsourceJson)
    {
        $AppsourceJsonObject = ($AppsourceJson | ConvertFrom-Json)
        $CurrentVersion = $AppsourceJsonObject.version
        if ($CurrentVersion -eq $File.Version) { 
            Write-Host -ForegroundColor Green "  * No update needed"
            continue 
        }

        $fileContentBytes = [System.IO.File]::ReadAllBytes($File.Path)
        $fileContentEncoded = [System.Convert]::ToBase64String($fileContentBytes)

        if ($null -eq ($AppsourceJsonObject | Get-Member -Name Version))
        {
            $AppsourceJsonObject | Add-Member -MemberType NoteProperty -Name version -Value ""
        }
        $AppsourceJsonObject.version = $File.Version
        $AppsourceJson = ($AppsourceJsonObject | ConvertTo-Json -Depth 100)

        $Changes = @()
        if (-not [string]::IsNullOrEmpty($File.PreviousFileName))
        {
            $Changes += @{
                            changeType = "delete" ;
                            item = @{ path = "App/.appSourceCopPackages/$($File.PreviousFileName)" };
                        }
        }

        $Changes += @{
                        changeType = "add" ;
                        item = @{ path = "App/.appSourceCopPackages/$($File.FileName)" };
                        newContent = @{
                            content     = "$($fileContentEncoded)";
                            contentType = "base64encoded"
                        }
                    },
                    @{
                        changeType = "edit" ;
                        item = @{ path = "App/AppSourceCop.json" };
                        newContent = @{
                            content = $AppsourceJson;
                            contentType = "rawtext"
                        }

                    } 

        $ChangedFile = @{
            refUpdates = @(
                @{
                    name = "refs/heads/master";
                    oldObjectId = "$(Get-LastCommitID -RepoID $File.Repo.id)"
                }
            );
            commits = @(
                @{
                    comment = "AppSource update to version [$($File.Version)]" ;
                    changes = $Changes
                }
            )
        }

        $TargetUri = "$($CollectionURI)/$($ProjectID)/_apis/git/repositories/($($File.Repo.id))/pushes?api-version=5.1"
        Write-Debug "Target Url: [$($TargetUri)]"  
        $result = Invoke-RestMethod -Method Post -Uri $TargetUri -ContentType "application/json" -Body ($ChangedFile | ConvertTo-Json -Depth 100) -Headers @{ "Authorization" = $Authorization }
    }
}

Hope it helps ..

fvet commented 7 months ago

@MortenRa

Can ALops offer any Task in Release Pipeline to use the artifact from the Release to be able to create a branch and update appsourceCop and .previous folder with new app file to be ready for Merge.

This was our initial expectation from ALOPs as well, not only to provide us with the building blocks to configure a fully functioning CI/CD setup for AL projects, but also including some out-of-the-box tooling to smoothen the post-release steps and minimize the friction for the developer as well. (PS: seems not supported by AL-Go on github either)

image

So, we decided to build something custom. High-level, we adopt below process:

image

image

which mainly updates the app.json file, appsource.json file and replaces the just released app in a separate folder, on a separate release branch (tagged) + makes the PR.

image

All the developer has to do when pulling the latest master changes, is to copy-paste the app file to his central alpackages path and done.

image

As an inspiration, here's part of the script (not created by a PS expert :( )

[CmdletBinding()]
param (        
    [Parameter(Mandatory = $true)]
    [string]$WorkingDirectory,
    [Parameter(Mandatory = $true)]
    [string]$ArtifactsDirectory,    
    [boolean]$UpdatePreviousApps = $true,
    [string]$AppSourceVersion,
    [boolean]$IncreaseAppVersion = $true,
    [string]$AppVersion
)

#Settings

$AppsDirectory = Join-Path $ArtifactsDirectory '\Apps'

# Functions

# Formats JSON in a nicer format than the built-in ConvertTo-Json does.
# https://stackoverflow.com/questions/33145377/how-to-change-tab-width-when-converting-to-json-in-powershell
function Format-Json([Parameter(Mandatory, ValueFromPipeline)][String] $json) {
    $indent = 0;

    # https://stackoverflow.com/questions/69345030/sign-is-converted-into-u0026-through-powershell
    $json = $json -replace '\\u0026', '&' -replace '\\u0027', "'" -replace '\\u003c', '<' -replace '\\u003e', '>'

    ($json -Split "`n" | % {
        if ($_ -match '[\}\]]\s*,?\s*$') {
            # This line ends with ] or }, decrement the indentation level
            $indent--
        }
        $line = ('  ' * $indent) + $($_.TrimStart() -replace '":  (["{[])', '": $1' -replace ':  ', ': ')
        if ($_ -match '[\{\[]\s*$') {
            # This line ends with [ or {, increment the indentation level
            $indent++
        }
        $line
    }) -Join "`n"
}

function Update-Dependencies {
    param (        
        [Parameter(Mandatory = $true)]
        [string]$Publisher,
        [Parameter(Mandatory = $true)]
        [string]$Version
    )

    if ($AppContent.dependencies) {        
        foreach ($Dependency in $AppContent.dependencies) {
            if ($null -ne $Dependency.publisher -and $Dependency.publisher -eq $Publisher) {
                Write-Host "  Update version to $AppVersion (dependency $($Dependency.name))"  
                $Dependency.version = $Version                
            }
        }
    }
}

function Get-Repositories() {
    param (
        $ParentFolder
    )
    Get-ChildItem -Path $ParentFolder -Directory -Depth 1 -Filter '.git' -Force `
    | Where-Object { $_.Parent.Name -ne 'Scripts' } `
    | ForEach-Object { $_.Parent }    
}

$Repositories = (Get-Repositories -ParentFolder $WorkingDirectory) # Should be limited to only one, but this work-around points to the correct folder name for the \App

foreach ($Repository in $Repositories) {

    Set-Location -Path $Repository.FullName

    git config core.eol lf
    git config core.autocrlf input
    git config core.safecrlf false

    $UserName = (git config user.name)
    $UserEmail = (git config user.email)

    if (-not $UserName) {
        $UserName = $env:BUILD_REQUESTEDFOR
        git config user.name "$UserName"
    }

    if (-not $UserEmail) {
        $UserEmail = $env:BUILD_REQUESTEDFOREMAIL
        git config user.email "$UserEmail"
    }     

    $AppDirectory = Join-Path $Repository.FullName '\App'
    $PackagesDirectory = Join-Path $AppDirectory '\.appSourceCopPackages'

    Write-Host "Settings ..." -ForegroundColor Yellow
    Write-Host "  Repository: $($Repository.Name)"    
    Write-Host "  AppDirectory: $AppDirectory"
    Write-Host "  PackagesDirectory: $PackagesDirectory"
    Write-host "  Running under $UserName / $UserEmail"

    # Update Previous Apps

    if ($UpdatePreviousApps) {
        Write-Host "Update Previous Apps" -ForegroundColor Green

        Write-Host "Settings ..." -ForegroundColor Yellow
        Write-Host "  AppSourceVersion: $AppSourceVersion"    

        # Remove old app
        Write-Host "Removing previous Dynavision apps ..." -ForegroundColor Yellow
        Write-Host "  Directory: $PackagesDirectory"

        if (Test-Path -Path $PackagesDirectory) {    
            Get-ChildItem -Path $PackagesDirectory -Filter 'Dynavision*.app' | ForEach-Object { 
                Write-Host "  Removing $($_.Name)"
                Remove-Item -Path $_.FullName -Force
                git rm -f "App/.appSourceCopPackages/$($_.Name)" # *> $null
            }
        }

        # Move latest app
        Write-Host "Replacing previous Dynavision apps..." -ForegroundColor Yellow
        Write-Host "  Directory: $AppsDirectory"

        if (-not (Test-Path $PackagesDirectory)) {
            new-item -Path $PackagesDirectory -ItemType Directory
        }

        Get-ChildItem -Path $AppsDirectory -Filter 'Dynavision*.app' | Foreach-Object {
            Write-Host "  Adding $($_.Name)"        
            Move-item -Path $_.FullName -Destination $PackagesDirectory 
            git add -f "App/.appSourceCopPackages/$($_.Name)" # *> $null
        }     

        # Update AppSourceCop.json
        Write-Host "Updating AppSourceCop.json ..." -ForegroundColor Yellow
        Write-Host "  Directory: $AppDirectory"

        $AppSourceCopFile = Join-Path -Path $AppDirectory -ChildPath 'AppSourceCop.json'
        $AppSourceCopContent = Get-Content $AppSourceCopFile -raw | ConvertFrom-Json

        $VersionParsed = $null
        if ([Version]::TryParse($AppSourceVersion, [ref]$VersionParsed)) {
            $AppSourceVersion = $VersionParsed.ToString()
            Write-Host "  Update version to $AppSourceVersion"
            $AppSourceCopContent.version = $AppSourceVersion
            $Updated = $true    
        }
        else {    
            Write-Host " Version ignored."
        }

        $VersionParsed = $null
        if ([Version]::TryParse($MajorMinor, [ref]$VersionParsed)) {
            $MajorMinor = $VersionParsed.ToString()
            Write-Host "  Update obsoleteTagVersion to $MajorMinor"
            $AppSourceCopContent.obsoleteTagVersion = $MajorMinor
            $Updated = $true    
        }
        else {    
            Write-Host "  obsoleteTagVersion ignored."
        }

        if ($Updated) {
            $AppSourceCopContent | ConvertTo-Json -Depth 32 | Format-Json | Set-Content $AppSourceCopFile                     
            git add "App/.appSourceCopPackages/$($AppSourceCopFile.Name)" # *> $null
        }

        $Message = "Post release $($AppSourceVersion) -> Update previous apps"
        git stage .    
        git commit -q -m $Message # *> $null
        git tag -a $AppSourceVersion -m "Release $AppSourceVersion" # *> $null
    }

    if ($IncreaseAppVersion) {

        $Updated = $false

        Write-Host "Increase App Version" -ForegroundColor Green

        Write-Host "Settings ..." -ForegroundColor Yellow
        Write-Host "  AppVersion: $AppVersion"
        # Write-Host "  PlatformVersion: $PlatformVersion"
        # Write-Host "  ApplicationVersion: $ApplicationVersion"
        # Write-Host "  RuntimeVersion: $RuntimeVersion"
        Write-Host "  Reference: https://dev.azure.com/xxxx/Dynavision/_library?itemType=VariableGroups&view=VariableGroupView&variableGroupId=2&path=PipelineVariables"        

        # Update App.json version

        Write-Host "Updating App.json ..." -ForegroundColor Yellow
        Write-Host "  Directory: $AppDirectory"

        $AppFile = Join-Path -Path $AppDirectory -ChildPath 'App.json'
        $AppContent = Get-Content $AppFile -raw | ConvertFrom-Json

        $VersionParsed = $null
        if ([Version]::TryParse($AppVersion, [ref]$VersionParsed)) {

            $AppVersion = $VersionParsed.ToString()
            if ($VersionParsed -lt [version]$AppContent.version) {
                throw "  version $AppVersion should be higher than $($AppContent.version)"
            }

            Write-Host "  Update version to $AppVersion ($($AppContent.name))"
            $AppContent.version = $AppVersion
            Update-Dependencies -Publisher 'Dynavision' -Version $AppVersion
            $Updated = $true    
        }
        else {    
            Write-Host " Version ignored."
        }

        # runtime, platform, application
        # Update-Dependencies -InputObject $Configuration -Publisher 'Microsoft' -Version $ApplicationVersion

        if ($Updated) {
            $AppContent | ConvertTo-Json -Depth 32 | Format-Json | Set-Content $AppFile         
            # git add $AppFile *> $null
        }

        # Update AppSourceCop.json - obsoleteTagVersion
        # $MajorMinor = "$($DestinationVersionParsed.Major).$($DestinationVersionParsed.Minor)"

        $Message = "Post release $($AppSourceVersion) -> Increase version to $($AppVersion)"
        git stage .    
        git commit -q -m $Message # *> $null
    }        
}
waldo1001 commented 7 months ago

Thank you for describing your solution, @fvet .

Too bad to hear we disappoint you in this matter - but do know that it's a very conscious choice to never add steps that "changes your code" or "adds commits to your repo" . That's a box of Pandora that we will keep shut.

Arthurvdv commented 7 months ago

I would like to join this discussion. We're also struggling with the same challenge: How do I automate the content of the .appSourceCopPackages folder?

While do don't disagree with "box of Pandora" that @waldo1001 wan't to keep closed, I'm thinking can we take an other approach on this? I'm just thinking out loud here: An ALOps building block that populates an .appSourceCopPackages folder in a local directory of the Agent (or even better: in a Azure Storage), where we ourselves create the logic to commit this into our repo?

We've created something in PowerShell that on every PR updates the content of the .appSourceCopPackages folder. You can see below why it's not a good idea for us to write our own PowerShell 🙈 and have the experts at ALOps provide this service to us. (I was doubting if I would share this monster ugly written PowerShell, please don't judge and and don't use this as you'll probably end up in misery, due to the poor quality of the code).

parameters:
  country: 'W1'
  suppress_pullrequest: false

steps:
- ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
  - task: PowerShell@2
    displayName: 'Checkout $(System.PullRequest.SourceBranch)'
    env:
      SYSTEM_PULLREQUEST_SOURCEBRANCH: $(System.PullRequest.SourceBranch)
    inputs:
      targetType: 'inline'
      script: |
        Write-Host "##[command]git fetch"
        git fetch

        Write-Host "Branch name is $($env:SYSTEM_PULLREQUEST_SOURCEBRANCH.Replace('refs/heads/',''))"
        git checkout $env:SYSTEM_PULLREQUEST_SOURCEBRANCH.Replace('refs/heads/','') --

- task: PowerShell@2
  displayName: 'Verify AppSourceCop.json'
  condition: succeeded()
  env:
    BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory)
  inputs:
    targetType: 'inline'
    script: |
      $appFolder = (Join-Path $env:BUILD_SOURCESDIRECTORY 'App')
      $appSourceCopFile = (Join-Path $appFolder 'AppSourceCop.json')

      if (!(Test-Path -PathType Leaf $appSourceCopFile)) {
          Write-Host "##[command]Creating $appSourceCopFile"

          $json = @{ 
              baselinePackageCachePath = "./.appSourceCopPackages"
              mandatoryAffixes         = @("AFFIX1", "AFFIX2", "AFFIX3")
              supportedCountries       = @("COUNTRY1", "COUNTRY2", "COUNTRY2")
          }
          $json | ConvertTo-Json | Set-Content $appSourceCopFile
      }

      ### Grab the version from the AppSourceCop file
      Write-Host "##[command]Read $appSourceCopFile"
      $appSourceCopObject = Get-Content -Raw -Path $appSourceCopFile | ConvertFrom-Json

      $baselinePackageCachePath = $(Join-Path(Join-Path $env:BUILD_SOURCESDIRECTORY 'App') $($appSourceCopObject.baselinePackageCachePath))

      if ($appSourceCopObject.psobject.Properties.name -contains 'baselinePackageCachePath') {
          Write-Host "##[command]Found baselinePackageCachePath '$($appSourceCopObject.baselinePackageCachePath)'."
          Write-Host "##vso[task.setvariable variable=APPSOURCECOP_BASELINEPACKAGECACHEPATH]$baselinePackageCachePath"
          Write-Host "##[command]APPSOURCECOP_BASELINEPACKAGECACHEPATH:$baselinePackageCachePath"
      }
      else {
          Throw "Missing property 'baselinePackageCachePath' in $appSourceCopObject"
      }

      # Create baselinePackageCachePath if not exits
      if (!(Test-Path $baselinePackageCachePath)) {
          New-Item -Path $baselinePackageCachePath -ItemType Directory -Force | Out-Null
      }

- task: PowerShell@2
  displayName: 'Populate baselinePackageCachePath'
  condition: succeeded()
  env:
    AGENT_TEMPDIRECTORY: $(Agent.TempDirectory)
    BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory)
    APPSOURCECOP_BASELINEPACKAGECACHEPATH: $(APPSOURCECOP_BASELINEPACKAGECACHEPATH)
    COUNTRY: ${{ parameters.country }}
  inputs:
    targetType: 'inline'
    script: |
      Import-Module BcContainerHelper -DisableNameChecking
      Import-Module D365BCAppHelper -DisableNameChecking

      function Get-BusinessCentralArtifactIndex {
          Param (
              [string] $country = 'W1',
              [string] $version = '',
              [string] $select = 'Daily',
              [switch] $silent
          )

          if ($country.ToUpper() -eq 'BASE') {
              $country = 'W1'
          }

          if ($version -ne '') {
                Write-Host "Get-BcArtifactUrl -type Sandbox -country $country -version $version"
                $artifactPaths = Download-Artifacts -artifactUrl (Get-BcArtifactUrl -type Sandbox -country $country -version $version) -includePlatform
          }
          else {
                Write-Host "Get-BcArtifactUrl -type Sandbox -country $country -select $select"
                $artifactPaths = Download-Artifacts -artifactUrl (Get-BcArtifactUrl -type Sandbox -country $country -select $select) -includePlatform
          }
          $artifactAppFilesIndex = @{}

          $artifactAppFiles = Get-ChildItem -Path (Join-Path $artifactPaths[0] 'Extensions') -Filter *.app -Recurse
          $artifactAppFiles += Get-ChildItem -Path $artifactPaths[1] -Filter *.app -Recurse

          $artifactAppFiles | % {

              $xmlManifest = Get-D365BCManifestFromAppFile -Filename $_.FullName

              if (!$silent) {
                  Write-Host "**************************************"
                  Write-Host "$($spaces)* App File = [$($_.FullName)]"
                  Write-Host "$($spaces)"
                  Write-Host "$($spaces)* App.ID        = $($xmlManifest.App.Id)"
                  Write-Host "$($spaces)* App.Name      = $($xmlManifest.App.Name)"
                  Write-Host "$($spaces)* App.Publisher = $($xmlManifest.App.Publisher)"
                  Write-Host "$($spaces)* App.Version   = $($xmlManifest.App.Version)"
                  Write-Host "$($spaces)"
                  Write-Host "$($spaces)"
              }

              if (!( $artifactAppFilesIndex.ContainsKey($xmlManifest.App.Id))) {
                  $artifactAppFilesIndex.Add($xmlManifest.App.Id, $_.FullName)
              }
          }
          $artifactAppFilesIndex
      }

      function Resolve-BusinessCentralDependenciesFromArtifactIndex {
          Param(
              [Parameter(Mandatory = $true)]
              [hashtable] $artifactIndex,
              [Parameter(Mandatory = $true)]
              [string] $appsFolder,
              [string] $outputFolder = (Join-Path $appsFolder '.appSourceCopPackages'),
              [int] $lvl = -1,
              [string[]] $ignoredDependencies = @(),
              [string[]] $ignoredPublishers = @()
          )

          if ([String]::IsNullOrWhiteSpace($outputFolder)) { 
              $outputFolder = (Join-Path $appsFolder '.appSourceCopPackages') 
          }

          try {

              $spaces = ''
              $lvl++

              For ($i = 0; $i -le $lvl; $i++) {
                  $spaces += '   '
              }

              # Create outputFolder if not exits
              if (!(Test-Path $outputFolder)) {
                  New-Item -Path $outputFolder -ItemType Directory -Force | Out-Null
              }

              # Search for app.json(s)
              $apps = @(Get-ChildItem -Path (Join-Path $appsFolder '/*/app.json'))
              if ($apps.Count -eq 0) {
                  # Look for a single app.json in root
                  try {
                      $apps = @(Get-ChildItem -Path (Join-Path $appsFolder '/app.json') -ErrorAction SilentlyContinue)
                  }
                  catch {}
              }
              if ($apps.Count -eq 0) {
                  # If no app.json(s) found look for .app files.
                  $apps = @(Get-ChildItem -Path (Join-Path $appsFolder '*.app'))
              }

              Write-Host "$($spaces)$($apps.Count) apps found";

              $apps | % {
                  ### Prevent retrieving itself, so adding the AppId to the $ignoredDependencies list
                  if ($_.Extension -eq ".app" -and $lvl -eq 0) {
                      $xmlManifest = Get-D365BCManifestFromAppFile -Filename $_
                      $ignoredDependencies += $xmlManifest.App.Id
                      Write-Host "$($spaces)Added itself '$($xmlManifest.App.Name)' with AppId [$($xmlManifest.App.Id)] to ignored apps";
                  }
                  if ($_.Extension -eq ".json" -and $lvl -eq 0) {
                      $AppJson = Get-Content -Raw -Path $_ | ConvertFrom-Json
                      if ($AppJson.psobject.Properties.name -contains "id") {
                          $ignoredDependencies += $AppJson.id
                          Write-Host "$($spaces)Added itself '$($AppJson.name)' with AppId [$($AppJson.id)] to ignored apps";
                      }
                  }

                  if ($_.Name -eq "app.json") {
                      $appContext = "$($_.DirectoryName | Split-Path -Leaf)\app.json"
                      Write-Host "$($spaces)Get dependencies from '$appContext'"
                      $dependencies = Get-D365BCDependenciesFromJson -Filename $_
                  }
                  else {
                      $appContext = "$($_.Name)"
                      Write-Host "$($spaces)Get dependencies from '$appContext'"
                      $dependencies = Get-D365BCDependenciesFromFile -Filename $_
                  }

                  $dependencies | % {

                      if (Test-IsGuid $_.Id) {

                          if ($_.Publisher -in $ignoredPublishers) {
                              $ignoredDependencies += $($_)
                              Write-Host "$($spaces)Publisher $($_.publisher) in ignore list. Added $($_.name) with AppId[$($_.Id)] to ignored apps";
                          }

                          if (($_.Id -notin $ignoredDependencies) -and ($artifactIndex.ContainsKey($_.Id))) {
                              # Create temp folders
                              $tempAppFolder = Join-Path ((Get-Item -Path $env:AGENT_TEMPDIRECTORY).FullName) ([Guid]::NewGuid().ToString())
                              $tempAppDependenciesFolder = Join-Path $tempAppFolder 'dependencies'
                              try {
                                  New-Item -path $tempAppFolder -ItemType Directory -Force | Out-Null
                                  New-Item -path $tempAppDependenciesFolder -ItemType Directory -Force | Out-Null

                                  try {
                                      $tempAppDependencyFolder = Join-Path $tempAppDependenciesFolder $_
                                      New-Item -path $tempAppDependencyFolder -ItemType Directory -Force | Out-Null

                                      Copy-Item -Path $artifactIndex[$_.Id] -Destination $tempAppDependencyFolder -Force

                                      $dep = @(Get-ChildItem -Path (Join-Path $tempAppDependencyFolder '*.app'))
                                      $dep | % {

                                          if (!(Test-Path (Join-Path $outputFolder $_.Name))) {
                                              Copy-Item -Path $_ -Destination $outputFolder -Force

                                              Write-Host "$($spaces)The file '$($_.Name)' Copied to $($outputFolder)"

                                              Resolve-BusinessCentralDependenciesFromArtifactIndex -artifactIndex $artifactIndex -appsFolder $(Split-Path $_) -outputFolder $outputFolder -lvl $lvl -ignoredDependencies $ignoredDependencies -ignoredPublishers $ignoredPublishers
                                          }
                                          else {
                                              Write-Host "$($spaces)The App '$($_.Name)' already exists!"
                                          }
                                      }
                                  }
                                  catch {
                                      Write-Host "$($spaces)Copied 0 .app file(s)"
                                      Write-Host $_
                                  }
                              }
                              finally {
                                  Remove-Item -Path $tempAppFolder -Recurse -Force -ErrorAction SilentlyContinue
                              }
                          }
                          else {
                              Write-Host "$($spaces) App '$($_.Name)' with AppId [$($_.Id)] is not a default App from the bcartifacts.cache"
                          }
                      }
                      else {
                          Write-Host "$($spaces)No dependencies found in '$appContext'"
                      }
                  }  #end loop $dependencies
              } #end loop $apps
          }
          catch { 
              Write-Host $_
          }
          finally { }
      }

      function Get-D365BCDependenciesFromJson {
          [OutputType([array])]
          Param(
              [Parameter(Mandatory = $true)]
              [string] $Filename
          )
          $appJson = Get-Content -Raw -Path $Filename | ConvertFrom-Json

          $result = @()

          if ($appJson.psobject.Properties.name -contains 'platform') {
              $result += [pscustomobject] @{ Id = '8874ed3a-0643-4247-9ced-7a7002f7135d'
                  Name                          = 'System'
                  Publisher                     = 'Microsoft'
                  MinVersion                    = $appJson.platform
                  CompatibilityId               = '0.0.0.0'
              }
          }

          if ($appJson.psobject.Properties.name -contains 'application') {
              $result += [pscustomobject] @{ Id = 'c1335042-3002-4257-bf8a-75c898ccb1b8'
                  Name                          = 'Application'
                  Publisher                     = 'Microsoft'
                  MinVersion                    = $appJson.application
                  CompatibilityId               = '0.0.0.0'
              }
          }
          ### Return empty result when no dependencies property exists
          if ($appJson.psobject.Properties.name -notcontains "dependencies") {
              Write-Host "No addition dependencies, return application and platform dependencies"
              return $result
          }

          $appJson.dependencies | % {

              if ($_.psobject.Properties.name -contains "id") {
                  $id = $_.id
              }
              elseif ($_.psobject.Properties.name -contains "appId") {
                  $id = $_.appId
              }

              $result += [pscustomobject] @{ Id = $id
                  Name                          = $_.name
                  Publisher                     = $_.publisher
                  MinVersion                    = $_.version
                  CompatibilityId               = '0.0.0.0'
              }
          }
          return $result
      }

      function Test-IsGuid {
          [OutputType([bool])]
          param
          (
              [Parameter(Mandatory = $true)]
              [AllowEmptyString()]
              [string]$StringGuid
          )

          $ObjectGuid = [System.Guid]::empty
          return [System.Guid]::TryParse($StringGuid, [System.Management.Automation.PSReference]$ObjectGuid) # Returns True if successfully parsed
      }

      function Prep-BusinessCentralDependenciesFromArtifactIndex {
          param
          (
              [Parameter(Mandatory = $true)]
              [string] $outputFolder
          )

          # If the folder is already there, remove all contents
          if ((Get-ChildItem $outputFolder -Force | Select-Object -First 1 | Measure-Object).Count -ne 0) {
              Remove-Item -Path $outputFolder -Recurse -Force -ErrorAction SilentlyContinue
              New-Item -Path $outputFolder -ItemType Directory -Force | Out-Null
          }
      }

      function Post-BusinessCentralDependenciesFromArtifactIndex {
        param
        (
            [Parameter(Mandatory = $true)]
            [string] $outputFolder
        )

        $microsoftSystemAppFile = Join-Path $outputFolder 'System.app'

        if (Test-Path $microsoftSystemAppFile) {
            $xmlManifest = Get-D365BCManifestFromAppFile -Filename $microsoftSystemAppFile
            Rename-Item -Path $microsoftSystemAppFile -NewName "Microsoft_System_$($xmlManifest.App.Version).app"
        }
      }

      function Get-BusinessCentralApplicationVersionFromAppJson {
          [OutputType([string])]
          Param (
              [Parameter(Mandatory = $true)]
              [string] $appFolder
          )

          Write-Host "Read $(Join-Path $appFolder app.json)"
          $AppFileObject = Get-Content -Raw -Path (Join-Path $appFolder app.json) | ConvertFrom-Json

          $ApplicationVersion = $null
          if ([System.Version]::TryParse($AppFileObject.application, [ref]$ApplicationVersion)) {
              Write-Host "##[command]Found Application version $ApplicationVersion in App.json [$($AppFileObject.Name)]"
          }
          else {
              Write-Error -Message "Invalid Application version in App.json" -ErrorAction Stop
          }

          $PlatformVersion = $null
          if ([System.Version]::TryParse($AppFileObject.platform, [ref]$PlatformVersion)) {
              Write-Host "##[command]Found Platform version $PlatformVersion in App.json [$($AppFileObject.Name)]"
          }
          else {
              Write-Error -Message "Invalid Platform version in App.json [$($AppFileObject.Name)]" -ErrorAction Stop
          }

          if ([version]$PlatformVersion -gt [version]$ApplicationVersion) {
            Write-Error -Message "The platform version '$($PlatformVersion)' is higher then the application version '$($ApplicationVersion)' in App.json [$($AppFileObject.Name)]" -ErrorAction Stop
          }

          $version = $ApplicationVersion.Major.ToString() + "." + $ApplicationVersion.Minor.ToString()
          return $version
      }

      Prep-BusinessCentralDependenciesFromArtifactIndex `
          -outputFolder $env:APPSOURCECOP_BASELINEPACKAGECACHEPATH

      $applicationVersion = Get-BusinessCentralApplicationVersionFromAppJson `
          -appFolder $(Join-Path $env:BUILD_SOURCESDIRECTORY 'App')

      Resolve-BusinessCentralDependenciesFromArtifactIndex `
          -artifactIndex (Get-BusinessCentralArtifactIndex -country $env:COUNTRY -version $applicationVersion -silent) `
          -appsFolder $(Join-Path $env:BUILD_SOURCESDIRECTORY 'App') `
          -outputFolder $env:APPSOURCECOP_BASELINEPACKAGECACHEPATH

      Post-BusinessCentralDependenciesFromArtifactIndex `
          -outputFolder $env:APPSOURCECOP_BASELINEPACKAGECACHEPATH

- task: PowerShell@2
  displayName: 'Download current version'
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
    AGENT_TEMPDIRECTORY: $(Agent.TempDirectory)
    BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory)
    APPSOURCECOP_BASELINEPACKAGECACHEPATH: $(APPSOURCECOP_BASELINEPACKAGECACHEPATH)
    SYSTEM_COLLECTIONURI: $(System.CollectionUri)
  inputs:
    targetType: 'inline'
    script: |
      Import-Module D365BCAppHelper -DisableNameChecking

      # Get the AppId from the app.json file
      $AppJson = Get-Content -Raw -Path (Join-Path (Join-Path $env:BUILD_SOURCESDIRECTORY 'App') 'app.json') | ConvertFrom-Json
      if (!($AppJson.psobject.Properties.name -contains "id")) {
          Throw "Can't find id in AppJson"
      }

      try {

          # Create temporary folder
          $tempAppFolder = Join-Path ((Get-Item -Path $env:AGENT_TEMPDIRECTORY).FullName) ([Guid]::NewGuid().ToString())
          New-Item -path $tempAppFolder -ItemType Directory -Force | Out-Null

          # Download the current version
          az artifacts universal download `
              --organization $env:SYSTEM_COLLECTIONURI `
              --feed 'BusinessCentral' `
              --name $($AppJson.id) `
              --version '*' `
              --path $tempAppFolder

          $appFile = Get-ChildItem -Path $tempAppFolder -Depth 0 -Filter *.app | Select-Object -First 1 | % { $_.FullName }

          if (-Not ([String]::IsNullOrWhiteSpace($appFile))) { 
              # Grab the version (xmlManifest) from the downloaded $appFile
              $xmlManifest = Get-D365BCManifestFromAppFile -Filename $appFile

              Write-Host "Copy-Item -Path $appFile -Destination (Join-Path $env:APPSOURCECOP_BASELINEPACKAGECACHEPATH (Split-Path $appFile -leaf)) -Force"
              Copy-Item -Path $appFile -Destination (Join-Path $env:APPSOURCECOP_BASELINEPACKAGECACHEPATH (Split-Path $appFile -leaf)) -Force

              # Get the AppSourceCop.json file (no check if file exists, in previous step already covered)
              $appFolder = (Join-Path $env:BUILD_SOURCESDIRECTORY 'App')
              $appSourceCopFile = (Join-Path $appFolder 'AppSourceCop.json')
              $appSourceCopObject = Get-Content -Raw -Path $appSourceCopFile | ConvertFrom-Json

              if ($appSourceCopObject.PSObject.Properties.name -notcontains 'version') {
                  $appSourceCopObject | Add-Member -MemberType NoteProperty -Name 'version' -Value $($xmlManifest.App.Version)
              }
              else {
                  $appSourceCopObject.version = $($xmlManifest.App.Version)
              }
              Write-Host "Set version $($appSourceCopObject.version) to $appSourceCopFile"
              $appSourceCopObject | ConvertTo-Json | Set-Content $appSourceCopFile
          }
      }
      catch {
          Write-Host $_
      }
      finally {
          exit 0
      }

### Get possible dependency apps
- template: dependency_artifacts.yml
  parameters:
    target_folder: $(APPSOURCECOP_BASELINEPACKAGECACHEPATH)

### Commit logic
- ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
  - task: PowerShell@2
    displayName: 'Create Commit'
    env:
      APPSOURCECOP_BASELINEPACKAGECACHEPATH: $(APPSOURCECOP_BASELINEPACKAGECACHEPATH)
    inputs:
      targetType: 'inline'
      script: |
        $env:GIT_REDIRECT_STDERR = '2>&1' # Do not report git output as error... However error is also only text now

        $appFolder = (Join-Path $env:BUILD_SOURCESDIRECTORY 'App')
        $appSourceCopFile = (Join-Path $appFolder 'AppSourceCop.json')

        $AreChangedAppSourceCopPackagesFiles = $(git status $($env:APPSOURCECOP_BASELINEPACKAGECACHEPATH) --porcelain | Measure-Object | Select-Object -expand Count)
        $IsChangedAppSourceCopJsonFile = $(git status $appSourceCopFile --porcelain | Measure-Object | Select-Object -expand Count)
        if (($AreChangedAppSourceCopPackagesFiles -gt 0) -or ($IsChangedAppSourceCopJsonFile -gt 0)) {

            git config user.email "noreply@vanroey.be"
            git config user.name "Azure DevOps"

            if ($AreChangedAppSourceCopPackagesFiles -gt 0) {
                Write-Host "##[command]git add '$($env:APPSOURCECOP_BASELINEPACKAGECACHEPATH)'"
                git add "$($env:APPSOURCECOP_BASELINEPACKAGECACHEPATH)"
            }
            if ($IsChangedAppSourceCopJsonFile -gt 0) {
                Write-Host "##[command]git add '$($appSourceCopFile)'"
                git add "$appSourceCopFile"
            }

            Write-Host "##[command]git commit -m 'Updated files in BaselinePackageCachePath'"
            git commit -m 'Updated files in BaselinePackageCachePath'

            Write-Host "##[command]git push"
            git push 
        }
        else {
            Write-Host "##[command]GIT: No untracked files found..."
        }
waldo1001 commented 7 months ago

Look, we won't do any git-changes. Don't ask that from us.

What we can do (suggestion): in the compile-step, a "previous_app" location setting which will download it from a step. We will extract version info form it, and put in AppSourceCopj.son. You can feed it simply with a file copy, like the default steps in DevOps: image Or any kind of copy you use to save your build artifact to a location that you use to save the previous of your apps.

How does that sound?

Arthurvdv commented 7 months ago

Is it possible to have this as a separate step and not integrated with the compile-step? Mostly because we want to add these files after the release has gone through on AppSource and not (re)compile the app again.

The steps that I would expect this to do is: A) Search for a baselinePackageCachePath property in the AppSourceCop.json file. B) Retrieve platform, application and dependencies from app.json file. C) Replace artifact App with the "released" version in .appSourceCopPackages folder D) Replace System, System Application, Application, and Base App files in .appSourceCopPackages folder E) Replace dependencies artifacts in .appSourceCopPackages folder F) Update version property in AppSourceCop.json file from the artifact of step C)

There are some challenges I believe A) How do we determine what the latest release is in AppSource? -- Do we need to provide this as a param? -- Can we utilize the API of the AppSource to retrieve current the published version? B) Artifact of the App -- Receiving the compiled artifact of the App itself could be challenging, due to the different ADO configurations out there. Maybe we should make the jump here to specify a NuGet feed, where the artifacts can be resolved? The benefit of an NuGet feed could also applied for the Microsoft Apps ánd we could retrieve symbols of all other AppSource apps.

I'm open for a discussion how other ALOps users are looking at this.

Arthurvdv commented 7 months ago

An additional thought: Currently we handle versioning of the application/platform different from dependencies.

The Platform and Application of the app.json, we match the with Major.Minor version of the artifact. So Application set to 22.3 will place the symbols of the artifact of 22.3 in the .appSourceCopPackages folder.

While the dependencies of the other apps, we target the newest release, not matter what the version of that dependency is set. The idea here is that on installing the App, the latest versions of the dependencies are installed (in case not already there).

waldo1001 commented 6 months ago

We'll see what we can do with the AppSource/PartnerCenter API ..