Publish-PerTenantExtensionApps Function Randomly Times Out

[Matteo406]

Description

The Publish-PerTenantExtensionApps function occasionally times out at a random app from the list of apps being published. This issue occurs intermittently and does not seem to be tied to any specific app or environment configuration. Additionally, it does not return a proper error message, just 0000 as shown in the image below.

Script Snippet

Here is the relevant part of the script where the issue occurs:

Environment Variables

BC_ENVIRONMENT: $bc_environment
BC_TENANT: $bc_tenant
BC_CLIENTID: $bc_clientid
BC_CLIENTSECRET: $bc_clientsecret
BC_SCHEMASYNCMODE: $bc_schemaSyncMode

Apps

D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Base_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Billing_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Device_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Finance_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Interfaces_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Logistics_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Order_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos POS_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Reports and Rolecenter_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Swiss Salary_24.0.0.75627.app,
D:\a\r1\a\Swisslos Project\output\Apps\bossinfo.ch Swisslos Win_24.0.0.75627.app

Publish-PerTenantExtensionApps

Publish-PerTenantExtensionApps -tenantId $bc_tenant -clientId $bc_clientid -clientSecret $bc_clientsecret -environment $bc_environment -appFiles $Apps -schemaSyncMode $bc_schemaSyncMode

• Blue: Not Found
• Green: The process was not found

Steps to Reproduce

Run the script in a DevOps pipeline after committing to the development branch. The script collects all .app files and attempts to publish them using the Publish-PerTenantExtensionApps function. Observe that the function occasionally times out at a random app.

Expected Behavior

The Publish-PerTenantExtensionApps function should publish all apps without timing out.

Actual Behavior

The function times out at a random app from the list, causing the script to fail intermittently.

Additional Information

• The issue does not seem to be tied to any specific app or environment configuration.
• The timeout occurs randomly, making it difficult to pinpoint the exact cause.

Script

$bc_environment = $ENV:BC_ENVIRONMENT
$bc_tenant = $ENV:BC_TENANT
$bc_clientid = $ENV:BC_CLIENTID
$bc_clientsecret = $ENV:BC_CLIENTSECRET
$bc_schemaSyncMode = $ENV:BC_SCHEMASYNCMODE

if (($bc_environment -eq '') -or ($bc_tenant -eq '') -or ($bc_clientid -eq '') -or ($bc_clientsecret -eq '')) {
  throw 'Variables missing: bc_environment, bc_tenant, bc_clientid, bc_clientsecret'
}

$root = $ENV:AGENT_RELEASEDIRECTORY
$artfact = $ENV:RELEASE_PRIMARYARTIFACTSOURCEALIAS
$rootdir = $root + '\' + $artfact + '\output\Apps'

$InstallBcContainerHelperPath = $root + '\' + $artfact + '\output\Install-BcContainerHelper.ps1'
. $InstallBcContainerHelperPath

$AppsFolder = Get-ChildItem -Path $rootdir -Filter "*.app"
$Apps = ''
$AppNames = @()
foreach ($AppsFile in $AppsFolder) {
  $Apps = $Apps + $AppsFile.FullName + ','
  $AppNames += $AppsFile.Name.Split("_")[1]
}
$Apps = $Apps.Substring(0, $Apps.Length - 1)

# use API to uninstall currently installed apps
$body = @{grant_type="client_credentials";scope="https://api.businesscentral.dynamics.com/.default";client_id=$bc_clientid;client_secret=$bc_clientsecret}
$oauth = Invoke-RestMethod -Method Post -Uri $("https://login.microsoftonline.com/$bc_tenant/oauth2/v2.0/token") -Body $body
$authHeaders = @{ 
    "Authorization" = "Bearer $($oauth.access_token)"; 
    "Content-Type" = "application/json" 
}

$companyId = (Invoke-RestMethod -Headers $authHeaders -Method Get -Uri "https://api.businesscentral.dynamics.com/v2.0/$bc_tenant/$bc_environment/api/microsoft/automation/v1.0/companies").value.id[0]
$extensions = Invoke-RestMethod -Headers $authHeaders -Method Get -Uri "https://api.businesscentral.dynamics.com/v2.0/$bc_tenant/$bc_environment/api/microsoft/automation/v1.0/companies($companyId)/extensions"

if ($extensions.value.displayName -contains "Swisslos Uninstall PTEs") {
    Write-Host "Uninstalling installed Apps"
    $AppHash = @{}
    for ($i = 0; $i -lt $extensions.value.packageId.Count; $i++) {
        if (!($AppHash.Contains($extensions.value.id[$i])) -and $extensions.value.displayName[$i] -in $AppNames) {
            $AppHash.Add($extensions.value.id[$i], $false)
        }
    }

    if ($AppHash.Count -gt 0) {
        $iterations = 0
        $AppIds = $AppHash.Keys.Clone()
        do {
            $iterations++
            foreach ($AppId in $AppIds) {
                if ($AppHash[$AppId]) {
                    continue
                }
                try {
                    $APIBody = @{ "guid" = $AppId } | ConvertTo-Json
                    $null = Invoke-WebRequest -Headers $authHeaders -Method post -Uri "https://api.businesscentral.dynamics.com/v2.0/$bc_tenant/$bc_environment/api/bossinfo/uninstallPTEs/v1.0/companies($companyId)/uninstallPTEs" -Body $APIBody
                    $AppHash[$AppId] = $true
                } catch { }
            }
        } while ($AppHash.values -contains $false -and $iterations -le $AppIds.Count)
    } else {
        Write-Host "No Apps to uninstall"
    }
} else {
    Write-Host "Swisslos Uninstall PTEs not installed in targeted tenant"
}

Publish-PerTenantExtensionApps -tenantId $bc_tenant -clientId $bc_clientid -clientSecret $bc_clientsecret -environment $bc_environment -appFiles $Apps -schemaSyncMode $bc_schemaSyncMode

Request

Could you please investigate this issue and provide guidance on how to prevent the Publish-PerTenantExtensionApps function from timing out? If additional information is needed, please let me know.

Thank you!

[freddydk]

I have heard about this problem before. How often does this happen? (1% of the times, 10% or 50%) I have tried to get a repro of this problem for quite some time. If you can provide me a .zip file with your apps (freddyk at microsoft dot com), then I can try publishing this to an environment until it fails and maybe find a solution on where to add resiliency to the process in order to not fail.

[Matteo406]

The problem occurs in about 50% of our pipeline runs. We have not been able to find the difference in environment or code between successful and failed runs. I will send you the zip file with all the app as soon as possible. Thanks in advance

[Matteo406]

Recent Scenario

I hope that you have received my email and that there was enough information in there to reproduce the bug.

Here is a recent scenario we had in more detail:

While inspecting the result of the failed pipeline run, we noticed that the application that caused the unknown error was still listed as installed with the correct version, as you can see in the second image. The list in the last image shows the same application listed again with the state "unknown" and the correct version.

The next pipeline run was successfully completed without any changes. I hope this gives a bit more context

error message app is listed as installed app has to correct version and state "unknown"