Pubish Apps to Sandbox recently fails sometimes

[HeWessel]

Describe the issue

We are using the function Publish-PerTenantExtensionApps in our Scripts to automate Publish new App versions to SaaS Environments. There are recently some Problems with the result of the Function. The strange thing is, that sometimes we can publish without any Problems and sometimes the Error occurs. In both Cases the App will be uploaded and also installed. Are we missing any Params for the function or is there a known issue of my problem? The Problem occurs also on different Environments but that can be a result of sharing the same Powershell Library and using the same script with different Params for our function.

Scripts used to create container and cause the issue

we just created a basic Powershell function to use the Function
function Import-BCAppsToSaaS {
    [cmdletbinding()]
    param (
         [string]$Path,     
         [Hashtable] $bcAuthContext,
         [string]$environment
    )

    $bcAuthContext = Renew-BcAuthContext -bcAuthContext $bcAuthContext       
    Publish-PerTenantExtensionApps -bcAuthContext $bcAuthContext -environment $environment -appFile $Path
}

Full output of scripts

Company  has id XXXX-XXXX-XX-XXX-XXXXX
Extensions before:
 - AMC Banking 365 Fundamentals, Version 21.4.52563.52836, Installed=True
 - API Reports - Finance, Version 21.4.52563.52836, Installed=True
 - Application, Version 21.4.52563.52836, Installed=True
 - Base Application, Version 21.4.52563.54065, Installed=True
 - Business Central Cloud Migration - Previous Release, Version 21.4.52563.52836, Installed=True
 - Business Central Cloud Migration - Previous Release (DE), Version 21.4.52563.52836, Installed=True
 - Business Central Cloud Migration API, Version 21.4.52563.52836, Installed=True
 - Business Central Intelligent Cloud, Version 21.4.52563.52836, Installed=True
 - Company Hub, Version 21.4.52563.52836, Installed=True
 - Contoso Coffee Demo Dataset, Version 21.4.52563.52836, Installed=True
 - Contoso Coffee Demo Dataset (DE), Version 21.4.52563.52836, Installed=True
 - Data Archive, Version 21.4.52563.52836, Installed=True
 - Data Search, Version 21.4.52563.52836, Installed=True
 - ELSTER VAT Localization for Germany, Version 21.4.52563.52836, Installed=True
 - Email - Current User Connector, Version 21.4.52563.52836, Installed=True
 - Email - Microsoft 365 Connector, Version 21.4.52563.52836, Installed=True
 - Email - Outlook REST API, Version 21.4.52563.52836, Installed=True
 - Email - SMTP API, Version 21.4.52563.52836, Installed=True
 - Email - SMTP Connector, Version 21.4.52563.52836, Installed=True
 - Essential Business Headlines, Version 21.4.52563.52836, Installed=True
 - Intelligent Cloud Base, Version 21.4.52563.52836, Installed=True
 - Intrastat Core, Version 21.4.52563.52836, Installed=True
 - Intrastat DE, Version 21.4.52563.52836, Installed=True
 - Late Payment Prediction, Version 21.4.52563.52836, Installed=True
 - customApp, Version 21.0.0.2, Installed=True
 - Payment Links to PayPal, Version 21.4.52563.52836, Installed=True
 - Recommended Apps, Version 21.4.52563.52836, Installed=True
 - Sales and Inventory Forecast, Version 21.4.52563.52836, Installed=True
 - Send To Email Printer, Version 21.4.52563.52836, Installed=True
 - Service Declaration, Version 21.4.52563.52836, Installed=True
 - Shopify Connector, Version 21.4.52563.52836, Installed=True
 - Simplified Bank Statement Import, Version 21.4.52563.52836, Installed=True
 - System Application, Version 21.4.52563.52836, Installed=True
 - Troubleshoot FA Ledger Entries, Version 21.4.52563.52836, Installed=True
 - Universal Print Integration, Version 21.4.52563.52836, Installed=True
 - WorldPay Payments Standard, Version 21.4.52563.52836, Installed=True

customApp_21.0.0.3.app - upgrading...ERROR Der Server hat eine Protokollverletzung ausgeführt.. Section=ResponseStatusLine
bei <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\BCContainerHelper\4.0.14\Saas\Publish-PerTenantExtensionApps.ps1: Zeile 173
bei <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\BCContainerHelper\4.0.14\AppHandling\Sort-AppFilesByDependencies.ps1: Zeile 176
bei Sort-AppFilesByDependencies, C:\Program Files\WindowsPowerShell\Modules\BCContainerHelper\4.0.14\AppHandling\Sort-AppFilesByDependencies.ps1: Zeile 175
bei Publish-PerTenantExtensionApps, C:\Program Files\WindowsPowerShell\Modules\BCContainerHelper\4.0.14\Saas\Publish-PerTenantExtensionApps.ps1: Zeile 116
bei Import-BCAppsToSaaS, D:\devops\agent1\_work\180\PowershellTools\Scripts\Import-BCAppsToSaaS.ps1: Zeile 70
bei <ScriptBlock>, D:\devops\agent1\_work\_temp\e0012e53-947f-4c72-b1ba-c4651624b8ae.ps1: Zeile 32
bei <ScriptBlock>, <Keine Datei>: Zeile 1

Extensions after:
 - AMC Banking 365 Fundamentals, Version 21.4.52563.52836, Installed=True
 - API Reports - Finance, Version 21.4.52563.52836, Installed=True
 - Application, Version 21.4.52563.52836, Installed=True
 - Base Application, Version 21.4.52563.54065, Installed=True
 - Business Central Cloud Migration - Previous Release, Version 21.4.52563.52836, Installed=True
 - Business Central Cloud Migration - Previous Release (DE), Version 21.4.52563.52836, Installed=True
 - Business Central Cloud Migration API, Version 21.4.52563.52836, Installed=True
 - Business Central Intelligent Cloud, Version 21.4.52563.52836, Installed=True
 - Company Hub, Version 21.4.52563.52836, Installed=True
 - Contoso Coffee Demo Dataset, Version 21.4.52563.52836, Installed=True
 - Contoso Coffee Demo Dataset (DE), Version 21.4.52563.52836, Installed=True
 - Data Archive, Version 21.4.52563.52836, Installed=True
 - Data Search, Version 21.4.52563.52836, Installed=True
 - ELSTER VAT Localization for Germany, Version 21.4.52563.52836, Installed=True
 - Email - Current User Connector, Version 21.4.52563.52836, Installed=True
 - Email - Microsoft 365 Connector, Version 21.4.52563.52836, Installed=True
 - Email - Outlook REST API, Version 21.4.52563.52836, Installed=True
 - Email - SMTP API, Version 21.4.52563.52836, Installed=True
 - Email - SMTP Connector, Version 21.4.52563.52836, Installed=True
 - Essential Business Headlines, Version 21.4.52563.52836, Installed=True
 - Intelligent Cloud Base, Version 21.4.52563.52836, Installed=True
 - Intrastat Core, Version 21.4.52563.52836, Installed=True
 - Intrastat DE, Version 21.4.52563.52836, Installed=True
 - Late Payment Prediction, Version 21.4.52563.52836, Installed=True
 - customApp, Version 21.0.0.2, Installed=True
 - Payment Links to PayPal, Version 21.4.52563.52836, Installed=True
 - Recommended Apps, Version 21.4.52563.52836, Installed=True
 - Sales and Inventory Forecast, Version 21.4.52563.52836, Installed=True
 - Send To Email Printer, Version 21.4.52563.52836, Installed=True
 - Service Declaration, Version 21.4.52563.52836, Installed=True
 - Shopify Connector, Version 21.4.52563.52836, Installed=True
 - Simplified Bank Statement Import, Version 21.4.52563.52836, Installed=True
 - System Application, Version 21.4.52563.52836, Installed=True
 - Troubleshoot FA Ledger Entries, Version 21.4.52563.52836, Installed=True
 - Universal Print Integration, Version 21.4.52563.52836, Installed=True
 - WorldPay Payments Standard, Version 21.4.52563.52836, Installed=True
Publish-PerTenantExtensionApps Telemetry Correlation Id: f3b241bc-8e33-43df-94f4-e169b3c9690e
##[error]Der Server hat eine Protokollverletzung ausgeführt.. Section=ResponseStatusLine
In C:\Program Files\WindowsPowerShell\Modules\BCContainerHelper\4.0.14\Saas\Publish-PerTenantExtensionApps.ps1:221 
Zeichen:13
+             throw (GetExtendedErrorMessage $_)
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Der Server hat ...ponseStatusLine:String) [], RuntimeException
    + FullyQualifiedErrorId : Der Server hat eine Protokollverletzung ausgeführt.. Section=ResponseStatusLine

Additional context

• It sometimes occurs but i cant recognize patterns there
• in the past it worked without any problems. The first time it occurs was the 22. Februrary

[Antebur]

Hello we face the same issue, is there any Solution or Workaround to solve ?

[freddydk]

Was the app installed/upgraded? I am asking in order to learn whether the problem is during getting the install/upgrade status - or whether the problem actually occurs when trying to install/upgrade

[jonaswre]

We recently also have issues when publishing from vscode. It seems like the sandboxes are generally less reliable then they used to be. Somethings publishing an app takes 10 min.

[HeWessel]

Until now we only faced the Issue during upgrade the App. I will try to reproduce the Problem while publishing a new App to the Sandbox.

[Antebur]

We face the issue using Devops Release Pipeline with outbound Powershell.

[freddydk]

Does this problem still occur? Does anybody have a good repro?

[Antebur]

Hi Freddy, yes the behaviour is still there. We currently have an workaround doing a Try and Catch, wait 3 minutes and then retry to install the app. Usually after 3 minutes the app is installed and when then second run is triggered it will say "app already installed" and the rollout is succesfully. How can i help you find the issue ? It could be repo installing an Devops Agent on your local PC an Deploy to an BC Online.

[freddydk]

Thanks - will investigate and get back

[Antebur]

Hi Freddy below the error Message still showing up

"appname_21.0.0.1.app - publishing and installing...ERROR The server committed a protocol violation. Section=ResponseStatusLine at , C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\5.0.3\Saas\Publish-PerTenantExtensionApps.ps1: line 175 at , C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\5.0.3\AppHandling\Sort-AppFilesByDependencies.ps1: line 185 at Sort-AppFilesByDependencies, C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\5.0.3\AppHandling\Sort-AppFilesByDependencies.ps1: line 184 at Publish-PerTenantExtensionApps, C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\5.0.3\Saas\Publish-PerTenantExtensionApps.ps1: line 118 at , D:\ProgramData\Agentpool\agent-1_work\r7\a_Azure Pipeline\application\scripts\Release\Release_Pipeline_Cloud.ps1: line 17 at , D:\ProgramData\Agentpool\agent-1_work_temp\a8fd1acb-451e-4a3f-92af-a5b5a40860a5.ps1: line 4 at , : line 1"

[HeWessel]

Hey Freddy,

i was able to check the different circumstances. Here is what I noticed so far. Publish in a fresh Environment

• 1x Time work without Problem.
• 2x I had the Problem above Publish in a Environment where it the App already exists and had to be upgraded
• 6x Time work without Problem.
• 8x I had the Problem above

I also saw one time that the App Upgrade was started at the same time to the Production and the Sandbox Environment. The Upgrade in the Sandbox failed and the Upgrade in the Production had no Problems. A few days later it was the opposite.

To temporary handle the Problem we changed our Pipeline and set the "retryCountOnTaskFailure" in the Pipeline. If any Error occurs while Upload the App, a Delay will start, before the next Retry. So the Script will try again after a few Minutes and see that the right Version is already installed. But thats hopefully only a temporary Solution for the Problem.

I hope that can help you. If you need any further informations let me now.

[freddydk]

Thanks

Would it be possible to get the apps you are publishing to see whether I can repro using your script above?

[Antebur]

if i remember correctly it occurs randomly and only with a local Agent. When using an Azure Agent there is no problem.

[HeWessel]

I also made tests with the Al Go Project, where the Costumerlist is only extended with an OnOpenPage Trigger. And I had the same Results. I attached the Example Project. But we also only uses local Agents.

Custom App.zip

[freddydk]

For AL-Go projects, you could try to set:

"runs-on": "ubuntu-latest",
"gitHubRunner": "self-hosted",
"gitHubRunnderShell"; "powershell"

in .github/AL-Go-Settings.json

and run Update AL-Go System Files.

This would cause you to use self-hosted runners for the time consuming build job, but GitHub hosted Linux runners (cheap and fast) for the rest.

[Antebur]

For AL-Go projects, you could try to set:

"runs-on": "ubuntu-latest",
"gitHubRunner": "self-hosted",
"gitHubRunnderShell"; "powershell"

in .github/AL-Go-Settings.json

and run Update AL-Go System Files.

This would cause you to use self-hosted runners for the time consuming build job, but GitHub hosted Linux runners (cheap and fast) for the rest.

Hi Freddy do you have some setting suggestions for Devops ?

[freddydk]

@Pomaroli - that all depends on the solution/script you are using on DevOps. AL-Go has a lot of functionality, which is controlled by settings - this is not GitHub or Azure DevOps functionality.

[Antebur]

Okey i understand, we not using AL-Go for now maybe in future. We are using Azure Devops and i developed myself Build and Release Pipelines based on YAML and Powershell. To Compile and Test the AL-Source code we use Docker Instance's. After the App is compiled sucessfully we deploy using the In-Build Release-Pipeline functionality of Azure Devops with Powershell. Basically we authenticate against BC Online an then Deploy PTE nothing special there. Maybe im completely wrong but i could imagine the issue is caused because of timeouts on responses occuring while the System ist upgrading.

[HeWessel]

@freddydk sorry for the late response. I forgot that there is also the Al-Go Project on GitHub. I just wanted to tell you that I just created in VS-Code a basic project with the function AL-GO. So there is an app initiated with the Page Extension on CustomerListExt with the "Hello World" Messagebox. I had the same problem uploading this basic app with one Page extension to my Saas Environment. For the moment, I extended our pipelines so that the PTE-Upload retries after a delay of 5 minutes. Afterward, the app is published, and no app will be uploaded again. So at the moment, I tried the first install and also upgraded over self hosted Agents. The problem still sadly occurs. Any other suggestions on what I can try? If you need any further information let me know.

[HeWessel]

@freddydk This problem still bothers us. Especially when we try to deliver more than one App into the Cloud and the Problem occurs at every App. Any Updates on this topic?

[freddydk]

@HeWessel - did you try to switch to GitHub or Azure DevOps hosted runners to see whether this solves the issue? I will grab the sample you provided last and make some tests with that to see if I can repro.

[Antebur]

Hi There, @freddydk did you changes something ? Since some days it seems do work fine now. Anybody else noticed some new behaviour ?

2024-01-10T09:26:38.7571725Z xy.app - upgrading....................completed

[freddydk]

I didn't - maybe they fixed something in BC online to solve this

[Antebur]

It did it again, sometimes it doesn't work. I haven't figured out what the connection is yet