Suggestion: More Resilient on Deployment (2)

[DanielGoehler]

We serialized the deployment process like suggested in #446, but we still get this error message from time to time. There was no other connection at the same time with the same ClientId to this tenant. If a http status 429 was returned, slow down would be best. Also in two cases, we got the server error message (Too many requests reached. Actual (101). Maximum (100).), mostly we saw the error message No overload can be found for "GetExtensions" and the following argument count: "1". (Translated)

##[section]Starting: ALOps Extension API
==============================================================================
Task         : ALOps Extension API
Description  : Get/Publish extensions with the Business Central API
Version      : 1.446.2728
Author       : Hodor
Help         : Get/Publish extensions with the Business Central API.
==============================================================================
*** Validate configuration
*** Task Inputs:

name                                                                                                              value
----                                                                                                              -----
usedocker                                                                                                         False
fixed_tag                                                                                                              
interaction                                                                                                       batch
api_endpoint            ...central.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api
apiversion                                                                                                         v1.0
authentication                                                                                                    oauth
azure_tenant_id                                                                    e825b00f-b640-4d87-b5d8-24a5d78cef18
azure_app_client_id                                                                ***
azure_app_client_secret                                                           ***
username                                                                                                               
password                                                                                                               
bccompany                                                                                                              
artifact_path                                                                                      C:\agent\_work\r29\a
artifact_filter                                                                                                   *.app
showdeploymentstatus                                                                                               True

*** For documentation, please visit   : https://www.alops.be/documentation

*** Imported required PS-Functions
*** Parse Task Parameters
*** Creating API Connector with endpoint [https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api]
*** API Version: v1.0
*** Interaction Mode: batch
*** API Authentication: Using Azure OAuth Authentication []
*** Get OAuth token for tenant [e825b00f-b640-4d87-b5d8-24a5d78cef18]
*** Communication: Service-2-Service
##[command]Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/e825b00f-b640-4d87-b5d8-24a5d78cef18/oauth2/token"
*** List Companies
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
*** Selected Company:  []
##[error]Für "GetExtensions" und die folgende Argumenteanzahl kann keine Überladung gefunden werden: "1".
##[warning]RetryHelper encountered task failure, will retry (attempt #: 1 out of 5) after 1000 ms
*** Validate configuration
*** Task Inputs:

name                                                                                                              value
----                                                                                                              -----
usedocker                                                                                                         False
fixed_tag                                                                                                              
interaction                                                                                                       batch
api_endpoint            ...central.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api
apiversion                                                                                                         v1.0
authentication                                                                                                    oauth
azure_tenant_id                                                                    e825b00f-b640-4d87-b5d8-24a5d78cef18
azure_app_client_id                                                                ***
azure_app_client_secret                                                           ***
username                                                                                                               
password                                                                                                               
bccompany                                                                                                              
artifact_path                                                                                      C:\agent\_work\r29\a
artifact_filter                                                                                                   *.app
showdeploymentstatus                                                                                               True

*** For documentation, please visit   : https://www.alops.be/documentation

*** Imported required PS-Functions
*** Parse Task Parameters
*** Creating API Connector with endpoint [https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api]
*** API Version: v1.0
*** Interaction Mode: batch
*** API Authentication: Using Azure OAuth Authentication []
*** Get OAuth token for tenant [e825b00f-b640-4d87-b5d8-24a5d78cef18]
*** Communication: Service-2-Service
##[command]Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/e825b00f-b640-4d87-b5d8-24a5d78cef18/oauth2/token"
*** List Companies
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies'

id                : 0d30d954-ca0b-49a1-937c-49b7000ffebb
systemVersion     : 19.2.32968.33612
name              : CRONUS
displayName       : CRONUS
businessProfileId : 
systemCreatedAt   : 0001-01-01T00:00:00Z
systemCreatedBy   : 00000000-0000-0000-0000-000000000000
systemModifiedAt  : 0001-01-01T00:00:00Z
systemModifiedBy  : 00000000-0000-0000-0000-000000000000

*** Selected Company: CRONUS [0d30d954-ca0b-49a1-937c-49b7000ffebb]
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies(0d30d954-ca0b-49a1-937c-49b7000ffebb)/extensions'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
*** Installed Extensions:

*** Published Extensions (not installed):

*** Import NAV/BC Management DLL's
*** ServiceTier not specified, loading default assemblies.
*** Default Assemblies not found. Load via BC-Artifacts
*** Install/Update BCContainerHelper [https://github.com/microsoft/navcontainerhelper]
*** Check BcContainerHelper
*** Load BcContainerHelper
BcContainerHelper version 3.0.0
BcContainerHelper emits usage statistics telemetry to Microsoft

ModuleType Version    Name                                ExportedCommands                                             
---------- -------    ----                                ----------------                                             
Script     3.0.0      BcContainerHelper                   {Add-FontsToBcContainer, Add-GitToAlProjectFolder, Backup-...

##[command]"Get-BCArtifactUrl" 
Name                           Value                                                                                   
----                           -----                                                                                   
select                         Latest                                                                                  
type                           OnPrem                                                                                  
country                        W1                                                                                      

 -select: Latest -type: OnPrem -country: W1
*** Platform: C:\bcartifacts.cache\onprem\19.2.32968.33504\platform
*** Localisation: C:\bcartifacts.cache\onprem\19.2.32968.33504\w1
*** BC Management DLL: C:\bcartifacts.cache\onprem\19.2.32968.33504\platform\ServiceTier\program files\Microsoft Dynamics NAV\190\Service\Microsoft.Dynamics.Nav.Management.dll
*** BC App Management DLL: C:\bcartifacts.cache\onprem\19.2.32968.33504\platform\ServiceTier\program files\Microsoft Dynamics NAV\190\Service\Microsoft.Dynamics.Nav.Apps.Management.dll
*** Batch Publish Extensions
  * Artifact Path  : C:\agent\_work\r29\a
*** Resolved Path  : C:\agent\_work\r29\a
*** Get Available Applications from [C:\agent\_work\r29\a].
*** Get [*.app] files recursive from [C:\agent\_work\r29\a]
*** Found [1] files.
*** Extract App detail from [C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_SomeApp_3.2.0.10063_sandbox_19.1_de.app]
*** Installing Applications:

Publisher              Name           Path                                                                             
---------              ----           ----                                                                             
Some Company SomeApp C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_PRI-B...

*** Publish App [C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_SomeApp_3.2.0.10063_sandbox_19.1_de.app]
*** Upload extension [C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_SomeApp_3.2.0.10063_sandbox_19.1_de.app]
*** Get App Info
 * App.ID        = 40682cca-a609-4d4b-851e-fbf7ee4f4b46
 * App.Name      = SomeApp
 * App.Publisher = Some Company
 * App.Version   = 3.2.0.10063
*** Get Extension status
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies(0d30d954-ca0b-49a1-937c-49b7000ffebb)/extensions'
##[command]Invoke-RestMethod -Method Patch -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies(0d30d954-ca0b-49a1-937c-49b7000ffebb)/extensionUpload(0)/content'
##[error]{
  "error": {
    "code": "Application_TooManyRequests",
    "message": "Too many requests reached.  Actual (101).  Maximum (100)."
  }
}
##[warning]RetryHelper encountered task failure, will retry (attempt #: 2 out of 5) after 4000 ms
*** Validate configuration
*** Task Inputs:

name                                                                                                              value
----                                                                                                              -----
usedocker                                                                                                         False
fixed_tag                                                                                                              
interaction                                                                                                       batch
api_endpoint            ...central.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api
apiversion                                                                                                         v1.0
authentication                                                                                                    oauth
azure_tenant_id                                                                    e825b00f-b640-4d87-b5d8-24a5d78cef18
azure_app_client_id                                                                ***
azure_app_client_secret                                                           ***
username                                                                                                               
password                                                                                                               
bccompany                                                                                                              
artifact_path                                                                                      C:\agent\_work\r29\a
artifact_filter                                                                                                   *.app
showdeploymentstatus                                                                                               True

*** For documentation, please visit   : https://www.alops.be/documentation

*** Imported required PS-Functions
*** Parse Task Parameters
*** Creating API Connector with endpoint [https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api]
*** API Version: v1.0
*** Interaction Mode: batch
*** API Authentication: Using Azure OAuth Authentication []
*** Get OAuth token for tenant [e825b00f-b640-4d87-b5d8-24a5d78cef18]
*** Communication: Service-2-Service
##[command]Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/e825b00f-b640-4d87-b5d8-24a5d78cef18/oauth2/token"
*** List Companies
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
*** Selected Company:  []
##[error]Für "GetExtensions" und die folgende Argumenteanzahl kann keine Überladung gefunden werden: "1".
##[warning]RetryHelper encountered task failure, will retry (attempt #: 3 out of 5) after 9000 ms
*** Validate configuration
*** Task Inputs:

name                                                                                                              value
----                                                                                                              -----
usedocker                                                                                                         False
fixed_tag                                                                                                              
interaction                                                                                                       batch
api_endpoint            ...central.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api
apiversion                                                                                                         v1.0
authentication                                                                                                    oauth
azure_tenant_id                                                                    e825b00f-b640-4d87-b5d8-24a5d78cef18
azure_app_client_id                                                                ***
azure_app_client_secret                                                           ***
username                                                                                                               
password                                                                                                               
bccompany                                                                                                              
artifact_path                                                                                      C:\agent\_work\r29\a
artifact_filter                                                                                                   *.app
showdeploymentstatus                                                                                               True

*** For documentation, please visit   : https://www.alops.be/documentation

*** Imported required PS-Functions
*** Parse Task Parameters
*** Creating API Connector with endpoint [https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api]
*** API Version: v1.0
*** Interaction Mode: batch
*** API Authentication: Using Azure OAuth Authentication []
*** Get OAuth token for tenant [e825b00f-b640-4d87-b5d8-24a5d78cef18]
*** Communication: Service-2-Service
##[command]Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/e825b00f-b640-4d87-b5d8-24a5d78cef18/oauth2/token"
*** List Companies
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
*** Selected Company:  []
##[error]Für "GetExtensions" und die folgende Argumenteanzahl kann keine Überladung gefunden werden: "1".
##[warning]RetryHelper encountered task failure, will retry (attempt #: 4 out of 5) after 16000 ms
*** Validate configuration
*** Task Inputs:

name                                                                                                              value
----                                                                                                              -----
usedocker                                                                                                         False
fixed_tag                                                                                                              
interaction                                                                                                       batch
api_endpoint            ...central.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api
apiversion                                                                                                         v1.0
authentication                                                                                                    oauth
azure_tenant_id                                                                    e825b00f-b640-4d87-b5d8-24a5d78cef18
azure_app_client_id                                                                ***
azure_app_client_secret                                                           ***
username                                                                                                               
password                                                                                                               
bccompany                                                                                                              
artifact_path                                                                                      C:\agent\_work\r29\a
artifact_filter                                                                                                   *.app
showdeploymentstatus                                                                                               True

*** For documentation, please visit   : https://www.alops.be/documentation

*** Imported required PS-Functions
*** Parse Task Parameters
*** Creating API Connector with endpoint [https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api]
*** API Version: v1.0
*** Interaction Mode: batch
*** API Authentication: Using Azure OAuth Authentication []
*** Get OAuth token for tenant [e825b00f-b640-4d87-b5d8-24a5d78cef18]
*** Communication: Service-2-Service
##[command]Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/e825b00f-b640-4d87-b5d8-24a5d78cef18/oauth2/token"
*** List Companies
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies'

id                : 0d30d954-ca0b-49a1-937c-49b7000ffebb
systemVersion     : 19.2.32968.33612
name              : CRONUS
displayName       : CRONUS
businessProfileId : 
systemCreatedAt   : 0001-01-01T00:00:00Z
systemCreatedBy   : 00000000-0000-0000-0000-000000000000
systemModifiedAt  : 0001-01-01T00:00:00Z
systemModifiedBy  : 00000000-0000-0000-0000-000000000000

*** Selected Company: CRONUS [0d30d954-ca0b-49a1-937c-49b7000ffebb]
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies(0d30d954-ca0b-49a1-937c-49b7000ffebb)/extensions'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
*** Installed Extensions:

*** Published Extensions (not installed):

*** Import NAV/BC Management DLL's
*** ServiceTier not specified, loading default assemblies.
*** Default Assemblies not found. Load via BC-Artifacts
*** Install/Update BCContainerHelper [https://github.com/microsoft/navcontainerhelper]
*** Check BcContainerHelper
*** Load BcContainerHelper
BcContainerHelper version 3.0.0
BcContainerHelper emits usage statistics telemetry to Microsoft

ModuleType Version    Name                                ExportedCommands                                             
---------- -------    ----                                ----------------                                             
Script     3.0.0      BcContainerHelper                   {Add-FontsToBcContainer, Add-GitToAlProjectFolder, Backup-...

##[command]"Get-BCArtifactUrl" 
Name                           Value                                                                                   
----                           -----                                                                                   
select                         Latest                                                                                  
type                           OnPrem                                                                                  
country                        W1                                                                                      

 -select: Latest -type: OnPrem -country: W1
*** Platform: C:\bcartifacts.cache\onprem\19.2.32968.33504\platform
*** Localisation: C:\bcartifacts.cache\onprem\19.2.32968.33504\w1
*** BC Management DLL: C:\bcartifacts.cache\onprem\19.2.32968.33504\platform\ServiceTier\program files\Microsoft Dynamics NAV\190\Service\Microsoft.Dynamics.Nav.Management.dll
*** BC App Management DLL: C:\bcartifacts.cache\onprem\19.2.32968.33504\platform\ServiceTier\program files\Microsoft Dynamics NAV\190\Service\Microsoft.Dynamics.Nav.Apps.Management.dll
*** Batch Publish Extensions
  * Artifact Path  : C:\agent\_work\r29\a
*** Resolved Path  : C:\agent\_work\r29\a
*** Get Available Applications from [C:\agent\_work\r29\a].
*** Get [*.app] files recursive from [C:\agent\_work\r29\a]
*** Found [1] files.
*** Extract App detail from [C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_SomeApp_3.2.0.10063_sandbox_19.1_de.app]
*** Installing Applications:

Publisher              Name           Path                                                                             
---------              ----           ----                                                                             
Some Company SomeApp C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_PRI-B...

*** Publish App [C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_SomeApp_3.2.0.10063_sandbox_19.1_de.app]
*** Upload extension [C:\agent\_work\r29\a\SomeApp\SomeApp\Some Company_SomeApp_3.2.0.10063_sandbox_19.1_de.app]
*** Get App Info
 * App.ID        = 40682cca-a609-4d4b-851e-fbf7ee4f4b46
 * App.Name      = SomeApp
 * App.Publisher = Some Company
 * App.Version   = 3.2.0.10063
*** Get Extension status
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies(0d30d954-ca0b-49a1-937c-49b7000ffebb)/extensions'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
##[command]Invoke-RestMethod -Method Patch -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies(0d30d954-ca0b-49a1-937c-49b7000ffebb)/extensionUpload(0)/content'
##[error]{
  "error": {
    "code": "Application_TooManyRequests",
    "message": "Too many requests reached.  Actual (101).  Maximum (100)."
  }
}
##[warning]RetryHelper encountered task failure, will retry (attempt #: 5 out of 5) after 25000 ms
*** Validate configuration
*** Task Inputs:

name                                                                                                              value
----                                                                                                              -----
usedocker                                                                                                         False
fixed_tag                                                                                                              
interaction                                                                                                       batch
api_endpoint            ...central.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api
apiversion                                                                                                         v1.0
authentication                                                                                                    oauth
azure_tenant_id                                                                    e825b00f-b640-4d87-b5d8-24a5d78cef18
azure_app_client_id                                                                ***
azure_app_client_secret                                                           ***
username                                                                                                               
password                                                                                                               
bccompany                                                                                                              
artifact_path                                                                                      C:\agent\_work\r29\a
artifact_filter                                                                                                   *.app
showdeploymentstatus                                                                                               True

*** For documentation, please visit   : https://www.alops.be/documentation

*** Imported required PS-Functions
*** Parse Task Parameters
*** Creating API Connector with endpoint [https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api]
*** API Version: v1.0
*** Interaction Mode: batch
*** API Authentication: Using Azure OAuth Authentication []
*** Get OAuth token for tenant [e825b00f-b640-4d87-b5d8-24a5d78cef18]
*** Communication: Service-2-Service
##[command]Invoke-RestMethod -Method Post -Uri "https://login.microsoftonline.com/e825b00f-b640-4d87-b5d8-24a5d78cef18/oauth2/token"
*** List Companies
##[command]Invoke-RestMethod -Method Get -Uri 'https://api.businesscentral.dynamics.com/v2.0/e825b00f-b640-4d87-b5d8-24a5d78cef18/Sandbox/api/microsoft/automation/v1.0/companies'
##[error]Exception in BCConnector.GetAPIData: Der Remoteserver hat einen Fehler zurückgegeben: (429).
*** Selected Company:  []
##[error]Für "GetExtensions" und die folgende Argumenteanzahl kann keine Überladung gefunden werden: "1".
##[section]Finishing: ALOps Extension API

[DanielGoehler]

We mitigated this issue with "Number of retries if task failed" set to 5. These error happens at one of our tenants almost every day. Different Apps, Different Environment sometimes Sandbox 1, sometimes Sandbox 2, sometime Production. No clear target.

[waldo1001]

We're running against the limitations of Business Central here - there are simply a limited amount of connections (ALOps is not doing 100 requests at a time.. so I expect other "software" to have connections as well at that same time - it is a bit of a mystery how MS counts these connections): https://docs.microsoft.com/en-us/dynamics365/business-central/dev-itpro/api-reference/v1.0/dynamics-current-limits

What we can do:

• provide a parameter to set up a number of retries
• provide a parameter to set up a "delay" for the next retry

What do you think?

[DanielGoehler]

Hi @waldo1001, thanks for you thoughts. You're right, ALOps are not the source of that simultaneous 100 requests. ALOps will cause only one simultaneous request. This customer have a lot integration LogicApps, we don't know details about. E.g. Webshop, which constanctly communicating with the Business Central backend would have this behavior. I will ask the customer if he scale back on this a little bit or if there is better time frame for importing the updates.

We can already set up a number of retries. Set up a "delay" for the next retry would be nice in the long run.

[AdminHodor]

v1.449: ALOpsExtensionAPI: More Resilient on Deployment, delay and max retries can be configured with [checksecondsdelay] and [maxtries]

[DanielGoehler]

Thank you! The result speaks for itself. I set Check Delay (Sec) to 60 and the errors are gone (Same cloud customer, same app, nothing changed, just deployed a new version today):

[waldo1001]

Nice! Difficult for us to see as we didn't have any test-case .. but glad it helps!