Is it possible to publish an AppSource app to a Sandbox from Azure DevOps?

[CarloAxians]

We are developing an Appsource app and want to publish this into an online Sandbox environment.

When we do this with the ALOps Extension API we get the error message: An App with same app Id 'xxx' is Already published with Global scope with name 'yyy'. Please use the Extension market place to install the extension or use a new app id for the current extension.

From VSCode is it possible to publish to the Dev scope and this is what we would to do from Azure DevOps.

Is there any solution for this or is this simply not possible?

[waldo1001]

You're probably trying to push to a snadbox that you also use with VSCode or where you have pushed with PowerShell.

You can't and shouldn't mix. If you publish from DevOps, you probably want to publish to Prod or a Test env, not a Dev env...

Can you try a new sandbox (empty) and report back?

[CarloAxians]

I've created a new sandbox, but problem still occurs. I did two tests, one with a PTE app and this succeeds and one with our AppSource app and this fails.

Do I need to do something special for an appsource app?

[waldo1001]

Should be no difference. An app is an app.

Was the AppSource app pre-installed? If so - then you can't just overwrite it with a pipeline. And that wouldn't make much sense either, I think (at least I don't see it).

Let's do something else: you can try this process manually as well. Just use the upload page in the client, and upload the app which you expect the pipeline would do.

This will most probably give the same error. And if so - there's nothing ALOps can do, I'm afraid. The API doesn't give us any flexibility for that matter.

[CarloAxians]

Manually I'm getting the same error.

I also tried BcContainerHelper function Publish-BcContainerApp: Publish-BcContainerApp -bcAuthContext $authcontext -environment $environment -appFile $apps

And this works perfectly, I can even upgrade a Globally installed version to a new Dev version. Would it be possible to provide a wrapper for this command in ALOps?

[waldo1001]

I'm confused.

Are you still mixing VSCode deployment and DevOps/API deployment? How do you get the app in there in Dev-scope?

[CarloAxians]

I'm not using VSCode for this, I want this to work from DevOps/API. I found some info on Freddy's blog: https://freddysblog.com/2021/01/27/publishing-apps-to-a-business-central-online-environment/

When I run the powershell script from this blog I'm getting my application installed in the Dev scope: Publish-BcContainerApp -bcAuthContext $authcontext -environment $environment -appFile $apps Results in: Publishing Axians Business Solutions_I-Suite_5.3.20210521.1410_APP.app to https: //api.businesscentral.dynamics.com/v2.0/sandbox/dev/apps?SchemaUpdateMode=synchronize

And the app is installed in Dev scope:

[waldo1001]

Ok, but why do you want it in the Dev scope? Publishing from DevOps is never intended to be in dev scope.. 🤷‍♂️

[waldo1001]

If you don't use it from VSCode, and you are using bccontainerhelper for anything else outside docker .. you could use it to publish in global scope by using Publish-PerTenantExtensionApps? And that way, you can combine both?

[CarloAxians]

Ok, but why do you want it in the Dev scope? Publishing from DevOps is never intended to be in dev scope.. 🤷‍♂️

I want to be able to test the application prior to submitting it into AppSource.

[CarloAxians]

If you don't use it from VSCode, and you are using bccontainerhelper for anything else outside docker .. you could use it to publish in global scope by using Publish-PerTenantExtensionApps? And that way, you can combine both?

This can only be used for Per Tenant Apps and this is an AppSource app.

[waldo1001]

What is it in your AppSource app that makes that function not work? (sorry, I don't have any example here to try myself ..). Can you show the output if you would?

Can you confirm you did try this:

1. Create new sandbox. Completely empty
2. ONLY use ALOps (not a single line of PowerShell of your own) to publish the apps to it

[CarloAxians]

I have created a new sandbox and run the ALOps task. Here is the logging: tasklog_6.log I have removed the secret id's from the file 😁

[waldo1001]

I'm baffled. This does not make any sense.

So .. a brand new sandbox has your extension in dev scope pre-installed.. ?

Can you send yaml and the entire log?

[CarloAxians]

I'm baffled. This does not make any sense.

Me too😁

So .. a brand new sandbox has your extension in dev scope pre-installed.. ?

No, only microsoft apps are installed

Can you send yaml and the entire log?

This is the entire log. This is part of a release pipeline, the only thing that runs before this is downloading the artifact. I have converted the task to yaml:

  displayName: 'ALOps Extension API'
  inputs:
    interaction: publish
    api_endpoint: 'https://api.businesscentral.dynamics.com/v2.0/$(azure_tenant_id)/isuite/api'
    azure_tenant_id: '<removed>'
    azure_app_client_id: '<removed>'
    azure_app_client_secret: '<removed>'
    artifact_filter: '*I-Suite_*_APP.app'

[waldo1001]

It errors on the fact that the extension already exists in dev scope. you tested it manually - and it was the same error.

The test with BcContainerHelper isn't really a good comparison, because that's intentionally putting it in dev scope.

so .. question is still .. how-oh-how is that extension being published in dev scope before ALOps is pushing it in normal scope.. .

I'm going to set this sink in .. we might have to have a call about this next week...

[Arthurvdv]

Should be no difference. An app is an app.

Isn't it that by design a AppSource App can only be deployed outside the marketplace to a Sandbox environment as a dev scope? The Business Central tenants in SaaS are multitenant (https://docs.microsoft.com/en-us/dynamics365/business-central/dev-itpro/deployment/multitenant-deployment-architecture), where application data is shared between tenants, where Apps for AppSource are published Global. A PTE is not shared between tenants off course.

I'm pretty sure uploading the AppSource App from the Sandbox Extension Management page is also going to throw an error. The ALOps API doesn't have a option to publish as Dev, so without that option you're not going to be able to publish to a Sandbox?

[waldo1001]

I am really going to need some yamls, I believe ;-)

[CarloAxians]

I’m gonna create an example next week. But I think it’s simply not possible to install an apppsource app with the API. But the error message we’re getting is rather confusing. 🥴

[CarloAxians]

I have created a YAML:

variables:
- group: 'Build variables'
- group: 'I-Suite Build variables'
- group: 'SaaS variables'

steps:
  - checkout: self
    clean: true

  - task: ALOpsAppCompiler@2
    displayName: 'Compile i-Suite Extension'
    inputs:
      artifacttype: Sandbox
      artifactversion: '$(Current-Sandbox-Version)'
      artifactcountry: nl
      alsourcepath: 'base/'
      appversiontemplate: '?.?.[yyyyMMdd].*'
      appfilenametemplate: '%APP_PUBLISHER%_%APP_NAME%_%APP_VERSION%_APP.app'
      alcodeanalyzer: 'UICop,CodeCop,AppSourceCop'
      failonwarnings: false
      showmycode: 'Disable'
      publishartifact: false

  - task: ALOpsAppSign@1
    displayName: 'Sign i-Suite Extension'
    inputs:
      usedocker: false
      artifact_path: $(ALOPS_COMPILE_ARTIFACT)
      pfx_path: $(pfx-path)
      pfx_password: $(pfx-password)
      timestamp_uri: $(timestamp_uri)
      publish_artifact: true

  - task: ALOpsExtensionAPI@1
    displayName: 'ALOps Extension API'
    inputs:
      interaction: publish
      api_endpoint: 'https://api.businesscentral.dynamics.com/v2.0/$(azure_tenant_id)/isuite/api'
      azure_tenant_id: $(azure_tenant_id)
      azure_app_client_id: $(azure_app_client_id)
      azure_app_client_secret: $(azure_app_client_secret)
      artifact_filter: '*I-Suite*_APP.app'

When I publish our AppSource app I get the error:

"An App with same app Id '<AppId>' is Already published with Global scope with name 'I-Suite'. Please use the Extension market place to install the extension or use a new app id for the current extension.

I have changed the app id and then I get the error: App publish failed. Check the 'Extension Deployment Status' page in the Business Central client.

When I check the status it shows: Package validation failed due to the following error(s): (1,18): error PTE0001: XmlPort 'AXI SF Users' has an ID of [11164001]. It must be between within the range '[50000..99999]'.

So as @Arthurvdv mentioned it's probably not allowed to publish AppSource apps this way. The only way is thru the dev-endpoint with the BCContainerHelper command Publish-BcContainerApp.

@waldo1001 Would it possible to add this command to ALOps? Or should I create a PowerShell script for this myself?

[waldo1001]

Hi @CarloAxians,

You can obvisously just use the BcContainerHelper in combination with ALOps .. . Like: install all AppSource apps with BCCH, and the rest with ALOps. Then you don't have to invest in any script.

Meantime, we'll investigate and we might invest in our own publish command as well.. .

[MortenRa]

@waldo1001 is this still on the roadmap, we might look into this now MS has Introduced BC partner Sandbox

[waldo1001]

It is .. but it's difficult to give an ETA at this point.. .

[MortenRa]

@waldo1001 any progress on this one, I would like to avoid using BcContainerHelper but all with ALops

[dsaveyn]

We are running into the same issue when we try to publish on a Sandbox.

As a workaround, we use BcContainerHelper to deploy the apps and this works fine for the first time. However, if the try to publish an app for the second time, we run into an internal server error. Manually uninstalling/unpublishing is the only solution... For so far I can figure out, this uninstall/unpublish on a Sandbox is not (yet) possible via BcContainerHelper.

@CarloAxians have you managed to figure this out?

[MortenRa]

@dsaveyn are you using build pipeline or a release pipeline for this task and would you mind sharing your setup as I´m baffling with this at the moment.

According to Freddy´s blog post this was stated:

If you run the exact same function again, it will fail. You cannot publish the same app twice. If you recompile, it will succeed. You do not need to update the version number, it just needs to be a different package id (a new compilation)

maybe that is the cause for the second time

[CarloAxians]

@dsaveyn, we are using the Publish-BcContainerApp command from a release pipeline. The versionnumber of the app file is always changed by the build pipeline.

One problem that we got was an error like Cannot publish an extension with packageId: xyz, because the same package ID already exists This issue is mentioned here: https://github.com/microsoft/navcontainerhelper/issues/2326 So we have added the Replace-DependenciesInAppFile command to resolve this, this results in a new package id.

[fvet]

@MortenRa

We've currently a very simple PS script that is ran manually and works like a charm:

• Clone all code / repo's (AppSource apps)
• Compile all apps
• Publish all apps via Publish-BcContainerApp to an online Sandbox

Moving this to DevOps is a very cumbersome process :(

Here's the current setup we've in mind

• Create a CI pipeline that runs on a PR (feature_branch) and generates an app file when the PR is completed (merged into dev/master branch)
• Above step is multiplied by the number of repo's / apps, we might end up with X updated apps throughout the day.
• With a release pipeline, we want to deploy the apps to a test environment (= online sandbox / CDX, possibly future partner sandbox) at a specific time(s), providing the testers with an update of all latest app versions (= to be AppSource apps). We've chosen not to auto deploy each individual app immediately after the CI pipeline succeeds, since this could interrupt the current user sessions / ongoing tests, and thus delayed the deploy step and bundle app deployment / batch publish the 'latest' apps (some are newly build, others are just the latest version that was build x days ago).
• Reading the above comments, I think it should work if we deploy individual apps immediately after the CI pipeline (since then they get a new package id / compilation hash), but this could result in various other issues (testers are blocked / receive messages, CI pipelines from dependent apps might follow afterwards adding extra delay, dependent apps might be uninstalled, not re-installed, ...)

If I summarize the solution of Carlo, we should rebuild / recompile (through Replace-DependenciesInAppFile) all apps before releasing? ... which makes the app artifacts stored on the CI pipelines rather useless?

@waldo1001 Any guidance from ALOps on this topic?

[fvet]

Would it be an option to first Get the installed apps via the ALOps Extension API, and based on the retrieved version (if we get the full app version, recent tests only listed '20.0' as version in the ALOps Extension API output) decide to only publish (via Publish-BcContainerApp) apps having a more recent version instead?

[MortenRa]

@CarloAxians, @dsaveyn anyone wanna share some insights in regards to the setup in your release pipeline for Publish-BcContainerApp I struggling and might need a push in the correct direction. Maybe a sample file and info on which BcAuthContext you are using.

[fvet]

@MortenRa

What we did was adding a task in the Release Pipeline

This task group calls 2 PS scripts

$publishedExtensions = ."$(System.DefaultWorkingDirectory)/_Scripts CI/Scripts/Development/Other/Get Extensions From Sandbox.ps1" `
    -TenantId '$(TenantId)' `
    -Environment '$(Environment)' `
    -ClientId '$(ClientId)' `
    -ClientSecret '$(ClientSecret)'

."$(System.DefaultWorkingDirectory)/_Scripts CI/Scripts/Development/Other/Publish Apps To Sandbox.ps1" `
    -AppFolder "$(System.ArtifactsDirectory)" `
    -Environment '$(Environment)' `
    -RefreshToken '$(RefreshToken)' `
    -PublishedApps $publishedExtensions

The first script checks for the installed app versions.

[CmdletBinding()]
param (

    [Parameter(Mandatory = $true)]
    [string]$TenantId,
    [Parameter(Mandatory = $true)]
    [string]$Environment,
    [Parameter(Mandatory = $true)]
    [string]$ClientId,
    [Parameter(Mandatory = $true)]
    [string]$ClientSecret,
    [Parameter(Mandatory = $false)]
    [string]$Publisher    
)

if (!(Get-Module -ListAvailable -Name MSAL.PS)) { Install-Module MSAL.PS -Force } 
Import-Module MSAL.PS

$baseUrl = "https://api.businesscentral.dynamics.com/v2.0/$environment/api/microsoft/automation/v1.0"
$scopes = "https://api.businesscentral.dynamics.com/.default"

# Get access token 
$token = Get-MsalToken `
    -ClientId $ClientId `
    -TenantId $tenantId `
    -Scopes $scopes `
    -ClientSecret (ConvertTo-SecureString -String $ClientSecret -AsPlainText -Force)

# Get first company
$companies = Invoke-RestMethod `
    -Method Get `
    -Uri $("$baseurl/companies") `
    -Headers @{Authorization = 'Bearer ' + $token.AccessToken }       

$companyId = $companies.value[0].id

# Get extensions
$extensions = Invoke-RestMethod `
    -Method Get `
    -Uri $("$baseurl/companies($companyId)/extensions") `
    -Headers @{Authorization = 'Bearer ' + $token.AccessToken }   

if (-not([string]::IsNullOrEmpty($Publisher))) {
    $extensions = $extensions.value | Where-Object Publisher -eq $Publisher
} 
else {
    $extensions = $extensions.value
}

$extensions

The seconds script will only publish those new apps that have a higher version.

[CmdletBinding()]
param (
    [Parameter(Mandatory = $true)]
    [string]$AppFolder,    
    [Parameter(Mandatory = $true)]
    [string]$Environment,
    [Parameter(Mandatory = $true)]
    [string]$RefreshToken,
    [Parameter(Mandatory = $false)]
    [array]$PublishedApps,
    [Parameter(Mandatory = $false)]
    [Switch]$PublishTestApps = $false    
)

if (!(Get-Module -ListAvailable -Name 'BcContainerHelper')) { Install-Module BcContainerHelper -Force } 
Import-Module BCContainerHelper

Write-Host "Sorting Apps"
$AppsToInstall = Sort-AppFilesByDependencies -appFiles ((Get-ChildItem -Path $AppFolder -Filter '*.app' -Recurse).FullName)

Write-Host "Retrieving Authentication Context"
$AuthContext = New-BcAuthContext -refreshToken $RefreshToken

foreach ($App in $AppsToInstall) {
    $AppFileNameSplitted = [System.IO.Path]::GetFileName($App).Split("_")
    $Publisher = $AppFileNameSplitted[0]
    $Name = $AppFileNameSplitted[1]
    $Version = $AppFileNameSplitted[2]

    if ((-not $PublishTestApps) -and ($Name -like "*-test*")) { continue }

    $PublishedApp = $PublishedApps | Where-Object { ($_.publisher -eq $Publisher) -and ($_.displayName -eq $Name) }
    if ($null -ne $PublishedApp) {
        $PublishedAppVersion = "$($PublishedApp.versionMajor).$($PublishedApp.versionMinor).$($PublishedApp.versionBuild).$($PublishedApp.versionRevision)"
        if ($Version -eq $PublishedAppVersion) { 
            Write-Host "App $Name with version $Version for publisher $Publisher already installed" -ForegroundColor Yellow
            continue 
        }
    }

    Write-Host "Installing $([System.IO.Path]::GetFileName($App))"
    Publish-BcContainerApp `
        -bcAuthContext $AuthContext `
        -environment $Environment `
        -appFile $App `
        -syncMode ForceSync 
}

Still testing, but it looks promising.

@waldo1001 Would be nice if ALOps could provide similar functionality out of the box ;)

[fvet]

@dsaveyn should be able to guide you to the blogs we've used for the authentication part (clientid/clientsecret).

[waldo1001]

Just talked to development. We always questioned the useability of this (still do, tbh - your usecases would be really interesting ;-)). But since so many people really want it, we'll push it up in our prio-list.

Thanks for all the feedback and pointers, that's definitely going to be useful!

[fvet]

@waldo1001 The usecase is partly described above, below some more details.

Our OTAP strategy / infra setup is fully based on non-docker based / online saas environments.

• DEV : developer test on own online sandbox (personal CDX tenant) - publish via VS Code
• TEST : functional test on central online sandbox (CDX tenant / or partner sandbox) - publish via DevOps / release pipelines
• ACCEPT : functional test on central online sandbox (CDX tenant / or partner sandbox) - publish via DevOps / release pipelines

[waldo1001]

Fair enough - we seem to be spoiled with Docker ;-).

We'll look into it!

[MortenRa]

@waldo1001 We really benefit at lot by using alops and adding this to the product would make it even better.

Our Usecase and as Our Strategy is SAAS and not far from Fredéric´setup.

• DEV: Developer uses partner sandbox for development via VS Code ,but we have some library apps/Standard Company Setup/Demo data to be publish through DevOps Release Pipeline. They can use docker for previous versions if needed but on request.
• UAT: Partner sandbox for our product managers and team to test feat/bug - Publish through DevOps Release Pipeline
• Release Test: Final Test before release to Appsource Publish through DevOps Release Pipeline to partner sandbox (hopefully this can in the future also be used for publishing to appsource API

[CarloAxians]

@waldo1001 Our situation is almost the same. We have a AppSource app with our standard solution and almost every customer has 1 or more PTE apps with some customizations. We develop against a docker image or a developer sandbox in CDX. Then we commit our changes and publish this in a Sandbox in the customer's tenant. In this tenant are both our AppSource app and the PTE apps published in Dev scope, so the customer / consultant can validate the solution.

Although this is working for most of the time, there are sometimes some issues. One is that after an update of BC is installed, Minor or Major, all Dev scope installed apps are removed and we need to install them again from DevOps. An other issue is publishing sometimes fails with a time out and we need to start the publish again.

I can understand you are struggling with the question whether this should be integrated in ALOps. The installing of apps in Dev scope is meant to be for development only and not for testing, but currently this is the only way we can let customers/consultant validate our solution with their own data.

I hope that Microsoft someday will provide a solution to submit an app in AppSource as a beta/test version and that this can be done automatically by an api, so this can be done from DevOps.

[MortenRa]

@waldo1001 I know you can not give an ETA, but maybe a best guess when you think you have something ready.

Just to know if I need to invest some time in some scripts or wait.

[waldo1001]

It's right up there on our prio-list (nr 1). That's all I can say. We have a few bits in place already - but we'll have to thoroughly test it .. (all authentications, OnPrem, SaaS, .. ).

[waldo1001]

We hit some roadblocks, but still prio 1. hard to give a release date at this point, but we aim for end of next week ..

[AdminHodor]

Dear @CarloAxians,

Please check our latest release v1.451 We added the Dev-Port-Deploy into the "ALOpsExtensionAPI" step, all major parameters are the same as for AutomationAPI deploy. When you specify the additional parameter [dev_endpoint] the task will automatically go into Dev-Port-Deploy mode.

Kind regards,

[MortenRa]

Currently testing this with s2s and devport and seems to be working fine if only one sandbox on the same tenant, but if you try to publish through a release pipeline to a second or third sandbox on same tenant with the same package following error occurs.

2022-08-05T10:17:55.4868474Z *** Using Dev-Port deploy instead of API
2022-08-05T10:17:55.5063583Z Publishing to https://api.businesscentral.dynamics.com/v2.0/f7b6995f-da03-4722-93bd-cafb3db452b2/afrodite/dev/apps?SchemaUpdateMode=synchronize
2022-08-05T10:17:55.9821515Z ##[error]Status Code InternalServerError : Internal Server Error
Object reference not set to an instance of an object.

Not sure if this relates to the problem described earlier as the log or error does not indicate much info.

One problem that we got was an error like Cannot publish an extension with packageId: xyz, because the same package ID already exists
So we have added the Replace-DependenciesInAppFile command to resolve this, this results in a new package id.

If a Create a new release I´m able to publish to the second sandbox (then same error on the first sandbox occurs)

Furthere more I can not see an option for the schemaUpdateMode that would be nice to also control

[fvet]

@dsaveyn This is the same we're experiencing as well.

[MortenRa]

@waldo1001 Maybe add the parameter -replacePackageId to be able to deploy to 2 sandbox on same tenant.

[waldo1001]

Will close as there is another issue for the multiple-environments.