Working with AppSourceCopPackages

The appSourceCopPackages-folder is a mix of dependency Apps and the previous version of the App. The dependency App are again a mix of Platform Apps from Microsoft and dependencies to other Apps

When creating a pipeline the goal is to create a Docker environment, install previous, install current (test upgrade) and runt test suite. I'm struggeling to grab the correct files from this folder, I need;

Only the dependency-Apps (without the Microsoft Apps)
The previous version

If the Microsoft App weren't there I could simply do something like this;

- task: ALOpsAppPublish@1
  displayName: 'Publish Dependencies'
  inputs:
    usedocker: true
    nav_artifact_app_filter: '*.app'
    batch_publish_folder: '$(System.DefaultWorkingDirectory)\App\.appSourceCopPackages\'

But this fail, because of the Microsoft Apps in this folder.

Describe the solution you'd like Would het be possible to exclude the Microsoft Base, Application and System from this folder to the repo? It feels committing these files to a repository as overhead and I'm having challenges when configuring the pipeline. If I exclude the Microsoft Apps in a .gitigone, then the Compiler (v2) thrown an error.

Is it possible to extend the Compiler (v2) to use the Microsoft Apps from the artifacts and something like detect the baselinePackageCachePath from the AppSourceCop.json-file and place these files there if they aren't there? ...and if possible remove these files back again after compiling?

AppSourceCop.json

{
    "mandatoryPrefix": "BSF",
    "supportedCountries": [
        "BE",
        "DE",
        "FR"
    ],
    "version": "0.8.2.0",
    "baselinePackageCachePath": "./.appSourceCopPackages"
}

Describe alternatives you've considered Maybe I'm making it more difficult then necessary and just let the ALOpsAppValidation solve this? Problem is that the ALOpsAppValidation doens't have any parameters for running the test's and we can't test again W1 as a country.

Why dont you split the packages in separate folders, so you can use a decent foldername in the batch_publish?

The CodeCop and/or AL-Compiler forces me to do this :-|

If I leave out one of de dependency Apps, for example Microsoft Apps, the project won't compile anymore;

\App\Source_Base\SalesHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Header' has changed from 'MISSING' to 'Sales Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.

Also the ALOps Compiler(v2) will fail because of the same reason;

2021-06-10T12:43:40.8822839Z ##[error]\App\Source\Contact\DevelopmentJobContact.Table.al(100,19): error AS0004: Field 'Contact Type' has changed type from '__MissingTypeSymbol__' to 'Enum "Contact Type"'. Type changes are not allowed.
2021-06-10T12:43:40.8832226Z ##[error]\App\Source\_Base\PurchaseHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purchase Header' has changed from 'MISSING' to 'Purchase Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8834567Z ##[error]\App\Source\_Base\PurchaseHeaderArchive.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purchase Header Archive' has changed from 'MISSING' to 'Purchase Header Archive'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8838552Z ##[error]\App\Source\_Base\PurchaseLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purchase Line' has changed from 'MISSING' to 'Purchase Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8852072Z ##[error]\App\Source\_Base\PurchaseLineArchive.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purchase Line Archive' has changed from 'MISSING' to 'Purchase Line Archive'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8865461Z ##[error]\App\Source\_Base\PurchCrMemoHdr.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. Cr. Memo Hdr.' has changed from 'MISSING' to 'Purch. Cr. Memo Hdr.'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8878991Z ##[error]\App\Source\_Base\PurchCrMemoLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. Cr. Memo Line' has changed from 'MISSING' to 'Purch. Cr. Memo Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8892691Z ##[error]\App\Source\_Base\PurchInvHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. Inv. Header' has changed from 'MISSING' to 'Purch. Inv. Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8906316Z ##[error]\App\Source\_Base\PurchInvLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. Inv. Line' has changed from 'MISSING' to 'Purch. Inv. Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8919880Z ##[error]\App\Source\_Base\PurchPaySetup.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. & Pay. Setup' has changed from 'MISSING' to 'Purchases & Payables Setup'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8933476Z ##[error]\App\Source\_Base\PurchRcptHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. Rcpt. Header' has changed from 'MISSING' to 'Purch. Rcpt. Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8946862Z ##[error]\App\Source\_Base\PurchRcptLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Purch. Rcpt. Line' has changed from 'MISSING' to 'Purch. Rcpt. Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8960334Z ##[error]\App\Source\_Base\ReturnShipmentHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Return Shipment Header' has changed from 'MISSING' to 'Return Shipment Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8973953Z ##[error]\App\Source\_Base\ReturnShipmentLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Return Shipment Line' has changed from 'MISSING' to 'Return Shipment Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.8989395Z ##[error]\App\Source\_Base\SalesCrMemoLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Cr. Memo Line' has changed from 'MISSING' to 'Sales Cr.Memo Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.9003603Z ##[error]\App\Source\_Base\SalesHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Header' has changed from 'MISSING' to 'Sales Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.9020753Z ##[error]\App\Source\_Base\SalesInvoiceHeader.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Invoice Header' has changed from 'MISSING' to 'Sales Invoice Header'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.9035676Z ##[error]\App\Source\_Base\SalesInvoiceLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Invoice Line' has changed from 'MISSING' to 'Sales Invoice Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.9048711Z ##[error]\App\Source\_Base\SalesLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Line' has changed from 'MISSING' to 'Sales Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.9061982Z ##[error]\App\Source\_Base\SalesShipmentLine.TableExt.al(1,24): error AS0068: The target table of table extension 'BSF RE Sales Shipment Line' has changed from 'MISSING' to 'Sales Shipment Line'. Changing the target of a table extension that has been published is not allowed, because this might break the upgrade of existing installations.
2021-06-10T12:43:40.9075368Z ##[error]\App\AppSourceCop.json(1,1): error AS0091: One or more dependencies of the previous version of the extension could not be found in the baseline package cache folder. Details: error AL1022: A package with publisher 'Microsoft', name 'System', and a version compatible with '17.0.0.0' could not be found in the package cache folder '\App\./.appSourceCopPackages'
2021-06-10T12:43:40.9426312Z ##[warning]error AL1022: A package with publisher 'Microsoft', name 'Application', and a version compatible with '17.0.0.0' could not be found in the package cache folder 'D:\azure-vsts-agent-build\_work\99\s\App\./.appSourceCopPackages'

I've realized today that in the appSourceCopPackages-folder it could be sufficient to have a App file with the symbols only. As you mentioned it could be an idea to have the installable .app files in separate folder.

It feels a bit unhandy when a depedency-app has a new version you have to update the file on two places. With more-and-more depedency-apps it feels the coping of files never ends :-|

Can I pitch an idea? The compiler (v2) already is capable of finding the depedency-apps in the project. Could the compiler detect the baselinePackageCachePath setting from the AppSourceCop.json file and just-in-time copy the needed App files to this directory?

Would this be a solution for you?

- task: ALOpsAppPublish@1
  displayName: 'Publish Dependencies'
  inputs:
    usedocker: true
    nav_artifact_app_filter: 'Vanroey*.app'
    batch_publish_folder: '$(System.DefaultWorkingDirectory)\App\.appSourceCopPackages\'

So, basically providing a filter? We would need to provide that ability first though..: #204

A filter could indeed create a working solution. To have more generic approach, it would be better to Exclude the Apps from Microsoft (not sure if the filter-param supports excluding?)

- task: ALOpsAppPublish@1
  displayName: 'Publish Dependencies'
  inputs:
    usedocker: true
    nav_artifact_app_filter: '<>Microsoft*.app'
    batch_publish_folder: '$(System.DefaultWorkingDirectory)\App\.appSourceCopPackages\'

For now I've created a work-around to remove the Microsoft Apps just before the Publish-step

- task: PowerShell@2
  displayName: 'Remove Microsoft Apps from baselinePackageCachePath'
  condition: succeeded()
  inputs:
    targetType: 'inline'
    script: |
      $AppSourceCopDir = "$(System.DefaultWorkingDirectory)\App\"
      $AppSourceCopFile = Join-Path $AppSourceCopDir "AppSourceCop.json"
      $obj = Get-Content -Raw -Path $AppSourceCopFile | ConvertFrom-Json

      $baselinePackageCachePath = $obj.baselinePackageCachePath

      $baselinePackageCachePathLocation = Join-Path $AppSourceCopDir -ChildPath $baselinePackageCachePath

      $files = Get-ChildItem $baselinePackageCachePathLocation | Where-Object {$_.FullName -match "Microsoft_"}

      foreach ($f in $files) {
            Write-Host "##[command]Remove-Item $f.FullName"
            Remove-Item $f.FullName
      }

I was hoping for some more magic from ALOps to completely handle the Microsoft dependencies a bit like the CompilerV2 dynamically grabs the dependencies app files from the artifacts.

The use case would be that the Microsoft Apps in the .appSourceCopPackages are included in the .gitignore, so only lives on the local disk of the developer's machine (a bit the same as the .alpackages-folder). When compiling the App the necessary dependencies app are (temporary?) copied into the baselinePackageCachePath folder from the artifacts if they are not already there.

This logic then could be applied with the ALOpsAppCompiler and ALOpsAppValidation steps.

@Arthurvdv , just a small question - did you try to put the MS-apps in a separate subfolder? So it would skip those files while publishing?

"baselinePackageCachePath": "./.appSourceCopPackages"

@waldo1001 Moving the MS-apps to a subfolder could be a solution. To be honest I'm hoping that ALOps could handle the dependency to the base App(s). This saves storing these files in source control (and manual update/committing these files).

A quick try to move the MS-apps to a own subfolder works works with the ALOpsAppCompiler@2, but fails on the ALOpsAppPublish@1 step. Default the ALOpsAppPublish-step works recursive (I think?) ** Get [.app] files recursive from [c:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82]

### Install Previous version for Breaking Changes Check
- task: ALOpsAppPublish@1
  displayName: 'Publish Previous version from appSourceCopPackages'
  condition: succeeded()
  inputs:
    usedocker: true
    batch_publish_folder: '$(System.DefaultWorkingDirectory)\App\.appSourceCopPackages\'
    nav_artifact_app_filter: '*.app'
    skip_verification: true

2021-08-02T07:28:36.4480881Z ##[section]Starting: Publish Previous version from appSourceCopPackages
2021-08-02T07:28:36.4575147Z ==============================================================================
2021-08-02T07:28:36.4575371Z Task         : ALOps App Publish
2021-08-02T07:28:36.4575533Z Description  : Publish AL Extension to Business Central
2021-08-02T07:28:36.4575684Z Version      : 1.444.2571
2021-08-02T07:28:36.4575813Z Author       : Hodor
2021-08-02T07:28:36.4575981Z Help         : Publish Business Central extension to service tier.
2021-08-02T07:28:36.4576181Z ==============================================================================
2021-08-02T07:28:37.3187182Z *** Configure AMSI for session: disable (in memory)
2021-08-02T07:28:37.3314211Z 0
2021-08-02T07:28:37.8124934Z *** Validate configuration
2021-08-02T07:28:38.0251048Z *** Task Inputs:
2021-08-02T07:28:38.0284222Z 
2021-08-02T07:28:38.0355276Z name                                                                                                              value
2021-08-02T07:28:38.0356740Z ----                                                                                                              -----
2021-08-02T07:28:38.0357267Z usedocker                                                                                                          True
2021-08-02T07:28:38.0358149Z fixed_tag                                                                                                              
2021-08-02T07:28:38.0359369Z installpublishedapps                                                                                               True
2021-08-02T07:28:38.0360905Z installaltesttool                                                                                                 False
2021-08-02T07:28:38.0361318Z install_al_app_names                                                                             Tests-TestLibraries...
2021-08-02T07:28:38.0362643Z strictappnames                                                                                                    False
2021-08-02T07:28:38.0363011Z nav_serverinstance                                                                                                BC140
2021-08-02T07:28:38.0363893Z artifact_path                                                                                                          
2021-08-02T07:28:38.0365055Z nav_artifact_app_filter                                                                                           *.app
2021-08-02T07:28:38.0365499Z skip_verification                                                                                                  True
2021-08-02T07:28:38.0367000Z forceinstall                                                                                                      False
2021-08-02T07:28:38.0367777Z forcesync                                                                                                         False
2021-08-02T07:28:38.0369210Z publish_scope                                                                                                    Global
2021-08-02T07:28:38.0369730Z tenant                                                                                                          default
2021-08-02T07:28:38.0371271Z batch_publish_folder                                    C:\azure-vsts-agent-build\_work\78\s\App\.appSourceCopPackages\
2021-08-02T07:28:38.0372487Z publisherazureactivedirectorytenantid                                                                                  
2021-08-02T07:28:38.0372674Z 
2021-08-02T07:28:38.0388192Z 
2021-08-02T07:28:38.0393203Z 
2021-08-02T07:28:38.0512862Z *** For documentation, please visit   : https://www.alops.be/documentation
2021-08-02T07:28:38.0514457Z 
2021-08-02T07:28:38.3581051Z *** ALOps License:
2021-08-02T07:28:38.3594494Z   * Licensed To: Van Roey (Collection License)
2021-08-02T07:28:38.3596935Z 
2021-08-02T07:28:38.3667338Z *** Importing required PS-Functions
2021-08-02T07:28:38.7355683Z *** Initiate Docker Session
2021-08-02T07:28:39.0734020Z *** Set Docker Container ErrorActionPreference = Stop
2021-08-02T07:28:39.2674978Z *** Start Batch App-Publish.
2021-08-02T07:28:39.3307936Z *** Copy Artifacts: [C:\azure-vsts-agent-build\_work\78\s\App\.appSourceCopPackages\] to [c:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82]
2021-08-02T07:28:43.4631226Z *** Import NAV/BC Management DLL's
2021-08-02T07:28:43.4642231Z *** Loading assemblies for ServiceTier [BC].
2021-08-02T07:28:43.6166622Z *** Loading assemblies from: [C:\program files\microsoft dynamics nav\180\service]
2021-08-02T07:28:43.6232618Z *** Selected module: [C:\program files\microsoft dynamics nav\180\service\Microsoft.Dynamics.Nav.Apps.Management.dll]
2021-08-02T07:28:43.6244313Z *** Loading module: [C:\program files\microsoft dynamics nav\180\service\Microsoft.Dynamics.Nav.Apps.Management.dll]
2021-08-02T07:28:43.6469445Z *** Selected module: [C:\program files\microsoft dynamics nav\180\service\Microsoft.Dynamics.Nav.Management.dll]
2021-08-02T07:28:43.6471220Z *** Loading module: [C:\program files\microsoft dynamics nav\180\service\Microsoft.Dynamics.Nav.Management.dll]
2021-08-02T07:28:43.8998435Z *** Get Available Applications from [c:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82].
2021-08-02T07:28:43.9061639Z *** Get [*.app] files recursive from [c:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82]
2021-08-02T07:28:43.9113409Z *** Found [5] files.
2021-08-02T07:28:43.9120426Z *** Extract App detail from [C:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82\VanRoey.be_EDS Archive_1.0.0.0.app]
2021-08-02T07:28:44.0036956Z *** Extract App detail from [C:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82\MSApps\Microsoft_Application_17.4.21491.22792.app]
2021-08-02T07:28:44.0100382Z *** Extract App detail from [C:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82\MSApps\Microsoft_Base Application_17.4.21491.22792.app]
2021-08-02T07:28:44.0814247Z *** Extract App detail from [C:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82\MSApps\Microsoft_System Application_17.4.21491.22792.app]
2021-08-02T07:28:44.0822114Z *** Extract App detail from [C:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82\MSApps\Microsoft_System_17.0.21485.22699.app]
2021-08-02T07:28:44.1285416Z *** Installing Applications:
2021-08-02T07:28:44.1514819Z 
2021-08-02T07:28:44.1539395Z **************************************
2021-08-02T07:28:44.1546044Z  * App File = [C:\Run\DevOps\Artifacts-f3aea609e7be498b99da5f1e8d3cef82\MSApps\Microsoft_System Application_17.4.21491.22792.app]
2021-08-02T07:28:44.1547910Z 
2021-08-02T07:28:44.1573636Z  * App.ID        = 63ca2fa4-4f03-4f2b-a480-172fef340d3f
2021-08-02T07:28:44.1577922Z  * App.Name      = System Application
2021-08-02T07:28:44.1582932Z  * App.Publisher = Microsoft
2021-08-02T07:28:44.1598432Z  * App.Version   = 17.4.21491.22792
2021-08-02T07:28:44.3928015Z  => Publishing App 'System Application v17.4.21491.22792'
2021-08-02T07:28:44.6289788Z *** Get Tenants
2021-08-02T07:28:44.7743632Z *** Sync Tenant default
2021-08-02T07:28:45.1161851Z *** Sync App [System Application] on tenant [default]
2021-08-02T07:28:45.1676539Z *** Tenant Dataupgrade default
2021-08-02T07:28:45.3331446Z The extension 'System Application by Microsoft 17.4.21491.22792' has an older version than the baseline schema version 
2021-08-02T07:28:45.3331978Z '18.3.27240.28575'. The extension cannot be used on this database.
2021-08-02T07:28:45.3332311Z At line:146 char:17
2021-08-02T07:28:45.3332647Z + ...             $Result = Invoke-Command -Session $DockerSession -ScriptB ...
2021-08-02T07:28:45.3332959Z +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-08-02T07:28:45.3333249Z     + CategoryInfo          : InvalidOperation: (:) [Start-NAVAppDataUpgrade], InvalidOperationException
2021-08-02T07:28:45.3333579Z     + FullyQualifiedErrorId : MicrosoftDynamicsNavServer$BC/default,Microsoft.Dynamics.Nav.Apps.Management.Cmdlets.Sta 
2021-08-02T07:28:45.3333821Z    rtNavAppDataUpgrade
2021-08-02T07:28:45.3334565Z  
2021-08-02T07:28:45.3334624Z 
2021-08-02T07:28:45.3363757Z ##[error]Batch App-Publish failed: The extension 'System Application by Microsoft 17.4.21491.22792' has an older version than the baseline schema version '18.3.27240.28575'. The extension cannot be used on this database.
2021-08-02T07:28:45.3671034Z ##[section]Async Command Start: Add Build Tag
2021-08-02T07:28:45.3671652Z Build '14737' has following tags now: AL.Analyzer.CodeCop, AL.Analyzer.UICop, AL.Analyzer.AppSourceCop, App.System Application.17.4.21491.22792, BC.Platform.18.0.27224.28536, myimage-sandbox-18.3.27240.28575-be-10.0.17763.2029-ltsc2019, BC.Version.18.3.27240.28575
2021-08-02T07:28:45.3671996Z ##[section]Async Command End: Add Build Tag
2021-08-02T07:28:45.3672925Z ##[section]Finishing: Publish Previous version from appSourceCopPackages

What we mean is that you put BOTH types of apps into a subfolder. Something like:

That way, you can address the exact subfolder in the publish-step, while the compile still (recursively) get all symbols when necessary.

This way, we believe it's a much more generic way to work, as you can simply ignore one folder, and get all apps, regardless the publisher, name, or anything from the other.

Agree, creating subfolders solves the ALOpsAppPublish@1-step.

My idea of a generic way to work would be that ALOps handles the 'MS' folder. This would be as simple that in the Compile-step there files could be 'Just In Time' added to the .appSourceCopPackages (and remove these after compiling). I'm getting the idea that you're not enthusiastic about this direction of approach?

It's not that simple, I'm afraid. You talk about MS apps. But same could apply to (any) ISV app and such (MS apps is also "just" as an ISV app, no?). I don't beleave it's that straight forward.

Best is to keep it simple and manage it yourself .. for example by the example above, don't you think?

Since there is a solution (working with subfolders), I'm going to close this issue, if you don't mind.

HodorNV / ALOps

Working with AppSourceCopPackages #383