Run-ALValidation and Download-File won't work for other urls???

[GreatScott000]

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!

Describe the issue Trying to use Run-ALValidation. I have been following your CICD workshop and have my library app stored in blob storage. I need that for my for validation so I put the blob SAS url in as "installApps" parameter. I get a download error because my url is not in the list accepted by Download-File.

Is there a reason why Download-File is exclusive?
Is there a trick I am missing?

Scripts used to create container and cause the issue

function Invoke-AlAppValidation {
    [CmdletBinding()]
    Param(
        [Parameter(Mandatory=$false)]
        [ValidateSet('ci', 'current', 'nextmajor', 'nextminor')]
        [string] $version = ""
    )

    $baseFolder = (Get-Item (Join-Path $PSScriptRoot "..")).FullName
    . (Join-Path $PSScriptRoot "Read-Settings.ps1") -environment 'Local' -version $version | Out-Null
    . (Join-Path $PSScriptRoot "Install-BcContainerHelper.ps1") -bcContainerHelperVersion $bcContainerHelperVersion -genericImageName $genericImageName

    if (("$vaultNameForLocal" -eq "") -or -not (Get-AzKeyVault -VaultName $vaultNameForLocal)) {
        throw "You need to setup a Key Vault for use with local pipelines"
    }
    Get-AzKeyVaultSecret -VaultName $vaultNameForLocal `
        | Where-Object {$_.Name -ilike 'LicenseFile*'} `
        | ForEach-Object {
            Write-Host "Get Secret $($_.Name)Secret"
            Set-Variable -Name "$($_.Name)Secret" -Value (Get-AzKeyVaultSecret -VaultName $vaultNameForLocal -Name $_.Name -WarningAction SilentlyContinue)
        }
    $licenseFile = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($licenseFileSecret.SecretValue))

    $countries = @()
    $countries += $appSourceCopSupportedCountries

    $affixes = @()
    $affixes += $appSourceCopMandatoryAffixes

    $apps = @()
    foreach ($appPath in $appFolders.Split(',')) {
        if ($appPath.Length -gt 0) {
            if (Test-Path (Join-Path $baseFolder $appPath) -PathType Container) {
                $appJsonFile = Join-Path (Join-Path $baseFolder $appPath) "app.json"
                $appSettings = Get-Content $appJsonFile | ConvertFrom-Json
                $appFile = (Join-Path (Join-Path $baseFolder $appPath) ("{0}_{1}_{2}.app" -f $appSettings.publisher, $appSettings.name, $appSettings.version))
                $apps += $appFile

                $appSourceCopFile = Join-Path (Join-Path $baseFolder $appPath) "appSourceCop.Json"
                if (Test-Path $appSourceCopFile -PathType Leaf) {
                    $appSourceCopSettings = Get-Content $appSourceCopFile | ConvertFrom-Json
                    foreach ($affix in $appSourceCopSettings.mandatoryAffixes) {
                        if (-not $affixes.Contains($affix)) {
                            $affixes += $affix
                        }
                    }
                    foreach ($country in $appSourceCopSettings.supportedCountries) {
                        if (-not $countries.Contains($country)) {
                            $countries += $country
                        }
                    }
                }
            }
        }
    }

    Run-AlValidation `
        -licenseFile $licenseFile `
        -apps $apps `
        -installApps $installApps `
        -previousApps $previousApps `
        -affixes $affixes `
        -countries $countries `
        -skipVerification
}

Start-Transcript -Path (Join-Path $PSScriptRoot 'validatation.log') -Append
Clear-Host 
Write-Host "".PadLeft(10,'-') "$(Get-Date)" "".PadLeft(50,'-')

try {
    Measure-Command {

        Invoke-AlAppValidation `
            -version current

    } | ForEach-Object { Write-Host "base App Validation took $($_.TotalMinutes) minutes" -ForegroundColor Cyan }
}
finally {
    Stop-Transcript
}

Full output of scripts

---------- 05/21/2021 08:56:30 --------------------------------------------------
Set artifact = ///gb/Current
Set pipelineName = FinTools-current
Set containerName = fintools-current
Set installApps = 'https://\<storagename\>.blob.core.windows.net/extensionmanagement-preview/latest/apps.zip?\<authorization info\>
Set previousApps = ''
Set appSourceCopMandatoryAffixes = 'iMS'
Set appSourceCopSupportedCountries = 'gb'
Set appFolders = 'DayBooks,SourceNames,TrialBalance,VATAudit,VATReturn'
Set testFolders = 'TrialBalance.Test'
Set memoryLimit = '6G'
Set additionalCountries = ''
Set genericImageName = ''
Set vaultNameForLocal = 'EvolveAppVault'
Set bcContainerHelperVersion = 'Latest'
Set installTestFramework = True
Set installTestLibraries = True
Set installPerformanceToolkit = False
Set enableCodeCop = True
Set enableAppSourceCop = True
Set enablePerTenantExtensionCop = False
Set enableUICop = True
Set doNotSignApps = True
Set doNotRunTests = False
Set cacheImage = False
Set CreateRuntimePackages = True
Set rulesetfile = '_RuleSets\ims.ruleset.json'
Set appPackages = '.packages'
Setting hostHelperFolder = d:\bccontainerhelper
Setting use7zipIfAvailable = true
Setting bcartifactsCacheFolder = d:\bcartifacts.cache
BcContainerHelper version 2.0.12
Get Secret LicenseFileSecret
  _____                               _                
 |  __ \                             | |               
 | |__) |_ _ _ __ __ _ _ __ ___   ___| |_ ___ _ __ ___ 
 |  ___/ _` | '__/ _` | '_ ` _ \ / _ \ __/ _ \ '__/ __|
 | |  | (_| | | | (_| | | | | | |  __/ |_  __/ |  \__ \
 |_|   \__,_|_|  \__,_|_| |_| |_|\___|\__\___|_|  |___/

Container name               bcserver
Image name                   
Credential                   admin/Vybi7428
License file                 Specified
MemoryLimit                  8G
validateVersion              
validateCurrent              False
validateNextMinor            False
validateNextMajor            False
SasToken                     Not Specified
countries                    gb,us,ca,nz,au
validateCountries            gb,us,ca,nz,au
affixes                      iMS
supportedCountries           
vsixFile                     
Install Apps
- https://\<containterName\>.blob.core.windows.net/extensionmanagement-preview/latest/apps.zip?\<authorization info\>
Previous Apps
- None
Apps
- D:\Repos\evolveims\FinTools\DayBooks\Evolve iMS Ltd._iMS Finance Tools - Day Books_17.0.0.0.app
- D:\Repos\evolveims\FinTools\SourceNames\Evolve iMS Ltd._iMS Finance Tools - Source Names_17.0.0.0.app
- D:\Repos\evolveims\FinTools\TrialBalance\Evolve iMS Ltd._iMS Finance Tools - Trial Balance_17.0.0.0.app
- D:\Repos\evolveims\FinTools\VATAudit\Evolve iMS Ltd._iMS Finance Tools - VAT Audit_17.0.0.0.app
- D:\Repos\evolveims\FinTools\VATReturn\Evolve iMS Ltd._iMS Finance Tools - VAT Return_17.0.0.0.app
  _____       _                      _                         _   _  __           _       
 |  __ \     | |                    (_)                       | | (_)/ _|         | |      
 | |  | | ___| |_ ___ _ __ _ __ ___  _ _ __   ___    __ _ _ __| |_ _| |_ __ _  ___| |_ ___ 
 | |  | |/ _ \ __/ _ \ '__| '_ ` _ \| | '_ \ / _ \  / _` | '__| __| |  _/ _` |/ __| __/ __|
 | |__| |  __/ |_  __/ |  | | | | | | | | | |  __/ | (_| | |  | |_| | || (_| | (__| |_\__ \
 |_____/ \___|\__\___|_|  |_| |_| |_|_|_| |_|\___|  \__,_|_|   \__|_|_| \__,_|\___|\__|___/

Found https://bcartifacts.azureedge.net/sandbox/18.1.24822.25819/gb
Found https://bcartifacts.azureedge.net/sandbox/18.1.24822.25819/us
Found https://bcartifacts.blob.core.windows.net/sandbox/18.1.24822.25819/ca
Found https://bcartifacts.azureedge.net/sandbox/18.1.24822.25819/nz
Found https://bcartifacts.azureedge.net/sandbox/18.1.24822.25819/au
Using https://bcartifacts.azureedge.net/sandbox/18.1.24822.25819/gb
  _____       _                      _       _                   _                           _                       
 |  __ \     | |                    (_)     (_)                 | |                         | |                      
 | |  | | ___| |_ ___ _ __ _ __ ___  _ _ __  _ _ __   __ _    __| | ___ _ __   ___ _ __   __| | ___ _ __   ___ _   _ 
 | |  | |/ _ \ __/ _ \ '__| '_ ` _ \| | '_ \| | '_ \ / _` |  / _` |/ _ \ '_ \ / _ \ '_ \ / _` |/ _ \ '_ \ / __| | | |
 | |__| |  __/ |_  __/ |  | | | | | | | | | | | | | | (_| | | (_| |  __/ |_) |  __/ | | | (_| |  __/ | | | (__| |_| |
 |_____/ \___|\__\___|_|  |_| |_| |_|_|_| |_|_|_| |_|\__, |  \__,_|\___| .__/ \___|_| |_|\__,_|\___|_| |_|\___|\__, |
                                                      __/ |            | |                                      __/ |
                                                     |___/             |_|                                     |___/ 
Downloading C:\Users\micro\AppData\Local\Temp\fba27686-f35f-4299-ad02-51f0eb7ff521
Transcript stopped, output file is D:\Repos\evolveims\FinTools\_scripts\validatation.log
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (403) Forbidden."
At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\2.0.12\Misc\Download-File.ps1:54 char:9
+         (New-Object TimeoutWebClient -ArgumentList (1000*$timeout)).D ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : WebException

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

• does it happen all the time?
• did it use to work?

[GreatScott000]

Okay so I misread the code but still don't understand. This implies a 100 second timeout, correct? My file is not that big and downloads quickly in Edge. Why would I fall into the Catch statement?

[int]    $timeout = 100
(New-Object TimeoutWebClient -ArgumentList (1000*$timeout)).DownloadFile($sourceUrl, $destinationFile)

[GreatScott000]

Ignore this a bit. Something else going on.

[freddydk]

My assumption is that the URL in installapps is wrong. The forbidden is not coming from Download-File - it comes from the remote server: "The remote server returned an error: (403) Forbidden." likely the auth info is wrong

[GreatScott000]

Yeah, can't find download it with (New-Object System.Net.WebClient).DownloadFile($uri, $OutFile) but can with the uri in edge. How does that make sense?

[freddydk]

I don't know. Download-file is used to download artifacts (also insider artifacts) so it works for secure urls as well. My assumption is that it is some encoding of some kind. If you get stuck, you can email me a url which doesn't work and I can have a look.

[GreatScott000]

Okay, I think I have found the problem.

    Run-AlValidation `
        -licenseFile $licenseFile `
        -apps $apps `  #Type is Object[]
        -installApps $installApps ` #Type is string
        -previousApps $previousApps `
        -countries 'gb' `
        -affixes $affixes `
        -skipVerification

in Run-AlValidation at Line 213

if ($installApps                    -is [String]) { $installApps = @($installApps.Split(',').Trim() | Where-Object { $_ }) }
#$installApps is now an Object[]

in Run-AlValidation at Line 220

$installApps = $installApps | % { GetFilePath $_ }
#$installApps is now a string

in Run-AlValidation at Line 331

@(CopyAppFilesToFolder -appFiles @($installApps+$apps) -folder $tmpAppsFolder)...
#@($installApps+$apps) try to combine string and object[] and returns an object[] of length 1 rather than an object[] of length 6 (in my case)
#that looks like installAppUrlApp1path App2path App3path App4path App5Path note there is no space between the installAppUrl and App1Path

That then gets passed to CopyAppFilesToFolder and then ultimately to Download-File with 'installAppUrlApp1path App2path App3path App4path App5Path' as the sourceUrl and then I get the 403 error.

Even if I pass $installApps as an object[] e.g. @($installApps) Run-AlValidation at line 220 converts it back to a string.

One way to fix this is after line 220 to the conversion of installApps to an array again like at line 213 if ($installApps -is [String]) { $installApps = @($installApps.Split(',').Trim() | Where-Object { $_ }) } However I am sure there is a more elegant way

[GreatScott000]

FYI, this is BCContainerHelper 2.12

[freddydk]

I will investigate

[GreatScott000]

Yes I tried that with the variable not the directly with the url, if that would make any difference.

If you need it I can provide my script/apps/etc.

[freddydk]

I will let you know if I cannot repro. Thanks for the detailed investigations.

[freddydk]

I tried to add this parameter:

-installApps @( "https://businesscentralapps.blob.core.windows.net/bingmaps/latest/apps.zip?sv=2020-04-08&st=2021-05-26T06%3A11%3A30Z&se=2021-05-27T06%3A11%3A30Z&sr=b&sp=r&sig=%2FvjrB0V0yFF4aes3leY7CZmEuJCPgD6lovje6j6AP3g%3D" )

that works fine.

Could you email me the URL you use?

[GreatScott000]

Hi Freddy, I sent an email with the url. It has the issue number in the subject.

[freddydk]

Shipped in 2.0.13