Alternative Package Sources for BCContainerHelper?

[staedter]

Describe the issue As of tonight the PSGallery is down for atleast 8 hours now and because of that all our pipelines are obviously failing during the Install-Navcontainer Task.

@freddydk I know that this is neither your fault nor your responsibility. And the responsible Team is apparently having a bad time with their CDN and are scrambling to get the Service back up. (https://github.com/PowerShell/PowerShellGallery/issues/135#issuecomment-719437957)

While we are trying to mitigate this issue in our own pipelines we were discussing multiple options (locally caching the Packages from PSGallery or checkout of this Repo in the Pipeline among others ), we were wondering if it would be possible for you to also provide the releases here on Github either as a Release or as Github Package or even both.

That would give us more options to use as the package source and would eliminate the PSGallery as the single point of failure that it apparently is (which we did not even see coming despite the recent availability issues https://github.com/PowerShell/PowerShellGallery/issues/132#issuecomment-704875043)

I thought I might as well just asked directly here although the issue is not directly related to BcContainerHelper. I hope that was ok?

Scripts used to create container and cause the issue

Param(
    [ValidateSet('AzureDevOps','Local','AzureVM')]
    [string] $buildEnv = "AzureDevOps",

    [Parameter(Mandatory=$false)]
    [string] $navContainerHelperPath = $env:navContainerHelperPath,

    [Parameter(Mandatory=$false)]
    [string] $navContainerHelperVersion = $env:navContainerHelperVersion
)

if (-not $navContainerHelperVersion) { $navContainerHelperVersion = "latest" }

Write-Host "Version: $navContainerHelperVersion"

if ($navContainerHelperPath -ne "" -and (Test-Path $navContainerHelperPath)) {

    Write-Host "Using NavContainerHelper from $navContainerHelperPath"
    . $navContainerHelperPath

}
else {

    $module = Get-InstalledModule -Name navcontainerhelper -ErrorAction SilentlyContinue
    if ($module) {
        $versionStr = $module.Version.ToString()
        Write-Host "NavContainerHelper $VersionStr is installed"
        if ($navContainerHelperVersion -eq "latest") {
            Write-Host "Determine latest NavContainerHelper version"
            $latestVersion = (Find-Module -Name navcontainerhelper).Version
            $navContainerHelperVersion = $latestVersion.ToString()
            Write-Host "NavContainerHelper $navContainerHelperVersion is the latest version"
        }
        if ($navContainerHelperVersion -ne $module.Version) {
            Write-Host "Updating NavContainerHelper to $navContainerHelperVersion"
            Update-Module -Name navcontainerhelper -Force -RequiredVersion $navContainerHelperVersion
            Write-Host "NavContainerHelper updated"
        }
    }
    else {
        if (!(Get-PackageProvider -Name NuGet -ListAvailable -ErrorAction SilentlyContinue)) {
            Write-Host "Installing NuGet Package Provider"
            Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.208 -Force -WarningAction SilentlyContinue | Out-Null
        }
        if ($navContainerHelperVersion -eq "latest") {
            Write-Host "Installing NavContainerHelper"
            Install-Module -Name navcontainerhelper -Force
        }
        else {
            Write-Host "Installing NavContainerHelper version $navContainerHelperVersion"
            Install-Module -Name navcontainerhelper -Force -RequiredVersion $navContainerHelperVersion
        }
        $module = Get-InstalledModule -Name navcontainerhelper -ErrorAction SilentlyContinue
        $versionStr = $module.Version.ToString()
        Write-Host "NavContainerHelper $VersionStr installed"
    }
}

Full output of scripts

2020-10-30T07:54:42.1176516Z ##[debug]Evaluating condition for step: 'Install BcContainerHelper'
2020-10-30T07:54:42.1179910Z ##[debug]Evaluating: SucceededNode()
2020-10-30T07:54:42.1181042Z ##[debug]Evaluating SucceededNode:
2020-10-30T07:54:42.1183071Z ##[debug]=> True
2020-10-30T07:54:42.1184105Z ##[debug]Result: True
2020-10-30T07:54:42.1185119Z ##[section]Starten: Install BcContainerHelper
2020-10-30T07:54:42.1330214Z ==============================================================================
2020-10-30T07:54:42.1330593Z Task         : PowerShell
2020-10-30T07:54:42.1331000Z Description  : Hiermit wird ein PowerShell-Skript für Linux, macOS oder Windows ausgeführt.
2020-10-30T07:54:42.1331391Z Version      : 2.170.1
2020-10-30T07:54:42.1331664Z Author       : Microsoft Corporation
2020-10-30T07:54:42.1332071Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2020-10-30T07:54:42.1332579Z ==============================================================================
2020-10-30T07:54:43.1418420Z ##[debug]VstsTaskSdk 0.11.0 commit 7ff27a3e0bdd6f7b06690ae5f5b63cb84d0f23f4
2020-10-30T07:54:43.4123031Z ##[debug]Entering C:\agent\_work\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.170.1\powershell.ps1.
2020-10-30T07:54:43.4269115Z ##[debug]Loading resource strings from: C:\agent\_work\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.170.1\task.json
2020-10-30T07:54:43.4557438Z ##[debug]Loaded 11 strings.
2020-10-30T07:54:43.4607356Z ##[debug]SYSTEM_CULTURE: 'de-DE'
2020-10-30T07:54:43.4635733Z ##[debug]Loading resource strings from: C:\agent\_work\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.170.1\Strings\resources.resjson\de-DE\resources.resjson
2020-10-30T07:54:43.4908701Z ##[debug]Loaded 11 strings.
2020-10-30T07:54:43.5178243Z ##[debug]INPUT_ERRORACTIONPREFERENCE: 'stop'
2020-10-30T07:54:43.5234053Z ##[debug]INPUT_FAILONSTDERR: 'true'
2020-10-30T07:54:43.5283390Z ##[debug] Converted to bool: True
2020-10-30T07:54:43.5316588Z ##[debug]INPUT_IGNORELASTEXITCODE: 'false'
2020-10-30T07:54:43.5345023Z ##[debug] Converted to bool: False
2020-10-30T07:54:43.5375205Z ##[debug]INPUT_PWSH: 'false'
2020-10-30T07:54:43.5403492Z ##[debug] Converted to bool: False
2020-10-30T07:54:43.5433254Z ##[debug]INPUT_WORKINGDIRECTORY: 'C:\agent\_work\54\s'
2020-10-30T07:54:43.5585532Z ##[debug]Asserting container path exists: 'C:\agent\_work\54\s'
2020-10-30T07:54:43.5624890Z ##[debug]INPUT_TARGETTYPE: 'filePath'
2020-10-30T07:54:43.5665115Z ##[debug]INPUT_FILEPATH: 'C:\agent\_work\54\s\Pipeline\Install-BcContainerHelper.ps1'
2020-10-30T07:54:43.5692182Z ##[debug]Asserting leaf path exists: 'C:\agent\_work\54\s\Pipeline\Install-BcContainerHelper.ps1'
2020-10-30T07:54:43.5742244Z ##[debug]INPUT_ARGUMENTS (empty)
2020-10-30T07:54:43.5816463Z Skript wird erstellt.
2020-10-30T07:54:43.5892239Z Formatierter Befehl: . 'C:\agent\_work\54\s\Pipeline\Install-BcContainerHelper.ps1'
2020-10-30T07:54:43.6043162Z ##[debug]AGENT_VERSION: '2.170.1'
2020-10-30T07:54:43.6142403Z ##[debug]AGENT_TEMPDIRECTORY: 'C:\agent\_work\_temp'
2020-10-30T07:54:43.6172981Z ##[debug]Asserting container path exists: 'C:\agent\_work\_temp'
2020-10-30T07:54:43.6472654Z ##[debug]Asserting leaf path exists: 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe'
2020-10-30T07:54:43.6492231Z ========================== Starting Command Output ===========================
2020-10-30T07:54:43.6639618Z ##[debug]Entering Invoke-VstsTool.
2020-10-30T07:54:43.6756338Z ##[debug] Arguments: '-NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent\_work\_temp\1fcc03a5-7450-40e1-b1d1-a38392567124.ps1'"'
2020-10-30T07:54:43.6781026Z ##[debug] FileName: 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe'
2020-10-30T07:54:43.6805502Z ##[debug] WorkingDirectory: 'C:\agent\_work\54\s'
2020-10-30T07:54:43.6868572Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent\_work\_temp\1fcc03a5-7450-40e1-b1d1-a38392567124.ps1'"
2020-10-30T07:54:44.0629894Z Version: latest
2020-10-30T07:54:54.7830648Z Installing BcContainerHelper
2020-10-30T08:01:22.8376865Z WARNUNG: Unable to resolve package source 'https://www.powershellgallery.com/api/v2'.
2020-10-30T08:01:22.9403414Z ##[debug]STDERR: PackageManagement\Install-Package : Für die angegebenen Suchkriterien und den Paketnamen "BcContainerHelper" wurde 
2020-10-30T08:01:22.9438426Z ##[debug]STDERR: keine Übereinstimmung gefunden. Verwenden Sie Get-PSRepository, um alle verfügbaren, registrierten Paketquellen 
2020-10-30T08:01:22.9458754Z ##[debug]
2020-10-30T08:01:22.9493588Z ##[debug]STDERR: anzuzeigen.
2020-10-30T08:01:22.9828696Z ##[debug]
2020-10-30T08:01:22.9849673Z ##[debug]STDERR: In C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809 Zeichen:21
2020-10-30T08:01:22.9877100Z ##[debug]
2020-10-30T08:01:23.0536515Z ##[debug]STDERR: + ...          $null = PackageManagement\Install-Package @PSBoundParameters
2020-10-30T08:01:23.0567518Z ##[debug]
2020-10-30T08:01:23.0585688Z ##[debug]STDERR: +                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-30T08:01:23.0600811Z ##[debug]
2020-10-30T08:01:23.0627181Z ##[debug]STDERR:     + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], Ex 
2020-10-30T08:01:23.0643365Z ##[debug]
2020-10-30T08:01:23.0671753Z ##[debug]STDERR:    ception
2020-10-30T08:01:23.0687235Z ##[debug]
2020-10-30T08:01:23.0715600Z ##[debug]STDERR:     + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
2020-10-30T08:01:23.0732404Z ##[debug]
2020-10-30T08:01:23.0754570Z ##[debug]STDERR:  
2020-10-30T08:01:23.0769749Z ##[debug]
2020-10-30T08:01:23.0791989Z ##[debug]Exit code: 1
2020-10-30T08:01:23.0832945Z ##[debug]Leaving Invoke-VstsTool.
2020-10-30T08:01:23.1123495Z ##[error]PackageManagement\Install-Package : Für die angegebenen Suchkriterien und den Paketnamen "BcContainerHelper" wurde 
keine Übereinstimmung gefunden. Verwenden Sie Get-PSRepository, um alle verfügbaren, registrierten Paketquellen 

anzuzeigen.

In C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809 Zeichen:21

+ ...          $null = PackageManagement\Install-Package @PSBoundParameters

+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], Ex 

   ception

    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
2020-10-30T08:01:23.1131001Z ##[debug]Processed: ##vso[task.logissue type=error]PackageManagement\Install-Package : Für die angegebenen Suchkriterien und den Paketnamen "BcContainerHelper" wurde %0D%0Akeine Übereinstimmung gefunden. Verwenden Sie Get-PSRepository, um alle verfügbaren, registrierten Paketquellen %0D%0A%0D%0Aanzuzeigen.%0D%0A%0D%0AIn C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1809 Zeichen:21%0D%0A%0D%0A+ ...          $null = PackageManagement\Install-Package @PSBoundParameters%0D%0A%0D%0A+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%0D%0A%0D%0A    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package%5D, Ex %0D%0A%0D%0A   ception%0D%0A%0D%0A    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
2020-10-30T08:01:23.1224089Z ##[error]PowerShell wurde beendet mit dem Code "1".
2020-10-30T08:01:23.1226479Z ##[debug]Processed: ##vso[task.logissue type=error]PowerShell wurde beendet mit dem Code "1".
2020-10-30T08:01:23.1319121Z ##[debug]Processed: ##vso[task.complete result=Failed]Error detected
2020-10-30T08:01:23.1344749Z ##[debug]Leaving C:\agent\_work\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.170.1\powershell.ps1.
2020-10-30T08:01:23.1633573Z ##[section]Abschließen: Install BcContainerHelper

...

[freddydk]

There has been quite a lot of updates to the scripts and also the install-containerhelper. You should be able to see here: https://dev.azure.com/businesscentralapps/_git/HelloWorld?path=%2Fscripts%2FInstall-BcContainerHelper.ps1 how you can grab the module directly from github and use it. That typically is not as stable as the preview builds, but today it is.

[staedter]

That was my preferred option anyways, but my colleagues pointed out, that the master branch might not be as stable as the released PSGallery Package. That is what prompted my question.

I will look into it but I guess this would work for me as a solution. Thanks :)

[Sven-Niehus]

Hi @freddydk,

can you maybe look into using Github Package as an alternative to the PowerShell Gallery? It would also be really helpful if you could use the release feature to have the ability to download a specific version of the container helper directly from Github.

[freddydk]

@Sven-Niehus - yes, I am considering other options.

[staedter]

I copied your new Install-BcContainerHelper.ps1 script over into one pipeline and I got it to work, but I had to modify it a little bit. This version is now working for us:

Param(
    [string] $bcContainerHelperVersion = "",
    [string] $genericImageName = ""
)

if ($bcContainerHelperVersion -eq "") {
    $bcContainerHelperVersion = "latest"
}
Write-Host "Use bcContainerHelper Version: '$bcContainerHelperVersion'"

Write-Host 'Skipping PSGallery by setting $bcContainerHelperVersion = "https://github.com/microsoft/navcontainerhelper/archive/master.zip'
$bcContainerHelperVersion = "https://github.com/microsoft/navcontainerhelper/archive/master.zip"

$buildMutexName = "bcContainerHelper"
$buildMutex = New-Object System.Threading.Mutex($false, $buildMutexName)
try {
    try {
        if (!$buildMutex.WaitOne(1000)) {
            Write-Host "Waiting for other process to update BcContainerHelper"
            $buildMutex.WaitOne() | Out-Null
            Write-Host "Other process completed"
        }
    }
    catch [System.Threading.AbandonedMutexException] {
       Write-Host "Other process terminated abnormally"
    }

    $bcContainerHelperVersion = $bcContainerHelperVersion.Replace('{HOME}',$HOME.TrimEnd('\'))
    if ($bcContainerHelperVersion -like "?:\*") {
        if (Test-Path $bcContainerHelperVersion) {
            $bch = Get-Item (Join-Path $bcContainerHelperVersion '*ContainerHelper.ps1')
            if ($bch) {
                Write-Host "Using $bch"
                . "$bch"
                return
            }
        }
        $bcContainerHelperVersion = "https://github.com/microsoft/navcontainerhelper/archive/dev.zip"
    }

    if ($bcContainerHelperVersion -notlike "https://*") {
        try {
            Remove-Module -Name BcContainerHelper -Force -ErrorAction SilentlyContinue
            if ($bcContainerHelperVersion -eq "preview") {
                Write-Host "Determine latest BcContainerHelper preview version"
                $latestVersion = (Find-Module -Name bccontainerhelper -AllowPrerelease).Version
                $bcContainerHelperVersion = $latestVersion.ToString()
                 Write-Host "BcContainerHelper $bcContainerHelperVersion is the latest preview version"
            }
            elseif ($bcContainerHelperVersion -eq "latest") {
                 Write-Host "Determine latest BcContainerHelper version"
                 $latestVersion = (Find-Module -Name bccontainerhelper).Version
                 $bcContainerHelperVersion = $latestVersion.ToString()
                 Write-Host "BcContainerHelper $bcContainerHelperVersion is the latest version"
            }
            $modules = Get-InstalledModule -Name bccontainerhelper -ErrorAction SilentlyContinue -AllVersions -AllowPrerelease
            if ($modules | Where-Object { $_.Version -eq $bcContainerHelperVersion }) {
                if ($bcContainerHelperVersion -like "*preview*") {
                    Import-Module -Name BcContainerHelper -DisableNameChecking
                }
                else {
                    Import-Module -Name BcContainerHelper -RequiredVersion $bcContainerHelperVersion -DisableNameChecking
                }
            }
            else {
                if ($bcContainerHelperVersion -like "*preview*") {
                    Install-Module -Name BcContainerHelper -AllowPrerelease -Force
                    Import-Module -Name BcContainerHelper -Force -DisableNameChecking
                }
                else {
                    Install-Module -Name BcContainerHelper -RequiredVersion $bcContainerHelperVersion -Force
                    Import-Module -Name BcContainerHelper -RequiredVersion $bcContainerHelperVersion -Force -DisableNameChecking
                }
            }
        }
        catch {
            $bcContainerHelperVersion = "https://github.com/microsoft/navcontainerhelper/archive/master.zip"
        }
    }
    if ($bcContainerHelperVersion -like "https://*") {
        Remove-Module BcContainerHelper -ErrorAction SilentlyContinue
        #$tempName = Join-Path $env:TEMP $containerName      
        #Remove-Item $tempName -Recurse -Force -ErrorAction SilentlyContinue
        #Remove-Item "$tempName.zip" -Force -ErrorAction SilentlyContinue
        $tempName = $env:TEMP
        Write-Host "Downloading $bcContainerHelperVersion"
        #(New-Object System.Net.WebClient).DownloadFile($bcContainerHelperVersion, "$tempName.zip")
        Invoke-Webrequest -Uri $bcContainerHelperVersion -OutFile "$tempName\BcContainerHelper.zip" -UseBasicParsing
        Expand-Archive -Path "$tempName\BcContainerHelper.zip" -DestinationPath $tempName
        $modulePath = (Get-Item (Join-Path $tempName "*\BcContainerHelper.psm1")).FullName
        Write-Host $modulePath
        Import-Module $modulePath -DisableNameChecking
    }
}
finally {
    $buildMutex.ReleaseMutex()
}

if ($genericImageName) {
    $bcContainerHelperConfig.genericImageName = $genericImageName
}

I first switched to Invoke-WebRequest but I think that is not necessary anymore but I think there is a Problem with the removal of the TempName Folder ... after I commented that out it worked.

It's only a workaround but enough to get us trough the weekend hopefully.

[MatthewMcCarthy7]

When I go to look at the Hello World DevOps Repo for that change I keep getting kicked out for not having access to the resource. Did permission change in those Projects?

[freddydk]

The project is public - but maybe devops want you to login?

[staedter]

I had the same thing... after unsuccessfully logging in with my O365 credentials I could at least access the Repos...

[MatthewMcCarthy7]

I do login and then get

{"$id":"1","innerException":null,"message":"TF400813: The user '72f988bf-86f1-41af-91ab-2d7cd011db47\matt@XXXX.com' is not authorized to access this resource.","typeName":"Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server","typeKey":"UnauthorizedRequestException","errorCode":0,"eventId":3000}

[tfenster]

Azure DevOps has issues right now, I can't access organizations that I created myself and am very sure that I am administrator...

[freddydk]

One of these days....:-(

[tfenster]

TGIF ;)

[freddydk]

Alternative source is enabled - https://bccontainerhelper.azureedge.net/public?restype=blob&comp=list Latest in PS Gallery is also here: https://bccontainerhelper.azureedge.net/public/latest.zip Latest preview in PS Gallery is also here: https://bccontainerhelper.azureedge.net/public/preview.zip

[freddydk]

I have changed the way I install the containerhelper to always use the storage account. It is faster and I have more control over what's happening. https://dev.azure.com/businesscentralapps/_git/HelloWorld?path=%2Fscripts%2FInstall-BcContainerHelper.ps1