search

PowerShell / PowerShellGallery

221 stars 61 forks source link

Is PowerShell Gallery server down? We are noticing an unusual delay to get response when we attempt to publish to PowerShell gallery. #132

Closed arunswarnam closed 3 years ago

arunswarnam commented 3 years ago

Even search through the console UI doesn't return expected results and times out.

mkht commented 3 years ago

Same for me. The package downloads very slowly and times out.

viralmodi commented 3 years ago

I see that too.

v-ronand commented 3 years ago

I am facing a similar issue. I am unable to install azure rm using powershell

o-l-a-v commented 3 years ago

Still down, or partially at least. PowerShellGallery status does not reflect that. image

hbuckle commented 3 years ago

Definitely having problems

2020-10-07T09:22:59Z:     azure-arm: VERBOSE: Downloading module 'Az' with version '4.7.0' from the repository 'https://www.powershellgallery.com/api/v2'.
2020-10-07T09:23:02Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az'' for ''.
2020-10-07T09:24:09Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'' for ''.
2020-10-07T09:24:53Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'&$skip=0&$top=40' for '2' more times
2020-10-07T09:26:29Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Advisor'' for ''.
2020-10-07T09:28:10Z:     azure-arm: VERBOSE: A task was canceled.
2020-10-07T09:28:10Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Advisor'&$skip=0&$top=40' for '2' more times
2020-10-07T09:28:10Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'' for ''.
2020-10-07T09:28:15Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Aks'' for ''.
2020-10-07T09:28:22Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.AnalysisServices'' for ''.
2020-10-07T09:28:22Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'' for ''.
2020-10-07T09:28:22Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'' for ''.
2020-10-07T09:31:13Z:     azure-arm: VERBOSE: A task was canceled.
2020-10-07T09:31:13Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'&$skip=0&$top=40' for '2' more times
2020-10-07T09:32:53Z:     azure-arm: VERBOSE: A task was canceled.
2020-10-07T09:32:53Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'&$skip=0&$top=40' for '1' more times
2020-10-07T09:33:15Z:     azure-arm: VERBOSE: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
2020-10-07T09:33:15Z: ==> azure-arm: Install-Package: C:\Program Files\PowerShell\Modules\PowerShellGet\2.2.5\PSModule.psm1:9711
2020-10-07T09:33:15Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'&$skip=0&$top=40' for '0' more times
2020-10-07T09:33:15Z: ==> azure-arm: Line |
2020-10-07T09:33:15Z: ==> azure-arm: 9711 |  … talledPackages = PackageManagement\Install-Package @PSBoundParameters
2020-10-07T09:33:15Z: ==> azure-arm:      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-07T09:33:15Z: ==> azure-arm:      | One or more errors occurred. (One or more errors occurred. (Object reference not set to an instance of
2020-10-07T09:33:15Z: ==> azure-arm:      | an object.)) (Object reference not set to an instance of an object.)
tunisiano187 commented 3 years ago

Yup, too

GreenBlood commented 3 years ago

Same here, its breaking one of our CI job.

SebastianGoers commented 3 years ago

Same here. installing modules from the Gallery takes ages and times out / failes eventually.

NikolaiO commented 3 years ago

https://psg-prod-eastus.azureedge.net/packages/awspowershell.netcore.4.0.5.nupkg

mirror works if you have local nuget repo/artifactory etc and you know the package names and versions

NikolaiO commented 3 years ago

also if your local repo thing is checking PSGallery - turn that off, else it'll be super slow...

NikolaiO commented 3 years ago

https://github.com/PowerShell/PowerShellGallery/pull/134/files

alerickson commented 3 years ago

The gallery's been experiencing ongoing low availability. The issue should be mitigated now. I unfortunately don't have write permissions to this repo, but I've put in a PR to update the status. Feel free to comment here with updates on whether builds are still breaking. We're working on completely resolving the issue, but I'm hoping the scenario's improved for most.

Skymogul commented 3 years ago

@alerickson The issue is definitely still ongoing. The gallery landing pages themselves appear fine but calls to download packages continue to fail.

siromega commented 3 years ago

In my CI/CD pipeline in AzDO it took 55 minutes to import-module Az. I don’t think it is fixed.

normj commented 3 years ago

We are still seeing issues doing our AWS PowerShell release. The release normally takes about an hour for the 200 packages we have to push but the release job has been going on for about 3.5 hours and we look to be only about half way through.

arunswarnam commented 3 years ago

Yes.. The availability is better than yesterday but the problem exists.

alerickson commented 3 years ago

We were seeing significant improvements when we switched to our backup backend cloud service, but the issue started to appear again. We're still working on mitigation right now.

NikolaiO commented 3 years ago

hope the issue can be revealed and talked about technically - seems like it would be interesting, given how much this service is used. As a SRE/DevOps person keen to be made aware of a thing and learn something new.

alerickson commented 3 years ago

I believe the issue is mitigated now, but we're monitoring to see if it presents itself again. @NikolaiO we'll get a post-mortem out on this issue as well.

lawrencegripper commented 3 years ago

I'm seeing a similar issue again here. Very long download times and connections hanging for an age.

mikeharder commented 3 years ago

@alerickson: We are seeing very long download times again.

MattJeanes commented 3 years ago

I'm seeing very long delays installing Az modules from PSGallery today, delays of approximately 30 minutes or more before it works

JanuszNowak commented 3 years ago

Here the same having issues for over 7 days.

alerickson commented 3 years ago

Unfortunately it might be another week until significant improvements start to show. Our database is processing a backlog of statistics right now, locking temporarily when running each batch. Today we've been hit by many malformed incoming urls as well. We're definitely prioritizing this issue and will continue to work on it until it's completely resolved.

MattJeanes commented 3 years ago

We've had a assortment of intermittent errors today, two of them being:

2020-10-23T11:04:19.0821567Z PackageManagement\Install-Package : Package 'Az.Accounts' failed to be installed because: End of Central Directory record could not be found.
2020-10-23T11:04:19.0825860Z At C:\program files\powershell\7\Modules\PowerShellGet\PSModule.psm1:9709 char:34
2020-10-23T11:04:19.0834946Z + … talledPackages = PackageManagement\Install-Package @PSBoundParameters
2020-10-23T11:04:19.0835461Z +                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-23T11:04:19.0835904Z + CategoryInfo          : InvalidResult: (Az.Accounts:String) [Install-Package], Exception
2020-10-23T11:04:19.0837230Z + FullyQualifiedErrorId : Package '{0}' failed to be installed because: {1},Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
2020-10-23T12:18:38.5223796Z PackageManagement\Install-Package : No match was found for the specified search criteria and module name 'Az.Storage'. Try Get-PSRepository to see all available registered module repositories.
2020-10-23T12:18:38.5228737Z At C:\program files\powershell\7\Modules\PowerShellGet\PSModule.psm1:9709 char:34
2020-10-23T12:18:38.5252604Z + … talledPackages = PackageManagement\Install-Package @PSBoundParameters
2020-10-23T12:18:38.5253090Z +                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-23T12:18:38.5253607Z + CategoryInfo          : ObjectNotFound: (Microsoft.PowerShel…lets.InstallPackage:InstallPackage) [Install-Package], Exception
2020-10-23T12:18:38.5254147Z + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage

This is running Install-Module from Powershell Core 7 on an Azure Pipelines Agent, and it worked after retrying but this is very frustrating. Statistics generation should really not impact the core download service, hopefully this issue is resolved soon it's causing our deployments to become very unstable.

mikeharder commented 3 years ago

@MattJeanes: If you are using the Microsoft-hosted agents, you might be able to use the AzurePowerShell task which uses a pre-installed copy of the Az modules:

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-powershell?view=azure-devops

MattJeanes commented 3 years ago

@mikeharder Our use case requires Az CLI and when using that task it doesn't seem to have the Az modules loaded so have to install, which is a shame.

mikeharder commented 3 years ago

@MattJeanes: In that case you might be able to use a workaround like we did: https://github.com/Azure/azure-sdk-tools/pull/1137/files

MattJeanes commented 3 years ago

@mikeharder that's super useful, I'll definitely give that a go - thank you!

staedter commented 3 years ago

I believe the issue is mitigated now, but we're monitoring to see if it presents itself again. @NikolaiO we'll get a post-mortem out on this issue as well.

Was there ever a post-mortem released on this?

SydneyhSmith commented 3 years ago

Blog post should be out covering this issue soon (at https://devblogs.microsoft.com/powershell/)