[NuGet.org Bug]: Face with 504 Gateway Timeout while Restore Packages

Javidleo commented 6 months ago

Impact

I'm unable to use NuGet.org

Describe the bug

until yesterday(last time I build up my application) everything was ok, but now I'm facing with 504 and 502 errors while building my dotnet application via docker compose. I hope following info will help: (names and project routes are fake)

Determining projects to restore... Restored /home/project/backend/src/Common/EventBus/Project.EventBus/Project..EventBus.csproj (in 116 ms). Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.fileproviders.physical/index.json'. Response status code does not indicate success: 502 (Bad Gateway). Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.abstractions/index.json'. Response status code does not indicate success: 502 (Bad Gateway). Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.cookies/index.json'. Response status code does not indicate success: 502 (Bad Gateway). Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/system.threading.channels/index.json'. Response status code does not indicate success: 502 (Bad Gateway). Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/swashbuckle.aspnetcore.swaggerui/index.json'. Response status code does not indicate success: 502 (Bad Gateway). Restored /route-to-project/src/Common/classlib/classlib.csproj (in 2.93 sec). Restored /route-to-project/src/Common/classlib/classlib.csproj (in 3.98 sec). Restored Restored /route-to-project/src/Common/serviceclasslib/serviceclasslib.csproj (in 5.88 sec). Failed to download package 'Geocoding.Microsoft.4.0.1' from 'https://api.nuget.org/v3-flatcontainer/geocoding.microsoft/4.0.1/geocoding.microsoft.4.0.1.nupkg'. Response status code does not indicate success: 504 (Gateway Time-out). Restored /route-to-project/src/Common/service/service.csproj (in 8.96 sec). Failed to download package 'Geocoding.Microsoft.4.0.1' from 'https://api.nuget.org/v3-flatcontainer/geocoding.microsoft/4.0.1/geocoding.microsoft.4.0.1.nupkg'. Response status code does not indicate success: 504 (Gateway Time-out). /usr/lib/dotnet/sdk/6.0.125/NuGet.targets(130,5): error : Failed to download package 'Geocoding.Microsoft.4.0.1' from 'https://api.nuget.org/v3-flatcontainer/geocoding.microsoft/4.0.1/geocoding.microsoft.4.0.1.nupkg'.

more details :

I recently updated my gitlab and its ssl certificate, is there any related stuff between downloading/restoring packages and ssl certificates? if it is, please someone share a blog post or a document about it.

Repro Steps

I don't know how this error produced at first time, but something that might help is

I was working on updating my gitlab and its ssl certificate
I'm using .NET 6 on this project
this project uses docker-compose and builds as a set of containers.
I also tried to restore just one of the projects with dotnet cli 'restore' command but it also failed.

Expected Behavior

as normal restore process I should be able to build my application as usual.

Screenshots

No response

Additional Context and logs

No response

jirawat070 commented 6 months ago

me too. in azure devops just retry 3 time and it back to work again i don't know why

zhhyu commented 6 months ago

Hi! My apologies for this inconvenience! Some intermittent failures were detected in some regions and likely our CDN provider in your region had a regression at that time. If it is still happening, could you please follow this document (https://learn.microsoft.com/en-us/nuget/nuget-org/nuget-org-faq#why-can-t-i-restore-packages-from-nuget-org) and send more diagnose information to the support mailbox? There are also some suggestions in the document that should help you enhance the resilience on the client side.

Feel free to reach out to the support mailbox or open an issue under this repo at any time when you meet failures!

Javidleo commented 6 months ago

Thanks for your review and document, I spend a lot of time on this problem to find a real practical solution for it. I close this issue. but can I ask you to describe this issue with more detail to others how looking in to this issue in the future to find a practical solution.

Javidleo commented 6 months ago

Thanks.

zhhyu commented 6 months ago

Hi! This is the server error, and we suggest you follow the document (https://learn.microsoft.com/en-us/nuget/nuget-org/nuget-org-faq#why-can-t-i-restore-packages-from-nuget-org) to enhance the client resilience (especially on CI machines without local cache). The infrastructure behind V3 API has the redundancy and can failover to another CDN provider when there is a serious outage. However, sometimes the failure is transient, and the regression is scoped within some regions (because of the nature of a CDN service). Please reach out to us when you see anything wrong.

If it blocks you and needs a quick mitigation, V2 API (https://learn.microsoft.com/en-us/nuget/nuget-org/overview-nuget-org#api-endpoint-for-nugetorg) is still working today but we don't recommend unless V3 API is not available. V3 API has better availability and performance and will have more feature support.

Javidleo commented 6 months ago

Thanks in advance

NuGet / NuGetGallery