Open ulisesh opened 1 year ago
Could we potentially include the "https://pkgs.dev.azure.com/dnceng/public/_packaging"
part of the message in the error string? Seems like we'd want to sort "AzDO package feeds are failing" from other random 500's that might occur during a build.
@ChadNedzlek done
Build retry seems to be helping but we keep getting a couple of hits every day
Unfortunately, we keep seeing some hits every day. FR should investigate more, we might need to create an IcM to get some help from AzDo
It is interesting to me that the only hits we see come from the aspnetcore repo
sure I'll create a fresh IcM asking for investigation
It is interesting to me that the only hits we see come from the aspnetcore repo
The reason this is specific to AspNet is likely that most .NET Core repos do not use NPM to this extent, so have no/ far fewer chances to get NPM problems.
Created https://portal.microsofticm.com/imp/v3/incidents/details/359098770 to ask for an investigation.
Replied to requests in the IcM; I left step-by-step instructions how to get precise timestamps of what failed and such, hopefully they actually believe us now.
Pinged the IcM, no replies since updating it. Tossing this issue into tracking.
Still reproing, IcM ticket is just claiming they don't have telemetry for the problem still. Added repro from last night to the IcM and pinged the ticket.
With 0 hits in the last 7 days, I am closing the issue on our side.
Reopening for the dotnet-public-npm
feed errors we're seeing in dotnet/aspnetcore (a lot). For example,
error An unexpected error occurred: "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-npm/npm/registry/istanbul-lib-report/-/istanbul-lib-report-3.0.0.tgz: Request failed \"500 Internal Server Error\"".
info If you think this is a bug, please open a bug report with the information provided in "/Users/runner/work/1/s/src/SignalR/clients/ts/common/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Process stalled
Active handles:
- Socket
- Socket
- Socket
- TLSSocket
- TLSSocket
- TLSSocket
- TLSSocket
- TLSSocket
/Users/runner/work/1/s/eng/targets/Npm.Common.targets(45,5): error MSB6006: "yarn" exited with code 1. [/Users/runner/work/1/s/src/SignalR/clients/ts/common/common.npmproj]
##[error]eng/targets/Npm.Common.targets(45,5): error MSB6006: (NETCORE_ENGINEERING_TELEMETRY=Restore) "yarn" exited with code 1.
Affected aspnetcore-ci rolling builds over the last week:
Will those rolling builds be included in the tracking information automatically❔ Or, did I just set things up to track future failures (again)❔
Any build that matches the error string should trigger it. We may need to edit the original post and tweak the string; I've asked Ulises to take a peek and see if there's a reason it missed these.
I looked at #20230211.3 and couldn't figure out why build analysis couldn't find a match with this issue but I found a rolling build where the right things happen. I'll continue investigating
This is still happening and I am being told that the linked IcM is the same as https://portal.microsofticm.com/imp/v3/incidents/details/353857134/home, one about NuPKG. This is frustrating and I believe inaccurate, but not something I can do about. I will assign this issue to @ilyas1974 to keep it moving.
@AlitzelMendez - any chance you could take a peek and see if the known issue is catching this now?
@AlitzelMendez - any chance you could take a peek and see if the known issue is catching this now?
If you just click the top link on the issue it shows an instance happening 1 hour ago so I'm pretty sure the system is working.
The AzDO packaging team is evidently aware of the problem and has some ideas for folks using the version of Yarn ASP.NET is using and prepped a guide; @dougbu FYI
I brought this up in our ASP.NET Build teams channel. @wtgodbe is going to try adding retries to our yarn
commands. It's a bit complicated because we use the Yarn.MSBuild package and that wraps commands.
Our efforts to move to npm
aren't really moving yet but are planned for this year (.NET 8 timeframe).
Build
https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=93516
Build leg reported
Test: Blazor E2E tests on Linux / Run E2E tests
Pull Request
https://github.com/dotnet/aspnetcore/pull/44834
Action required for the engineering services team
To triage this issue (First Responder / @dotnet/dnceng):
If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:
Release Note Category
Release Note Description
Additional information about the issue reported
No response
Report
Summary