dotnet / arcade

Tools that provide common build infrastructure for multiple .NET Foundation projects.
MIT License
661 stars 335 forks source link

Checkout failure: "Git fetch failed with exit code 128" #9009

Open carlossanlop opened 2 years ago

carlossanlop commented 2 years ago

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

  1. Add the label "Known Build Error"
  2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

    {
    "ErrorMessage" : "Git fetch failed with exit code: 128",
    "BuildRetry": true
    }

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build Definition Step Name Console log Pull Request
759633 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42449
759632 dotnet/sdk Checkout dotnet/sdk@refs/pull/42449/merge to s Log dotnet/sdk#42449
2504860 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41381/merge to s Log #41381
757895 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/622/merge to s Log maestro-auth-test/helix-queue-insights-test#622
757850 dotnet/runtime Checkout dotnet/runtime@refs/pull/105589/merge to s Log dotnet/runtime#105589
757795 dotnet/runtime Checkout dotnet/runtime@refs/pull/105591/merge to s Log dotnet/runtime#105591
757189 dotnet/sdk Checkout dotnet/sdk@refs/pull/42404/merge to s Log dotnet/sdk#42404
757190 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42404
757082 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42403
757053 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42402
757081 dotnet/sdk Checkout dotnet/sdk@refs/pull/42403/merge to s Log dotnet/sdk#42403
757054 dotnet/sdk Checkout dotnet/sdk@refs/pull/42402/merge to s Log dotnet/sdk#42402
757055 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42402
756125 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/619/merge to s Log maestro-auth-test/helix-queue-insights-test#619
755791 dotnet/runtime Checkout dotnet/runtime@refs/pull/105164/merge to s Log dotnet/runtime#105164
755773 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/618/merge to s Log maestro-auth-test/helix-queue-insights-test#618
755539 dotnet/installer Checkout dotnet/installer@refs/pull/20034/merge to s Log dotnet/installer#20034
754989 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56945/merge to s Log dotnet/aspnetcore#56945
754991 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56945/merge to s Log dotnet/aspnetcore#56945
754616 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/617/merge to s Log maestro-auth-test/helix-queue-insights-test#617
754567 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/616/merge to s Log maestro-auth-test/helix-queue-insights-test#616
754547 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/615/merge to s Log maestro-auth-test/helix-queue-insights-test#615
754473 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/614/merge to s Log maestro-auth-test/helix-queue-insights-test#614
752420 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42316
752266 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42301
752145 dotnet/performance Checkout dotnet/performance@refs/pull/4334/merge to s Log dotnet/performance#4334
751741 dotnet/runtime Checkout dotnet/runtime@refs/pull/105306/merge to s Log dotnet/runtime#105306
751434 dotnet/runtime Checkout dotnet/runtime@refs/pull/104793/merge to s Log dotnet/runtime#104793
751403 dotnet/runtime Checkout dotnet/runtime@refs/pull/103755/merge to s Log dotnet/runtime#103755
751269 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/611/merge to s Log maestro-auth-test/helix-queue-insights-test#611
749823 dotnet/arcade-services Checkout dotnet/arcade-services@refs/pull/3747/merge to s Log dotnet/arcade-services#3747
749481 dotnet/runtime Checkout dotnet/runtime@refs/pull/105207/merge to s Log dotnet/runtime#105207
749457 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/609/merge to s Log maestro-auth-test/helix-queue-insights-test#609
749422 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/607/merge to s Log maestro-auth-test/helix-queue-insights-test#607
748869 dotnet/runtime Checkout dotnet/runtime@refs/pull/105182/merge to s Log dotnet/runtime#105182
2498629 dotnet-performance Checkout dotnet-performance@main to s Log
748328 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/606/merge to s Log maestro-auth-test/helix-queue-insights-test#606
747858 dotnet/runtime Checkout dotnet/runtime@refs/pull/105149/merge to s Log dotnet/runtime#105149
747582 dotnet/runtime Checkout dotnet/runtime@refs/pull/105140/merge to s Log dotnet/runtime#105140
747249 dotnet/runtime Checkout dotnet/runtime@refs/pull/105128/merge to s Log dotnet/runtime#105128
2497789 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41196/merge to s Log #41196
2497788 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41195/merge to s Log #41195
745860 dotnet/runtime Checkout dotnet/runtime@refs/pull/104503/merge to s dotnet/runtime#104503
745739 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/604/merge to s Log maestro-auth-test/helix-queue-insights-test#604
745626 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1364/merge to s Log maestro-auth-test/build-result-analysis-test#1364
745627 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1364/merge to s Log maestro-auth-test/build-result-analysis-test#1364
745550 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/600/merge to s Log maestro-auth-test/helix-queue-insights-test#600
745538 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/599/merge to s Log maestro-auth-test/helix-queue-insights-test#599
745531 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/598/merge to s Log maestro-auth-test/helix-queue-insights-test#598
745482 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1354/merge to s Log maestro-auth-test/build-result-analysis-test#1354
745481 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1354/merge to s Log maestro-auth-test/build-result-analysis-test#1354
745444 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1352/merge to s Log maestro-auth-test/build-result-analysis-test#1352
745089 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/597/merge to s Log maestro-auth-test/helix-queue-insights-test#597
744786 dotnet/runtime Checkout dotnet/runtime@refs/pull/104964/merge to s Log dotnet/runtime#104964
744131 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/595/merge to s Log maestro-auth-test/helix-queue-insights-test#595
744100 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/593/merge to s Log maestro-auth-test/helix-queue-insights-test#593
744049 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/592/merge to s Log maestro-auth-test/helix-queue-insights-test#592
2496390 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41107/merge to s Log #41107
743818 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/591/merge to s Log maestro-auth-test/helix-queue-insights-test#591
742643 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/590/merge to s Log maestro-auth-test/helix-queue-insights-test#590
742097 dotnet/runtime Checkout dotnet/runtime@refs/pull/103597/merge to s Log dotnet/runtime#103597
742043 dotnet/runtime Checkout dotnet/runtime@refs/pull/104395/merge to s Log dotnet/runtime#104395
741788 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/587/merge to s Log maestro-auth-test/helix-queue-insights-test#587
741311 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56798/merge to s Log dotnet/aspnetcore#56798
741313 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56798/merge to s Log dotnet/aspnetcore#56798
740863 dotnet/sdk Checkout dotnet/sdk@refs/pull/42148/merge to s Log dotnet/sdk#42148
740808 dotnet/runtime Checkout dotnet/runtime@refs/pull/104437/merge to s Log dotnet/runtime#104437
739787 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/585/merge to s Log maestro-auth-test/helix-queue-insights-test#585
739113 dotnet/arcade Checkout dotnet/arcade@refs/pull/14933/merge to s Log dotnet/arcade#14933
2493509 dotnet-runtime Checkout dotnet-runtime@release/6.0-staging to s Log
738721 dotnet/sdk Checkout dotnet/sdk@refs/pull/42114/merge to s Log dotnet/sdk#42114
738447 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/583/merge to s Log maestro-auth-test/helix-queue-insights-test#583
738408 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/582/merge to s Log maestro-auth-test/helix-queue-insights-test#582
738324 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/581/merge to s Log maestro-auth-test/helix-queue-insights-test#581
738237 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/580/merge to s Log maestro-auth-test/helix-queue-insights-test#580
737319 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56743/merge to s Log dotnet/aspnetcore#56743
737317 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56743/merge to s Log dotnet/aspnetcore#56743
737318 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56743/merge to s Log dotnet/aspnetcore#56743
737222 dotnet/runtime Checkout dotnet/runtime@refs/pull/103985/merge to s Log dotnet/runtime#103985
737123 dotnet/sdk Checkout dotnet/sdk@refs/pull/42095/merge to s Log dotnet/sdk#42095
735259 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/576/merge to s Log maestro-auth-test/helix-queue-insights-test#576
734843 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42047
734491 dotnet/runtime Checkout dotnet/runtime@refs/pull/104597/merge to s Log dotnet/runtime#104597
734337 dotnet/runtime Checkout dotnet/runtime@refs/pull/104226/merge to s Log dotnet/runtime#104226
733879 dotnet/runtime Checkout dotnet/runtime@refs/pull/104581/merge to s Log dotnet/runtime#104581
733855 dotnet/runtime Checkout dotnet/runtime@refs/pull/101631/merge to s Log dotnet/runtime#101631
733653 dotnet/performance Checkout dotnet/performance@refs/pull/4300/merge to s Log dotnet/performance#4300
2489943 dotnet-performance Checkout dotnet-performance@main to s Log
731531 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/574/merge to s Log maestro-auth-test/helix-queue-insights-test#574
731378 dotnet/runtime Checkout dotnet/runtime@refs/pull/104471/merge to s Log dotnet/runtime#104471
731004 dotnet/runtime Checkout dotnet/runtime@refs/pull/104452/merge to s Log dotnet/runtime#104452
730522 dotnet/sdk Checkout dotnet/sdk@refs/pull/41979/merge to s Log dotnet/sdk#41979
730521 dotnet/sdk Checkout dotnet/sdk@refs/pull/41977/merge to s Log dotnet/sdk#41977
730458 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41966
730460 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41966
728860 dotnet/installer Checkout dotnet/installer@refs/pull/19924/merge to s Log dotnet/installer#19924
728522 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/572/merge to s Log maestro-auth-test/helix-queue-insights-test#572
728001 dotnet/performance Checkout dotnet/performance@refs/pull/4295/merge to s Log dotnet/performance#4295
727857 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/571/merge to s Log maestro-auth-test/helix-queue-insights-test#571
726722 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/568/merge to s Log maestro-auth-test/helix-queue-insights-test#568

Displaying 100 of 102 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
3 27 102

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960 Error message validated: Git fetch failed with exit code: 128 Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 6/28/2023 9:15:14 PM UTC

ChadNedzlek commented 2 years ago

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

ulisesh commented 2 years ago

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

markwilkie commented 2 years ago

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

ulisesh commented 2 years ago

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues | where IssueId == 9009 and IssueRepository == "dotnet/arcade"

ViktorHofer commented 1 year ago

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?

premun commented 1 year ago

GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b We saw problems cloning all over. Maybe it's related to that?

dougbu commented 6 months ago

This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.com and a few also showed fatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.

Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including "BuildRetry": true in the JSON content in this description❔

carlossanlop commented 6 months ago

It's fine, we can try that, as long as we monitor it closely, because if we keep seeing the failure even with a retry, I think we should just go back to "BuildRetry": false, to avoid wasting resources.

dougbu commented 6 months ago

you mean "BuildRetry": true was tried in the past and didn't help much @carlossanlop❔ if the extra time it takes PR creators to click the Rerun failed builds actually helps avoid issues in the retry, I'd lean toward leaving things as-is.

carlossanlop commented 6 months ago

I do not know if it was tried in the past.

AlitzelMendez commented 6 months ago

sounds like a good plan @dougbu, let's try "BuildRetry": true, I will put together a query to monitor if it is helpful for this issue

dougbu commented 6 months ago

updated the description to try retries

AlitzelMendez commented 6 months ago

query:

let buildIds =
KnownIssues
| where ingestion_time() > datetime(2024-01-08 23:10:44.8683418)
| where IssueId  == 9009
| distinct BuildId;
TimelineBuilds
| where BuildId  in (buildIds)
| summarize arg_max(FinishTime, *) by BuildId
| summarize count() by Result, bin(FinishTime, 1d)
| order by FinishTime asc 
| render columnchart with (xcolumn=FinishTime)

I will check this on Monday to see how it is going, posting here the query so I don't loose it :)

AlitzelMendez commented 6 months ago

the success rate (having a green build) after the retry was between 30% to 50% per day, so this doesn't solve the issue, it just helps to mitigate it a little bit.