search

dotnet / dnceng

.NET Engineering Services
MIT License
25 stars 18 forks source link

restarted. Azure DevOps can't recover from restarts. #3879

Open lewing opened 1 month ago

lewing commented 1 month ago

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

https://github.com/dotnet/runtime/pull/106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

Additional information about the issue reported

No response

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852 Error message validated: [restarted. Azure DevOps can't recover from restarts.] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build Definition Step Name Console log Pull Request
846075 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108352
847491 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108876
847485 dotnet/runtime browser-wasm linux Release LibraryTests_Threading
2563840 dotnet-dotnet AzureLinux_x64_Cross_Alpine_x64
847170 dotnet/runtime browser-wasm linux Release LibraryTests_Threading
847017 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#107769
846952 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108806
846739 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108968
846570 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108636
846537 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#107889
846532 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108982
846401 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108482
846366 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108636
846301 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108701
846206 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108324
846183 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108188
845938 dotnet/runtime browser-wasm linux Release LibraryTests_Threading
845879 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108701
845885 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108757
845438 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108482
845530 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#107843
2562362 dotnet-dotnet Ubuntu2404_x64
2562262 dotnet-sdk PGO: linux (x64)
845009 dotnet/runtime browser-wasm linux Release LibraryTests
844992 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108620
844862 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108685
844629 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108925
844468 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108701
844331 dotnet/sdk TestAsTools: windows (x64) dotnet/sdk#44152
844111 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#107889
844098 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108910
843842 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#107843
843776 dotnet/runtime browser-wasm linux Release LibraryTests_Threading
843503 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108853
843221 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108701
843118 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108837
843043 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108670
842876 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108876
842846 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108757
2560687 dotnet-dotnet AzureLinux_x64_Cross_x64
842587 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108867
842416 dotnet/runtime CoreCLR Pri1 Runtime Tests Run windows x64 checked
842558 dotnet/sdk AoT: windows (x64) dotnet/sdk#43889
842319 dotnet/runtime CoreCLR Pri0 Runtime Tests Run windows arm64 checked dotnet/runtime#108300
2560451 dotnet-roslyn Build_Windows_Debug
842277 dotnet/runtime browser-wasm linux Release LibraryTests
842306 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#107843
842057 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108748
841320 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108701
841950 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108837
841904 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#107909
841753 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#104608
841616 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108748
841312 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108819
2559755 dotnet-dotnet LinuxBionic_Shortstack_x64
841284 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108757
841176 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108818
841154 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108770
2559477 dotnet-dotnet AzureLinux_x64_Cross_x64
841104 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108636
841089 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108707
841081 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108757
841039 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#102249
841038 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#102249
841022 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108819
841004 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#103375
840993 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108819
840978 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108818
840934 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108808
840921 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#103375
840878 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#103375
840767 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#103375
840766 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#103375
840745 dotnet/runtime browser-wasm linux Release LibraryTests_Threading dotnet/runtime#108757
840746 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108757
840674 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#103375
840592 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108808
840511 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108808
840432 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108809
840418 dotnet/runtime browser-wasm linux Release LibraryTests_Threading
840378 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108807
840357 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108795
840352 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108765
840302 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108802
840279 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#107378
840268 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108795
840243 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108772
840182 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108703
840161 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108727
840142 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108775
840129 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108795
840128 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#108795
840121 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108789
840006 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#107378
840005 dotnet/runtime browser-wasm linux Release LibraryTests dotnet/runtime#107378
2558373 dotnet-dotnet AzureLinux_x64_Cross_Mono_arm64
839983 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108770
839967 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#105099
839947 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#106695
839938 dotnet/runtime linux-x64 debug Libraries_AllConfigurations dotnet/runtime#108727

Displaying 100 of 715 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
8 86 715
am11 commented 1 week ago

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

am11 commented 1 week ago

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

image

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

steveisok commented 1 week ago

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

ericstj commented 1 week ago

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples: https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857 https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923 https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.