search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.98k stars 4.66k forks source link

`System.IO.Tests` getting killed (OOM?) for `Libraries Test Run release mono Linux arm64 Debug` #56165

Closed radical closed 2 years ago

radical commented 3 years ago

Build Log

/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.IO.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.Tests (found 678 of 681 test cases)
  Starting:    System.IO.Tests (parallel test collections = on, max threads = 6)
./RunTests.sh: line 162:    99 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Tests.runtimeconfig.json --depsfile System.IO.Tests.deps.json xunit.console.dll System.IO.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Thu Jul 22 01:29:17 UTC 2021 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill
ulimit -c value: unlimited
[463515.031418] [ 8294]  1000  8294    60030      559      19       5      498             0 gvfs-udisks2-vo
[463515.031421] [ 8295]  1000  8295    78109       93      19       4      292             0 gvfsd-trash
[463515.031424] [ 8326]  1000  8326    78002        0      19       5      787             0 gvfs-afc-volume
[463515.031428] [ 8337]  1000  8337    58449        2      15       4      219             0 gvfs-gphoto2-vo
[463515.031431] [ 8343]  1000  8343    96059        2      21       4      435             0 ibus-daemon
[463515.031434] [ 8348]  1000  8348   130049        2      43       3      900             0 evolution-addre
[463515.031437] [ 8351]  1000  8351    57949        0      14       4      185             0 gvfs-mtp-volume
[463515.031440] [ 8361]  1000  8361    57980        2      15       3      186             0 gvfs-goa-volume
[463515.031444] [ 8364]  1000  8364    58934        0      16       3      252             0 ibus-dconf
[463515.031447] [ 8369]  1000  8369    64870        0      28       4     1173             0 ibus-x11
[463515.031450] [ 8377]  1000  8377    58911        0      18       5      288             0 ibus-portal
[463515.031453] [ 8379]  1000  8379    40464        0      14       3      238             0 ibus-engine-sim
[463515.031456] [ 8403]  1000  8403   220173      181      58       5      802             0 evolution-addre
[463515.031459] [ 8515]  1000  8515      568        0       6       4       32             0 sh
[463515.031462] [ 8516]  1000  8516    58097        0      15       3      230             0 pxgsettings
[463515.031465] [ 8522]  1000  8522    39593        0      12       4      161             0 gvfsd-metadata
[463515.031468] [ 8545]  1000  8545    96589        0      22       4      853             0 gvfsd-network
[463515.031472] [ 8557]  1000  8557    77650       11      18       4      255             0 gvfsd-dnssd
[463515.031475] [ 8581]  1000  8581   137784        0      35       4     1070             0 zeitgeist-datah
[463515.031478] [ 8593]  1000  8593    95411      159      19       3      126             0 zeitgeist-daemo
[463515.031481] [ 8599]  1000  8599    66614        0      20       5      321             0 zeitgeist-fts
[463515.031484] [ 8715]  1000  8715   249740      772     152       5    49607             0 gnome-software
[463515.031487] [ 8764]  1000  8764   120880      118      36       5     1171             0 update-notifier
[463515.031490] [ 8831]  1000  8831      568        0       6       3       32             0 sh
[463515.031494] [ 8834]  1000  8834    98246       57      50       4     4887             0 check-new-relea
[463515.031497] [ 8997]     0  8997    96290       52      42       3     6192             0 fwupd
[463515.031500] [ 9009]  1000  9009      568        0       5       5       31             0 sh
[463515.031504] [ 9010]  1000  9010    58097        0      14       3      232             0 pxgsettings
[463515.031507] [ 9073]  1000  9073   174623     3572      94       4    10692             0 update-manager
[463515.031510] [ 9969]  1000  9969   128051      160      51       3     1378             0 deja-dup-monito
[463515.031513] [10442]  1000 10442   112670        0      57       3     2651             0 deja-dup
[463515.031516] [13810]  1000 13810    91547      383      36       3     1152             0 unity-panel-ser
[463515.031520] [28552]  1000 28552    98257       55      49       3     4894             0 check-new-relea
[463515.031526] [ 6171]     0  6171     2530      249      10       5        0          -999 containerd-shim
[463515.031529] [ 6230]  1001  6230      467        0       5       3       22             0 helix_docker_wo
[463515.031532] [ 6380]  1001  6380      467        0       4       3       24             0 execute.sh
[463515.031535] [ 6382]  1001  6382     1272        2       6       3       82             0 bash
[463515.031539] [ 6392]  1001  6392  2496741  1411294    2938      11    17607             0 dotnet
[463515.031542] [10040]     0 10040     1190       16       7       4        0             0 sleep
[463515.031545] [10169]     0 10169     3921      138       8       3      299             0 systemd-udevd
[463515.031548] [10170]     0 10170     3921      117       8       3      318             0 systemd-udevd
[463515.031552] [10171]     0 10171     3921      109       8       3      325             0 systemd-udevd
[463515.031554] [10172]     0 10172     3921      109       8       3      325             0 systemd-udevd
[463515.031557] [10173]     0 10173     3921      150       8       3      287             0 systemd-udevd
[463515.031561] [10174]     0 10174     3921      145       8       3      292             0 systemd-udevd
[463515.031564] [10198]     0 10198     3921      150       8       3      287             0 systemd-udevd
[463515.031567] [10201]     0 10201     3921      109       8       3      325             0 systemd-udevd
[463515.031570] [10202]     0 10202     3921      105       8       3      329         -1000 systemd-udevd
[463515.031573] Out of memory: Kill process 6392 (dotnet) score 474 or sacrifice child
[463515.054897] Killed process 6392 (dotnet) total-vm:9986964kB, anon-rss:5645176kB, file-rss:0kB, shmem-rss:0kB
dotnet-issue-labeler[bot] commented 3 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 3 years ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details
[Build](https://dev.azure.com/dnceng/public/_build/results?buildId=1251997&view=logs&jobId=3f95ca23-483b-5dc9-4b05-6bd24fb925c2&j=3f95ca23-483b-5dc9-4b05-6bd24fb925c2) [Log](https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-56117-merge-c5ee18108e2446fba0/System.IO.Tests/console.c40b4b9f.log?sv=2019-07-07&se=2021-08-11T01%3A06%3A19Z&sr=c&sp=rl&sig=xamsDrp5j2n0LGilrCLAGAiy0LiU6xeb4vL%2BGkhB3%2F8%3D) ``` /root/helix/work/workitem /root/helix/work/workitem Discovering: System.IO.Tests (method display = ClassAndMethod, method display options = None) Discovered: System.IO.Tests (found 678 of 681 test cases) Starting: System.IO.Tests (parallel test collections = on, max threads = 6) ./RunTests.sh: line 162: 99 Killed "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Tests.runtimeconfig.json --depsfile System.IO.Tests.deps.json xunit.console.dll System.IO.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE /root/helix/work/workitem ----- end Thu Jul 22 01:29:17 UTC 2021 ----- exit code 137 ---------------------------------------------------------- exit code 137 means SIGKILL Killed eg by kill ulimit -c value: unlimited [463515.031418] [ 8294] 1000 8294 60030 559 19 5 498 0 gvfs-udisks2-vo [463515.031421] [ 8295] 1000 8295 78109 93 19 4 292 0 gvfsd-trash [463515.031424] [ 8326] 1000 8326 78002 0 19 5 787 0 gvfs-afc-volume [463515.031428] [ 8337] 1000 8337 58449 2 15 4 219 0 gvfs-gphoto2-vo [463515.031431] [ 8343] 1000 8343 96059 2 21 4 435 0 ibus-daemon [463515.031434] [ 8348] 1000 8348 130049 2 43 3 900 0 evolution-addre [463515.031437] [ 8351] 1000 8351 57949 0 14 4 185 0 gvfs-mtp-volume [463515.031440] [ 8361] 1000 8361 57980 2 15 3 186 0 gvfs-goa-volume [463515.031444] [ 8364] 1000 8364 58934 0 16 3 252 0 ibus-dconf [463515.031447] [ 8369] 1000 8369 64870 0 28 4 1173 0 ibus-x11 [463515.031450] [ 8377] 1000 8377 58911 0 18 5 288 0 ibus-portal [463515.031453] [ 8379] 1000 8379 40464 0 14 3 238 0 ibus-engine-sim [463515.031456] [ 8403] 1000 8403 220173 181 58 5 802 0 evolution-addre [463515.031459] [ 8515] 1000 8515 568 0 6 4 32 0 sh [463515.031462] [ 8516] 1000 8516 58097 0 15 3 230 0 pxgsettings [463515.031465] [ 8522] 1000 8522 39593 0 12 4 161 0 gvfsd-metadata [463515.031468] [ 8545] 1000 8545 96589 0 22 4 853 0 gvfsd-network [463515.031472] [ 8557] 1000 8557 77650 11 18 4 255 0 gvfsd-dnssd [463515.031475] [ 8581] 1000 8581 137784 0 35 4 1070 0 zeitgeist-datah [463515.031478] [ 8593] 1000 8593 95411 159 19 3 126 0 zeitgeist-daemo [463515.031481] [ 8599] 1000 8599 66614 0 20 5 321 0 zeitgeist-fts [463515.031484] [ 8715] 1000 8715 249740 772 152 5 49607 0 gnome-software [463515.031487] [ 8764] 1000 8764 120880 118 36 5 1171 0 update-notifier [463515.031490] [ 8831] 1000 8831 568 0 6 3 32 0 sh [463515.031494] [ 8834] 1000 8834 98246 57 50 4 4887 0 check-new-relea [463515.031497] [ 8997] 0 8997 96290 52 42 3 6192 0 fwupd [463515.031500] [ 9009] 1000 9009 568 0 5 5 31 0 sh [463515.031504] [ 9010] 1000 9010 58097 0 14 3 232 0 pxgsettings [463515.031507] [ 9073] 1000 9073 174623 3572 94 4 10692 0 update-manager [463515.031510] [ 9969] 1000 9969 128051 160 51 3 1378 0 deja-dup-monito [463515.031513] [10442] 1000 10442 112670 0 57 3 2651 0 deja-dup [463515.031516] [13810] 1000 13810 91547 383 36 3 1152 0 unity-panel-ser [463515.031520] [28552] 1000 28552 98257 55 49 3 4894 0 check-new-relea [463515.031526] [ 6171] 0 6171 2530 249 10 5 0 -999 containerd-shim [463515.031529] [ 6230] 1001 6230 467 0 5 3 22 0 helix_docker_wo [463515.031532] [ 6380] 1001 6380 467 0 4 3 24 0 execute.sh [463515.031535] [ 6382] 1001 6382 1272 2 6 3 82 0 bash [463515.031539] [ 6392] 1001 6392 2496741 1411294 2938 11 17607 0 dotnet [463515.031542] [10040] 0 10040 1190 16 7 4 0 0 sleep [463515.031545] [10169] 0 10169 3921 138 8 3 299 0 systemd-udevd [463515.031548] [10170] 0 10170 3921 117 8 3 318 0 systemd-udevd [463515.031552] [10171] 0 10171 3921 109 8 3 325 0 systemd-udevd [463515.031554] [10172] 0 10172 3921 109 8 3 325 0 systemd-udevd [463515.031557] [10173] 0 10173 3921 150 8 3 287 0 systemd-udevd [463515.031561] [10174] 0 10174 3921 145 8 3 292 0 systemd-udevd [463515.031564] [10198] 0 10198 3921 150 8 3 287 0 systemd-udevd [463515.031567] [10201] 0 10201 3921 109 8 3 325 0 systemd-udevd [463515.031570] [10202] 0 10202 3921 105 8 3 329 -1000 systemd-udevd [463515.031573] Out of memory: Kill process 6392 (dotnet) score 474 or sacrifice child [463515.054897] Killed process 6392 (dotnet) total-vm:9986964kB, anon-rss:5645176kB, file-rss:0kB, shmem-rss:0kB ```
Author: radical
Assignees: -
Labels: `area-System.IO`, `untriaged`
Milestone: -
buyaa-n commented 2 years ago

failed again in build, log:

C:\h\w\ACCB09AD\w\B19709CA\e>"C:\h\w\ACCB09AD\p\dotnet.exe" exec --runtimeconfig System.IO.Tests.runtimeconfig.json --depsfile System.IO.Tests.deps.json xunit.console.dll System.IO.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing  
  Discovering: System.IO.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.Tests (found 6 of 685 test cases)
  Starting:    System.IO.Tests (parallel test collections = on, max threads = 2)
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:02:11
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:04:11
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:06:12
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:08:12
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:10:12
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:12:12
   System.IO.Tests: [Long Running Test] 'System.IO.Tests.BufferedStream_StreamAsync.NoInt32OverflowInTheBufferingLogic', Elapsed: 00:14:12
['System.IO.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
danmoseley commented 2 years ago

I am seeing a failure which I suspect is another manifestation:

    <test name="System.IO.Tests.FileStream_Read.NoInt32OverflowInTheBufferingLogic" type="System.IO.Tests.FileStream_Read" method="NoInt32OverflowInTheBufferingLogic" time="1.4505152" result="Fail">
        <traits>
          <trait name="category" value="outerloop" />
        </traits>
        <failure exception-type="System.IO.IOException">
          <message><![CDATA[System.IO.IOException : There is not enough space on the disk. : 'C:\\Users\\danmose\\AppData\\Local\\Temp\\FileStream_Read_veaai11q.g5k\\NoInt32OverflowInTheBufferingLogic_26_5e4f963f']]></message>
          <stack-trace><![CDATA[   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\RandomAccess.Windows.cs:line 153
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 282
   at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 269
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 1020
   at System.IO.Strategies.BufferedFileStreamStrategy.Flush(Boolean flushToDisk) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 804
   at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 150
   at System.IO.Stream.Dispose() in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Stream.cs:line 158
   at System.IO.Tests.FileStream_Read.NoInt32OverflowInTheBufferingLogic() in C:\git\runtime\src\libraries\System.IO.FileSystem\tests\FileStream\Read.cs:line 38]]></stack-trace>

This test is reading and writing a few bytes at the start and end of a 4GB file. Is this actually reading and writing 4GB, so it's taking a very long time? Is this related to running out of disk space also? It's not clear as my disk seems to have 14GB free space.

cc @adamsitnik ?

danmoseley commented 2 years ago

Ah I see two other failures concurrently - both of these reading/writing multi gigabyte files. Still, not entirely clear why they all failed if (at least after tests were done) I had 14GB.

Perhaps these 3 tests should be segregated into a fixture where they are run serially?

      <test name="System.IO.Tests.File_ReadWriteAllBytesAsync.ReadFileOverMaxArrayLengthAsync" type="System.IO.Tests.File_ReadWriteAllBytesAsync" method="ReadFileOverMaxArrayLengthAsync" time="0.0078009" result="Fail">
        <traits>
          <trait name="category" value="outerloop" />
        </traits>
        <failure exception-type="System.IO.IOException">
          <message><![CDATA[System.IO.IOException : There is not enough space on the disk. : 'C:\\Users\\danmose\\AppData\\Local\\Temp\\File_ReadWriteAllBytesAsync_b3sf1ema.fvq\\ReadFileOverMaxArrayLengthAsync_93_c50ebc9e']]></message>
          <stack-trace><![CDATA[   at System.IO.Strategies.FileStreamHelpers.SetFileLength(SafeFileHandle handle, Int64 length) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\FileStreamHelpers.Windows.cs:line 128
   at System.IO.Strategies.OSFileStreamStrategy.SetLengthCore(Int64 value) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 226
   at System.IO.Tests.File_ReadWriteAllBytesAsync.ReadFileOverMaxArrayLengthAsync() in C:\git\runtime\src\libraries\System.IO.FileSystem\tests\File\ReadWriteAllBytesAsync.cs:line 96
--- End of stack trace from previous location ---]]></stack-trace>
        </failure>
      </test>
      <test name="System.IO.Tests.File_ReadWriteAllBytesAsync.ReadFileOver2GBAsync" type="System.IO.Tests.File_ReadWriteAllBytesAsync" method="ReadFileOver2GBAsync" time="0.0081179" result="Fail">
        <traits>
          <trait name="category" value="outerloop" />
        </traits>
        <failure exception-type="System.IO.IOException">
          <message><![CDATA[System.IO.IOException : There is not enough space on the disk. : 'C:\\Users\\danmose\\AppData\\Local\\Temp\\File_ReadWriteAllBytesAsync_lvobmbw2.21p\\ReadFileOver2GBAsync_78_fb9b2321']]></message>
          <stack-trace><![CDATA[   at System.IO.Strategies.FileStreamHelpers.SetFileLength(SafeFileHandle handle, Int64 length) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\FileStreamHelpers.Windows.cs:line 128
   at System.IO.Strategies.OSFileStreamStrategy.SetLengthCore(Int64 value) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 226
   at System.IO.Tests.File_ReadWriteAllBytesAsync.ReadFileOver2GBAsync() in C:\git\runtime\src\libraries\System.IO.FileSystem\tests\File\ReadWriteAllBytesAsync.cs:line 81
--- End of stack trace from previous location ---]]></stack-trace>
        </failur
elinor-fung commented 2 years ago

Hit in another build - Libraries Test Run release mono Linux x64: https://dev.azure.com/dnceng/public/_build/results?buildId=1616197&view=results Console log

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.IO.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.Tests (found 694 of 700 test cases)
  Starting:    System.IO.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168:    24 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Tests.runtimeconfig.json --depsfile System.IO.Tests.deps.json xunit.console.dll System.IO.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Wed 16 Feb 2022 05:16:35 PM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill
ulimit -c value: unlimited
[11447.963370] [    464]     0   464    24428       44    90112        0             0 lvmetad
[11447.963372] [    474]     0   474    11073      443   122880        0         -1000 systemd-udevd
[11447.963373] [    540]     0   540     3067      267    65536        0             0 hv_kvp_daemon
[11447.963374] [    891] 62583   891    35446      109   188416        0             0 systemd-timesyn
[11447.963375] [    980]   100   980    20009      162   180224        0             0 systemd-network
[11447.963376] [    998]   101   998    17655      149   176128        0             0 systemd-resolve
[11447.963378] [   1205]   103  1205    12549      193   143360        0          -900 dbus-daemon
[11447.963379] [   1256]     0  1256    42772     1984   233472        0             0 networkd-dispat
[11447.963380] [   1257]     0  1257    72000      214   196608        0             0 accounts-daemon
[11447.963381] [   1275]     0  1275    20059     3263   204800        0             0 python3
[11447.963382] [   1277]     0  1277     7084       52    98304        0             0 atd
[11447.963383] [   1278]     0  1278    40270       34    81920        0             0 lxcfs
[11447.963385] [   1279]     0  1279    17639      176   172032        0             0 systemd-logind
[11447.963386] [   1280]     0  1280     7964       78   106496        0             0 cron
[11447.963387] [   1283]     0  1283    27605       55   114688        0             0 irqbalance
[11447.963388] [   1287]     0  1287   356251     6623   323584        0          -999 containerd
[11447.963389] [   1305]     0  1305     4104       38    77824        0             0 agetty
[11447.963390] [   1307]     0  1307     3723       33    69632        0             0 agetty
[11447.963391] [   1317]     0  1317    72221      197   200704        0             0 polkitd
[11447.963392] [   1448]     0  1448     1128       16    57344        0             0 none
[11447.963393] [   1515]     0  1515    96955     5088   270336        0             0 python3
[11447.963395] [   1742]     0  1742    18076      189   180224        0         -1000 sshd
[11447.963396] [   1873]     0  1873   411144    12504   589824        0          -500 dockerd
[11447.963397] [   2258]  1000  2258     2899       64    65536        0             0 helix.sh
[11447.963398] [   2810]     0  2810    37138     3160   192512        0             0 python3
[11447.963399] [   2815]     0  2815   919434     9206   327680        0             0 amacoreagent
[11447.963400] [   2949]     0  2949    79371       64   114688        0             0 mdsdmgr
[11447.963401] [   3333]  1000  3333    44344     6855   249856        0             0 python3
[11447.963402] [   3337]   106  3337     7150       45   102400        0             0 uuidd
[11447.963403] [   3362]  1000  3362    63771     7350   274432        0             0 python3
[11447.963404] [   3363]  1000  3363   150120    36994   622592        0             0 python3
[11447.963405] [   3724]     0  3724   298595     7220   274432        0             0 auoms
[11447.963406] [   3761]     0  3761   232121     8233   225280        0             0 auomscollect
[11447.963408] [   4022]   102  4022    67333      432   176128        0             0 rsyslogd
[11447.963409] [   4132]   102  4132   217767    17912   897024        0             0 mdsd
[11447.963410] [  15953]   111 15953    76694     1588   434176        0             0 freshclam
[11447.963411] [   6519]     0  6519   229350      337   147456        0             0 agentlauncher
[11447.963413] [   6606]     0  6606   238779     5499   356352        0             0 azsecd
[11447.963414] [   6625]     0  6625   195532     3255   196608        0             0 azsecmond
[11447.963415] [   8068]     0  8068   164951     4316   212992        0             0 azsecd
[11447.963416] [   8091]     0  8091   196985     3537   208896        0             0 azsec-clamscan
[11447.963417] [   8498]     0  8498   342477   299845  2666496        0             0 clamscan
[11447.963418] [  21627]     0 21627   178052     1606    94208        0          -998 containerd-shim
[11447.963420] [  21646]  1000 21646      596       18    45056        0             0 helix_docker_wo
[11447.963421] [  21716]  1000 21716      596       17    40960        0             0 execute.sh
[11447.963422] [  21718]  1000 21718     1714       86    49152        0             0 bash
[11447.963423] [  21728]  1000 21728  2398631  1504028 12693504        0             0 dotnet
[11447.963424] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/80594eb213552c83fb5fed48e52b898d82e04f31b28fcac43e44722685ff8a8d,task=dotnet,pid=21728,uid=1000
[11447.963463] Out of memory: Killed process 21728 (dotnet) total-vm:9594524kB, anon-rss:6016112kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12396kB oom_score_adj:0
[11448.206047] oom_reaper: reaped process 21728 (dotnet), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
adamsitnik commented 2 years ago

It was most likely solved by https://github.com/dotnet/runtime/pull/66387, I am going to close it for now. Please reopen if it ever happens again.