ghost
commented
3 months ago Tagging subscribers to this area: @dotnet/area-system-runtime See info in area-owners.md if you want to be subscribed.
Issue Details
### Error Blob ```json { "ErrorMessage": "", "BuildRetry": false, "ErrorPattern": "Killed.+System\.Runtime\.Tests\.runtimeconfig\.json.+OuterLoop", "ExcludeConsoleLog": false } ``` ### Reproduction Steps - Queues: - `linux_musl-x64 Debug CoreCLR_Debug` - `linux-x64 Debug CoreCLR_Debug` - Job results: - https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=592eed89-78f5-5202-9da5-ad17df3bef4a&t=4feb010c-08b1-5201-8e2c-97e3caf24223 - https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=df56ac33-e9fe-5eb2-5cdb-3303d7aad000&t=f62247f8-d051-5cb6-d5f0-06f532f9a005 - Log files: - https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-6fc0e798b0d84fd99e/System.Runtime.Tests/1/console.c8ffe024.log?helixlogtype=result - https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-ca234b0b7f3e48249a/System.Runtime.Tests/1/console.40eef85c.log?helixlogtype=result - Output:
```
========================== End custom configuration settings ===============================
----- start Mon Feb 26 23:56:03 UTC 2024 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Tests (found 29 of 9308 test cases)
Starting: System.Runtime.Tests (parallel test collections = on [2 threads], stop on fail = off)
./RunTests.sh: line 180: 21 Killed "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Mon Feb 26 23:56:28 UTC 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
----- start =============== XUnitLogChecker Output =====================================================
dmesg: klogctl: Operation not permitted
No dumps found in /home/helixbot/dotnetbuild/dumps.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================
+ export '_commandExitCode=137'
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 13931354 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhMTBlYjk0MS0xNzM2LTRjNzYtOWFiZC1iZDBlZGYxNjg0ODgiLCJzaWQiOiIxZTViN2VlMC1iNmQyLTQzNjgtODZjOS0zOTQ5MGIxM2FlODAiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzU4MDE5NSIsImpvYnJlZiI6IjMxNTZhMmQwLTExYWItNGExMC04NTA0LWY3Y2UxM2ZiMTk2OTo1OTJlZWQ4OS03OGY1LTUyMDItOWRhNS1hZDE3ZGYzYmVmNGEiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvNTgwMTk1Iiwib3JjaGlkIjoiMzE1NmEyZDAtMTFhYi00YTEwLTg1MDQtZjdjZTEzZmIxOTY5LmJ1aWxkLmJ1aWxkX2xpbnV4X211c2xfeDY0X2RlYnVnX2NvcmVjbHJfZGVidWcuX19kZWZhdWx0IiwicmVwb0lkcyI6IiIsImlzcyI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb20iLCJhdWQiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tfHZzbzo2ZmNjOTJlNS03M2E3LTRmODgtOGQxMy1kOTA0NWI0NWZiMjciLCJuYmYiOjE3MDg5ODg5MzgsImV4cCI6MTcwOTAwMDkzOH0.rjlcWYrlHLfuHwUtUIiDkhhUDwy8cnQZ3rCBJjmd8FuCN0BDFzjz0jEMQEbQ9WZ7LHqdWN40vrD1MHH2p9a2tC9JYY7ca4b2w8Yy05wlbG-_cjF_ksT3jfhgz4EOLBiSmE7-bW_m1b3gtKHo-rdFDyMt_vciLiIB_C3543QhcPGJJdph5ok1SjBVKbl1q9wzw4Tut1lLyo8-ToLRotVPvv2Fahz3TSp--EDbdhY2SYtNrPyGVW7t9F8o3TZdG5iyrtyTnCAlr2Qdu-cOUJhyXM_4vhztXvLdgDMuDgGINPpA7sDXoqo_7LB9RP1iNc35uIEM-430j7jeG9MDrZzEZQ
2024-02-26T23:56:35.053Z INFO run.py run(48) main Beginning reading of test results.
2024-02-26T23:56:35.054Z INFO run.py __init__(42) read_results Searching '/root/helix/work/workitem/e' for test results files
2024-02-26T23:56:35.054Z INFO run.py __init__(42) read_results Searching '/root/helix/work/workitem/uploads' for test results files
2024-02-26T23:56:35.062Z WARNING run.py __init__(55) read_results No results file found in any of the following formats: xunit, junit, trx
2024-02-26T23:56:35.063Z INFO run.py packing_test_reporter(30) report_results Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2024-02-26T23:56:35.067Z INFO run.py packing_test_reporter(33) report_results Packed 1541 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 580195 -workitem System.Runtime.Tests -jobid 6fc0e798-b0d8-4fd9-9e43-a36cfb9441c4 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
+ exit 137
+ export '_commandExitCode=137'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 137
[END EXECUTION]
Exit Code:137
```
| Author: | carlossanlop |
|---|---|
| Assignees: | - |
| Labels: | `area-System.Runtime`, `os-linux`, `arch-x64`, `blocking-clean-ci`, `runtime-coreclr`, `os-linux-musl`, `Known Build Error` |
| Milestone: | - |
danmoseley
ilonatommy
pavelsavara
Found in a PR that fixed an unrelated failure affecting all outerloop runs, hiding other failures. https://github.com/dotnet/runtime/pull/97135
Error Blob
Reproduction Steps
linux_musl-x64 Debug CoreCLR_Debuglinux-x64 Debug CoreCLR_DebugReport
Displaying 100 of 248 results
Summary
Known issue validation
Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195 Error message validated:
[Killed.+System\.Runtime\.Tests\.runtimeconfig\.json.+OuterLoop] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 2/27/2024 11:09:08 PM UTC