dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.97k stars 4.65k forks source link

Test failure JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh #55162

Closed VincentBu closed 3 years ago

VincentBu commented 3 years ago

Run: runtime 20210704.33

Failed test:

Mono Linux x64 Release @ Ubuntu.1804.Amd64.Open

- JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

Error message:

=================================================================
External Debugger Dump:
[New LWP 7357]
[New LWP 7359]
[New LWP 7361]
[New LWP 7427]
[New LWP 7428]
[New LWP 7429]
[New LWP 7462]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa53442932a in __waitpid (pid=7503, stat_loc=0x7ffc2ce0a27c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id   Target Id         Frame

1    Thread 0x7fa534a50740 (LWP 7354) "corerun" 0x00007fa53442932a in __waitpid (pid=7503, stat_loc=0x7ffc2ce0a27c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
2    Thread 0x7fa530bff700 (LWP 7357) "SGen worker" 0x00007fa534424ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa5336b0488 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
3    Thread 0x7fa52eeab700 (LWP 7359) "corerun" 0x00007fa533726356 in __GI___sigsuspend (set=0x7fa5336a64f8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
4    Thread 0x7fa52ecaa700 (LWP 7361) "Finalizer" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
5    Thread 0x7fa52c72c700 (LWP 7427) ".NET ThreadPool" 0x00007fa533726356 in __GI___sigsuspend (set=0x7fa5336a64f8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
6    Thread 0x7fa52dd27700 (LWP 7428) ".NET ThreadPool" 0x00007fa533726356 in __GI___sigsuspend (set=0x7fa5336a64f8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
7    Thread 0x7fa52cff9700 (LWP 7429) ".NET ThreadPool" 0x00007fa533726356 in __GI___sigsuspend (set=0x7fa5336a64f8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
8    Thread 0x7fa52cdf8700 (LWP 7462) ".NET ThreadPool" 0x00007fa533726356 in __GI___sigsuspend (set=0x7fa5336a64f8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
Thread 8 (Thread 0x7fa52cdf8700 (LWP 7462)):

0  0x00007fa533726356 in __GI___sigsuspend (set=0x7fa5336a64f8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
1  0x00007fa5319b43f2 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7fa52cdf7280) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
2  <signal handler called>
3  0x00007fa534424fb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa52cdf78a0, expected=0, futex_word=0x7fa52cdf79d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
4  __pthread_cond_wait_common (abstime=0x7fa52cdf7958, mutex=0x560960848f80, cond=0x7fa52cdf79b0) at pthread_cond_wait.c:533
5  __pthread_cond_timedwait (cond=0x7fa52cdf79b0, mutex=0x560960848f80, abstime=0x7fa52cdf7958) at pthread_cond_wait.c:667
6  0x00007fa5319a55a4 in mono_os_cond_timedwait (cond=0x7fa52cdf79b0, mutex=0x560960848f80, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
7  0x00007fa5319ac934 in mono_coop_cond_timedwait (cond=0x7fa52cdf79b0, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
8  mono_lifo_semaphore_timed_wait (semaphore=0x560960848f80, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
9  0x0000000040ba3787 in ?? ()
10 0x00007fa530c3f678 in ?? ()
11 0x00007fa51800c208 in ?? ()
12 0x00007fa5336e5a08 in mono_runtime_callbacks () from /datadisks/disk1/work/9FBD08D3/p/libcoreclr.so
13 0x0000000000004e20 in ?? ()
14 0xff00000000000000 in ?? ()
15 0x00007fa50c002450 in ?? ()
16 0xfffffffeffffffff in ?? ()
17 0x00007fa52cdf7a20 in ?? ()
18 0x00007fa5318face0 in ?? () at /__w/1/s/src/mono/mono/metadata/icall.c:7835 from /datadisks/disk1/work/9FBD08D3/p/libcoreclr.so
19 0x0000000040ba3671 in ?? ()
20 0x00007fa530c3f5f8 in ?? ()
21 0x0000000040ba306e in ?? ()
22 0x00007fa51c252610 in ?? ()
23 0

Stack trace
   at JIT_Performance._CodeQuality_Roslyn_CscBench_CscBench_._CodeQuality_Roslyn_CscBench_CscBench_sh() in JIT.Performance.XUnitWrapper.dll:token 0x6000094+0x245
dotnet-issue-labeler[bot] commented 3 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

AndyAyersMS commented 3 years ago

Guessing this is a mono related issue, the pipline is Mono Linux x64 Release, and internal details show it's mono:

... at /__w/1/s/src/mono/mono/metadata/icall.c

Resetting the area path.

SamMonoRT commented 3 years ago

Moving to 7.0.0

VincentBu commented 3 years ago

Failed again in runtime-coreclr r2r-extra 20210724.1

Failed tests:

R2R OSX arm64 Checked jitstress1_tiered @ OSX.1100.ARM64.Open
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

R2R Linux arm Checked jitstress1 @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

R2R Linux x64 Checked jitstress1_tiered @ Ubuntu.1804.Amd64.Open
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

R2R Linux arm64 Checked jitstress1 @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

R2R Linux arm64 Checked jitstress1_tiered @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

R2R Linux arm Checked jitstress1_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

R2R Linux x64 Checked jitstress1 @ Ubuntu.1804.Amd64.Open
JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

Error Message:

/private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh: line 236: -r:/private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/IL-CG2/*.dll: No such file or directory rm: /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.dll.rsp: No such file or directory
Assert failure(PID 57581 [0x0000e0ed], Thread: 68895843 [0x41b4463]): Assertion failed 'kind == Overwrite' in 'Microsoft.CodeAnalysis.CodeGen.ILBuilder:RealizeBlocks():this' during 'Do value numbering' (IL size 537)
File: /Users/runner/work/1/s/src/coreclr/jit/jithashtable.h Line: 238
Image: /private/tmp/helix/working/AE98094B/p/corerun
task_for_pid(57581) FAILED 5 (os/kern) failure /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh: line 360: 57581 Abort trap: 6 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"
Return code: 1 Raw output file: /tmp/helix/working/AE98094B/w/ABAA098F/uploads/Reports/JIT.Performance/CodeQuality/Roslyn/CscBench/CscBench.output.txt Raw output: BEGIN EXECUTION in takeLock /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/IL-CG2/CscBench.dll Response file: /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.dll.rsp /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/IL-CG2/CscBench.dll -o:/private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.dll -r:/tmp/helix/working/AE98094B/p/System..dll -r:/tmp/helix/working/AE98094B/p/Microsoft..dll -r:/tmp/helix/working/AE98094B/p/mscorlib.dll --verify-type-and-field-layout --targetarch:arm64 -O Running CrossGen2: dotnet /tmp/helix/working/AE98094B/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.dll.rsp
Emitting R2R PE file: /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.dll in ReleaseLock /tmp/helix/working/AE98094B/p/corerun CscBench.dll '' Expected: 100 Actual: 134 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test:
set CORE_ROOT=/tmp/helix/working/AE98094B/p /private/tmp/helix/working/AE98094B/w/ABAA098F/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh Expected: True Actual: False

Stack trace
   at JIT_Performance._CodeQuality_Roslyn_CscBench_CscBench_._CodeQuality_Roslyn_CscBench_CscBench_sh() in JIT.Performance.XUnitWrapper.dll:token 0x6000096+0x295
AndyAyersMS commented 3 years ago

Failures just above were from CoreCLR tests and should be fixed by #56184.

ViktorHofer commented 3 years ago

Failed again in https://dev.azure.com/dnceng/public/_build/results?buildId=1304172&view=ms.vss-test-web.build-test-results-tab&runId=38414918&resultId=102097&paneView=debug

=================================================================
External Debugger Dump:
[New LWP 14480]
[New LWP 14481]
[New LWP 14482]
[New LWP 14491]
[New LWP 14492]
[New LWP 14499]
[New LWP 14538]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f784ebc432a in __waitpid (pid=14580, stat_loc=0x7fff0458e8bc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id   Target Id         Frame

1    Thread 0x7f784f1eb740 (LWP 14479) "corerun" 0x00007f784ebc432a in __waitpid (pid=14580, stat_loc=0x7fff0458e8bc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
2    Thread 0x7f784afff700 (LWP 14480) "SGen worker" 0x00007f784ebbfad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f784de4b568 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
3    Thread 0x7f7849603700 (LWP 14481) "corerun" 0x00007f784dec1356 in __GI___sigsuspend (set=0x7f784de415d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
4    Thread 0x7f7849402700 (LWP 14482) "Finalizer" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
5    Thread 0x7f784101c700 (LWP 14491) ".NET ThreadPool" 0x00007f784dec1356 in __GI___sigsuspend (set=0x7f784de415d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
6    Thread 0x7f7841fb5700 (LWP 14492) ".NET ThreadPool" 0x00007f784dec1356 in __GI___sigsuspend (set=0x7f784de415d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
7    Thread 0x7f7841a45700 (LWP 14499) ".NET ThreadPool" 0x00007f784dec1356 in __GI___sigsuspend (set=0x7f784de415d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
8    Thread 0x7f7841844700 (LWP 14538) ".NET ThreadPool" 0x00007f784dec1356 in __GI___sigsuspend (set=0x7f784de415d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
Thread 8 (Thread 0x7f7841844700 (LWP 14538)):

0  0x00007f784dec1356 in __GI___sigsuspend (set=0x7f784de415d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
1  0x00007f784c143172 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f7841843180) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
2  <signal handler called>
3  0x00007f784ebbffb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f7841843770, expected=0, futex_word=0x7f78418438a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
4  __pthread_cond_wait_common (abstime=0x7f7841843828, mutex=0x55761f39ac80, cond=0x7f7841843880) at pthread_cond_wait.c:533
5  __pthread_cond_timedwait (cond=0x7f7841843880, mutex=0x55761f39ac80, abstime=0x7f7841843828) at pthread_cond_wait.c:667
6  0x00007f784c1343b4 in mono_os_cond_timedwait (cond=0x7f7841843880, mutex=0x55761f39ac80, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
7  0x00007f784c13b6b4 in mono_coop_cond_timedwait (cond=0x7f7841843880, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
8  mono_lifo_semaphore_timed_wait (semaphore=0x55761f39ac80, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
9  0x0000000040e4ea77 in ?? ()
10 0x00007f784b03ecd0 in ?? ()
11 0x00007f783409dbf8 in ?? ()
12 0x00007f784de80d68 in mono_runtime_callbacks () from /datadisks/disk1/work/BC770A4B/p/libcoreclr.so
13 0x0000000000004e20 in ?? ()
14 0xff00000000000000 in ?? ()
15 0x00007f7828002450 in ?? ()
16 0xfffffffeffffffff in ?? ()
17 0x00007f78418438f0 in ?? ()
18 0x00007f784c08bf40 in ?? () at /__w/1/s/src/mono/mono/metadata/icall.c:7860 from /datadisks/disk1/work/BC770A4B/p/libcoreclr.so
19 0x0000000040e4e961 in ?? ()
20 0x00007f784b03ec50 in ?? ()
21 0x0000000040e4e0ce in ?? ()
22 0x00007f78380bc

Console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57627-merge-dbea856c729a47759f/JIT/1/console.35e92f2b.log?sv=2019-07-07&se=2021-09-08T13%3A37%3A44Z&sr=c&sp=rl&sig=dyIB1%2FQsEjtBqU9CX8%2FUm0gG3NSB1NbkcQT6Gbbwmts%3D

ViktorHofer commented 3 years ago

Failed again in https://dev.azure.com/dnceng/public/_build/results?buildId=1303556&view=ms.vss-test-web.build-test-results-tab&runId=38397882&resultId=102247&paneView=debug

Configuration: Mono Linux x64 Release @ Ubuntu.1804.Amd64.Open

=================================================================
External Debugger Dump:
[New LWP 22887]
[New LWP 22888]
[New LWP 22889]
[New LWP 22893]
[New LWP 22894]
[New LWP 22895]
[New LWP 22906]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f9201d9832a in __waitpid (pid=22910, stat_loc=0x7ffeae05df7c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id   Target Id         Frame

1    Thread 0x7f92023bf740 (LWP 22886) "corerun" 0x00007f9201d9832a in __waitpid (pid=22910, stat_loc=0x7ffeae05df7c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
2    Thread 0x7f91fe3ff700 (LWP 22887) "SGen worker" 0x00007f9201d93ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f920101f568 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
3    Thread 0x7f91fc718700 (LWP 22888) "corerun" 0x00007f9201095356 in __GI___sigsuspend (set=0x7f92010155d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
4    Thread 0x7f91fc517700 (LWP 22889) "Finalizer" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
5    Thread 0x7f91f43ac700 (LWP 22893) ".NET ThreadPool" 0x00007f9201095356 in __GI___sigsuspend (set=0x7f92010155d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
6    Thread 0x7f91f7639700 (LWP 22894) ".NET ThreadPool" 0x00007f9201095356 in __GI___sigsuspend (set=0x7f92010155d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
7    Thread 0x7f91f4c7a700 (LWP 22895) ".NET ThreadPool" 0x00007f9201095356 in __GI___sigsuspend (set=0x7f92010155d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
8    Thread 0x7f91ebfff700 (LWP 22906) ".NET ThreadPool" 0x00007f9201095356 in __GI___sigsuspend (set=0x7f92010155d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
Thread 8 (Thread 0x7f91ebfff700 (LWP 22906)):

0  0x00007f9201095356 in __GI___sigsuspend (set=0x7f92010155d8 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
1  0x00007f91ff317172 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f91ebffe180) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
2  <signal handler called>
3  0x00007f9201d93fb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f91ebffe770, expected=0, futex_word=0x7f91ebffe8a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
4  __pthread_cond_wait_common (abstime=0x7f91ebffe828, mutex=0x55f09c14be80, cond=0x7f91ebffe880) at pthread_cond_wait.c:533
5  __pthread_cond_timedwait (cond=0x7f91ebffe880, mutex=0x55f09c14be80, abstime=0x7f91ebffe828) at pthread_cond_wait.c:667
6  0x00007f91ff3083b4 in mono_os_cond_timedwait (cond=0x7f91ebffe880, mutex=0x55f09c14be80, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
7  0x00007f91ff30f6b4 in mono_coop_cond_timedwait (cond=0x7f91ebffe880, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
8  mono_lifo_semaphore_timed_wait (semaphore=0x55f09c14be80, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
9  0x0000000040367e97 in ?? ()
10 0x00007f91fe43fb80 in ?? ()
11 0x00007f91e0025b18 in ?? ()
12 0x00007f9201054d88 in mono_runtime_callbacks () from /datadisks/disk1/work/A74B0911/p/libcoreclr.so
13 0x0000000000004e20 in ?? ()
14 0xff00000000000000 in ?? ()
15 0x00007f91d8002450 in ?? ()
16 0xfffffffeffffffff in ?? ()
17 0x00007f91ebffe8f0 in ?? ()
18 0x00007f91ff25ff40 in ?? () at /__w/1/s/src/mono/mono/metadata/icall.c:7860 from /datadisks/disk1/work/A74B0911/p/libcoreclr.so
19 0x0000000040367d81 in ?? ()
20 0x00007f91fe43fb00 in ?? ()
21 0x00000000403675fe in ?? ()
22 0x00007f91ec143

@marek-safar @steveisok who should take a look at this?

ViktorHofer commented 3 years ago

Failed again in https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-70e8666793454dd4bb/JIT/1/console.55762a00.log?sv=2019-07-07&se=2021-09-10T05%3A21%3A31Z&sr=c&sp=rl&sig=slNZXuCJwQmllMApa5FNO8TfDI59K5keMhq9YsgXBRg%3D

Configuration: Mono Linux x64 Release @ Ubuntu.1804.Amd64.Open

SamMonoRT commented 3 years ago

@fanyang-mono -please take a look

VincentBu commented 3 years ago

Failed again in: runtime 20210902.89

Failed test:

Mono Linux x64 Release @ Ubuntu.1804.Amd64.Open

- JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

Mono Linux x64 Release no_tiered_compilation @ Ubuntu.1804.Amd64.Open

- JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh

Error message:

=================================================================
External Debugger Dump:
[New LWP 18013]
[New LWP 18014]
[New LWP 18015]
[New LWP 18016]
[New LWP 18017]
[New LWP 18018]
[New LWP 18021]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f055531332a in __waitpid (pid=18026, stat_loc=0x7ffce57e51c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id   Target Id         Frame

1    Thread 0x7f055593a740 (LWP 18012) "corerun" 0x00007f055531332a in __waitpid (pid=18026, stat_loc=0x7ffce57e51c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
2    Thread 0x7f05517ff700 (LWP 18013) "SGen worker" 0x00007f055530ead3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f055459acb8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
3    Thread 0x7f054fc90700 (LWP 18014) "corerun" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
4    Thread 0x7f054fa8f700 (LWP 18015) "Finalizer" 0x00007f05553117c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f055458bb40 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
5    Thread 0x7f054794b700 (LWP 18016) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
6    Thread 0x7f054c7ae700 (LWP 18017) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
7    Thread 0x7f0547fff700 (LWP 18018) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
8    Thread 0x7f0546fc8700 (LWP 18021) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
Thread 8 (Thread 0x7f0546fc8700 (LWP 18021)):

0  0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
1  0x00007f0552895302 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f0546fc7140) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
2  <signal handler called>
3  0x00007f055530efb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f0546fc7760, expected=0, futex_word=0x7f0546fc7898) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
4  __pthread_cond_wait_common (abstime=0x7f0546fc7818, mutex=0x5651b2f00af0, cond=0x7f0546fc7870) at pthread_cond_wait.c:533
5  __pthread_cond_timedwait (cond=0x7f0546fc7870, mutex=0x5651b2f00af0, abstime=0x7f0546fc7818) at pthread_cond_wait.c:667
6  0x00007f0552886544 in mono_os_cond_timedwait (cond=0x7f0546fc7870, mutex=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
7  0x00007f055288d844 in mono_coop_cond_timedwait (cond=0x7f0546fc7870, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
8  mono_lifo_semaphore_timed_wait (semaphore=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
9  0x0000000041500027 in ?? ()
10 0x00007f055183f698 in ?? ()
11 0x00007f0538071888 in ?? ()
12 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
13 0x0000000000004e20 in ?? ()
14 0xff00000000000000 in ?? ()
15 0x00007f052c002450 in ?? ()
16 0xfffffffeffffffff in ?? ()
17 0x00007f0546fc78e0 in ?? ()
18 0x00000000003c766b in ?? ()
19 0x00000000414fff11 in ?? ()
20 0x00007f055183f618 in ?? ()
21 0x00000000414ff2be in ?? ()
22 0x0000

Stack trace
   at JIT_Performance._CodeQuality_Roslyn_CscBench_CscBench_._CodeQuality_Roslyn_CscBench_CscBench_sh()
fanyang-mono commented 3 years ago

This is a dup of https://github.com/dotnet/runtime/issues/58062

lambdageek commented 3 years ago

@ViktorHofer @VincentBu hey folks, just FYI: when you get failures like this, the "Error message" in AzDO is usually cut off: it only seems to include the stack trace from one thread (usually not the failing one). If you go to the "Attachments" tab and look at the console log, you can get the rest of the crash.

For example in the most recent failing run it's "Thread 1" that has the actual crash cause. (Which as Fan noted is a dupe of an existing issue).

=================================================================
External Debugger Dump:
=================================================================
[New LWP 18013]
[New LWP 18014]
[New LWP 18015]
[New LWP 18016]
[New LWP 18017]
[New LWP 18018]
[New LWP 18021]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f055531332a in __waitpid (pid=18026, stat_loc=0x7ffce57e51c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7f055593a740 (LWP 18012) "corerun" 0x00007f055531332a in __waitpid (pid=18026, stat_loc=0x7ffce57e51c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  2    Thread 0x7f05517ff700 (LWP 18013) "SGen worker" 0x00007f055530ead3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f055459acb8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7f054fc90700 (LWP 18014) "corerun" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
  4    Thread 0x7f054fa8f700 (LWP 18015) "Finalizer" 0x00007f05553117c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f055458bb40 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7f054794b700 (LWP 18016) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
  6    Thread 0x7f054c7ae700 (LWP 18017) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
  7    Thread 0x7f0547fff700 (LWP 18018) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
  8    Thread 0x7f0546fc8700 (LWP 18021) ".NET ThreadPool" 0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26

Thread 8 (Thread 0x7f0546fc8700 (LWP 18021)):
#0  0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x00007f0552895302 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f0546fc7140) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
#2  <signal handler called>
#3  0x00007f055530efb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f0546fc7760, expected=0, futex_word=0x7f0546fc7898) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#4  __pthread_cond_wait_common (abstime=0x7f0546fc7818, mutex=0x5651b2f00af0, cond=0x7f0546fc7870) at pthread_cond_wait.c:533
#5  __pthread_cond_timedwait (cond=0x7f0546fc7870, mutex=0x5651b2f00af0, abstime=0x7f0546fc7818) at pthread_cond_wait.c:667
#6  0x00007f0552886544 in mono_os_cond_timedwait (cond=0x7f0546fc7870, mutex=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#7  0x00007f055288d844 in mono_coop_cond_timedwait (cond=0x7f0546fc7870, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#8  mono_lifo_semaphore_timed_wait (semaphore=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#9  0x0000000041500027 in ?? ()
#10 0x00007f055183f698 in ?? ()
#11 0x00007f0538071888 in ?? ()
#12 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#13 0x0000000000004e20 in ?? ()
#14 0xff00000000000000 in ?? ()
#15 0x00007f052c002450 in ?? ()
#16 0xfffffffeffffffff in ?? ()
#17 0x00007f0546fc78e0 in ?? ()
#18 0x00000000003c766b in ?? ()
#19 0x00000000414fff11 in ?? ()
#20 0x00007f055183f618 in ?? ()
#21 0x00000000414ff2be in ?? ()
#22 0x00007f053c26ee40 in ?? ()
#23 0x00007f055183f698 in ?? ()
#24 0x0000000000004e20 in ?? ()
#25 0x0000000000000046 in ?? ()
#26 0x00007f053c09d001 in ?? ()
#27 0xfffeffffffffffff in ?? ()
#28 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#29 0x00000000414f57ff in ?? ()
#30 0x000000453c09d0b8 in ?? ()
#31 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#32 0x00005651b307fcb8 in ?? ()
#33 0x00007f053c09d110 in ?? ()
#34 0x00007f053c09d0f0 in ?? ()
#35 0x00007f053c09d0e0 in ?? ()
#36 0x00007f055183f140 in ?? ()
#37 0x00000000414f1fa6 in ?? ()
#38 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f0547fff700 (LWP 18018)):
#0  0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x00007f0552895302 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f0547ffe140) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
#2  <signal handler called>
#3  0x00007f055530efb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f0547ffe760, expected=0, futex_word=0x7f0547ffe898) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#4  __pthread_cond_wait_common (abstime=0x7f0547ffe818, mutex=0x5651b2f00af0, cond=0x7f0547ffe870) at pthread_cond_wait.c:533
#5  __pthread_cond_timedwait (cond=0x7f0547ffe870, mutex=0x5651b2f00af0, abstime=0x7f0547ffe818) at pthread_cond_wait.c:667
#6  0x00007f0552886544 in mono_os_cond_timedwait (cond=0x7f0547ffe870, mutex=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#7  0x00007f055288d844 in mono_coop_cond_timedwait (cond=0x7f0547ffe870, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#8  mono_lifo_semaphore_timed_wait (semaphore=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#9  0x0000000041500027 in ?? ()
#10 0x00007f055183f698 in ?? ()
#11 0x00007f0538071888 in ?? ()
#12 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#13 0x0000000000004e20 in ?? ()
#14 0xff00000000000000 in ?? ()
#15 0x00007f0538002450 in ?? ()
#16 0xfffffffeffffffff in ?? ()
#17 0x00007f0547ffe8e0 in ?? ()
#18 0x00007f05527e3790 in ?? () at /__w/1/s/src/mono/mono/metadata/icall.c:7413 from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#19 0x00000000414fff11 in ?? ()
#20 0x00007f055183f618 in ?? ()
#21 0x00000000414ff2be in ?? ()
#22 0x00007f053c26ee40 in ?? ()
#23 0x00007f055183f698 in ?? ()
#24 0x0000000000004e20 in ?? ()
#25 0x0000000000000046 in ?? ()
#26 0x00007f053c09d001 in ?? ()
#27 0xfffeffffffffffff in ?? ()
#28 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#29 0x00000000414f57ff in ?? ()
#30 0x000000453c09d0b8 in ?? ()
#31 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#32 0x00005651b307fcb8 in ?? ()
#33 0x00007f053c09d110 in ?? ()
#34 0x00007f053c09d0f0 in ?? ()
#35 0x00007f053c09d0e0 in ?? ()
#36 0x00007f055183f140 in ?? ()
#37 0x00000000414f1fa6 in ?? ()
#38 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f054c7ae700 (LWP 18017)):
#0  0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x00007f0552895302 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f054c7acf80) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
#2  <signal handler called>
#3  0x00007f055530efb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f054c7ad570, expected=0, futex_word=0x7f053429ae64) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#4  __pthread_cond_wait_common (abstime=0x7f054c7ad610, mutex=0x7f053429ae10, cond=0x7f053429ae38) at pthread_cond_wait.c:533
#5  __pthread_cond_timedwait (cond=0x7f053429ae38, mutex=0x7f053429ae10, abstime=0x7f054c7ad610) at pthread_cond_wait.c:667
#6  0x00007f0551c85bd7 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f053429ae10, timeoutMilliseconds=500) at /__w/1/s/src/libraries/Native/Unix/System.Native/pal_threading.c:192
#7  0x00000000414fe1d7 in ?? ()
#8  0x0000000000000001 in ?? ()
#9  0x00007f0534008220 in ?? ()
#10 0x00007f0534008230 in ?? ()
#11 0x00000000003c760f in ?? ()
#12 0x00007f0534008238 in ?? ()
#13 0x0000000016e1f2ea in ?? ()
#14 0x0000000000000001 in ?? ()
#15 0x00007f0534002450 in ?? ()
#16 0x00000000000001f4 in ?? ()
#17 0x00007f054c7ad650 in ?? ()
#18 0x00005651b12629c0 in ?? ()
#19 0x00007f0534000b20 in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f054794b700 (LWP 18016)):
#0  0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x00007f0552895302 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f054794a140) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
#2  <signal handler called>
#3  0x00007f055530efb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f054794a760, expected=0, futex_word=0x7f054794a898) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#4  __pthread_cond_wait_common (abstime=0x7f054794a818, mutex=0x5651b2f00af0, cond=0x7f054794a870) at pthread_cond_wait.c:533
#5  __pthread_cond_timedwait (cond=0x7f054794a870, mutex=0x5651b2f00af0, abstime=0x7f054794a818) at pthread_cond_wait.c:667
#6  0x00007f0552886544 in mono_os_cond_timedwait (cond=0x7f054794a870, mutex=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#7  0x00007f055288d844 in mono_coop_cond_timedwait (cond=0x7f054794a870, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#8  mono_lifo_semaphore_timed_wait (semaphore=0x5651b2f00af0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#9  0x0000000041500027 in ?? ()
#10 0x00007f055183f698 in ?? ()
#11 0x00007f0538071888 in ?? ()
#12 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#13 0x0000000000004e20 in ?? ()
#14 0xff00000000000000 in ?? ()
#15 0x00007f053c002450 in ?? ()
#16 0xfffffffeffffffff in ?? ()
#17 0x00007f054794a8e0 in ?? ()
#18 0x00000000003c766f in ?? ()
#19 0x00000000414fff11 in ?? ()
#20 0x00007f055183f618 in ?? ()
#21 0x00000000414ff2be in ?? ()
#22 0x00007f053c26ee40 in ?? ()
#23 0x00007f055183f698 in ?? ()
#24 0x0000000000004e20 in ?? ()
#25 0x0000000000000046 in ?? ()
#26 0x00007f053c09d001 in ?? ()
#27 0xfffeffffffffffff in ?? ()
#28 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#29 0x00000000414f57ff in ?? ()
#30 0x000000453c09d0b8 in ?? ()
#31 0x00007f05545d04c8 in mono_runtime_callbacks () from /datadisks/disk1/work/B459093D/p/libcoreclr.so
#32 0x00005651b307fcb8 in ?? ()
#33 0x00007f053c09d110 in ?? ()
#34 0x00007f053c09d0f0 in ?? ()
#35 0x00007f053c09d0e0 in ?? ()
#36 0x00007f055183f140 in ?? ()
#37 0x00000000414f1fa6 in ?? ()
#38 0x00007f054794ae20 in ?? ()
#39 0x0000000100000001 in ?? ()
#40 0x000000000000f748 in ?? ()
#41 0x0000000000010000 in ?? ()
#42 0x00007f055183f0e0 in ?? ()
#43 0xffffffff00000001 in ?? ()
#44 0x0000565100000000 in ?? ()
#45 0x00005651b307fcb8 in ?? ()
#46 0x000000014794add8 in ?? ()
#47 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f054fa8f700 (LWP 18015)):
#0  0x00007f05553117c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f055458bb40 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7f055458bb40 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f05553118b8 in __new_sem_wait_slow (sem=0x7f055458bb40 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f05528595d6 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:876
#6  0x00007f0552831d5a in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1200
#7  0x00007f0552831be9 in start_wrapper (data=0x5651af411b10) at /__w/1/s/src/mono/mono/metadata/threads.c:1262
#8  0x00007f05553086db in start_thread (arg=0x7f054fa8f700) at pthread_create.c:463
#9  0x00007f05546f271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f054fc90700 (LWP 18014)):
#0  0x00007f0554610356 in __GI___sigsuspend (set=0x7f0554590d28 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x00007f0552895302 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7f054fc8f740) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
#2  <signal handler called>
#3  0x00007f05546e5cb9 in __GI___poll (fds=0x7f0548002d40, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#4  0x00007f0552ab7c4a in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#5  ds_ipc_poll (poll_handles_data=0x7f0548002530, poll_handles_data_len=1, timeout_ms=4294967295, callback=0x7f0552ab7140 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1082
#6  0x00007f0552ab5295 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7f0552ab7140 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#7  0x00007f0552ab3af9 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#8  0x00007f0552ab7121 in ep_rt_thread_mono_start_func (data=0x5651af2f77f0) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1266
#9  0x00007f05553086db in start_thread (arg=0x7f054fc90700) at pthread_create.c:463
#10 0x00007f05546f271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f05517ff700 (LWP 18013)):
#0  0x00007f055530ead3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f055459acb8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f055459ac68 <lock>, cond=0x7f055459ac90 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f055459ac90 <work_cond>, mutex=0x7f055459ac68 <lock>) at pthread_cond_wait.c:655
#3  0x00007f05528dd113 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x00007f05553086db in start_thread (arg=0x7f05517ff700) at pthread_create.c:463
#7  0x00007f05546f271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f055593a740 (LWP 18012)):
#0  0x00007f055531332a in __waitpid (pid=18026, stat_loc=0x7ffce57e51c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007f05529f0f77 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:842
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7ffce57e5d30, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:869
#3  0x00007f0552993c8e in mono_handle_native_crash (signal=0x7f0553e8cc73 "SIGSEGV", mctx=0x7ffce57e5d30, info=0x7ffce57e5ff0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2940
#4  0x00007f05528f4d61 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7ffce57e5ff0, context=0x7ffce57e5ec0, debug_fault_addr=0x8) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3717
#5  <signal handler called>
#6  copy_object_no_checks (obj=0x7f05518f2b78, queue=0x7ffce57e66f0) at /__w/1/s/src/mono/mono/mini/../sgen/sgen-copy-object.h:69
#7  0x00007f05528d3e00 in simple_nursery_serial_copy_object_from_obj (obj_slot=0x7f0551cff688, queue=0x7ffce57e66f0) at /__w/1/s/src/mono/mono/mini/../sgen/sgen-minor-copy-object.h:238
#8  simple_nursery_serial_scan_object (full_object=<optimized out>, desc=<optimized out>, queue=0x7ffce57e66f0) at /__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:64
#9  0x00007f05528d4ea7 in simple_nursery_serial_drain_gray_stack (queue=0x7ffce57e66f0) at /__w/1/s/src/mono/mono/mini/../sgen/sgen-minor-scan-object.h:142
#10 0x00007f05528aa89b in sgen_drain_gray_stack (ctx=...) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:578
#11 finish_gray_stack (generation=0, ctx=...) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:1140
#12 0x00007f05528a9a09 in collect_nursery (reason=0x7f0553e8e28f "Nursery full", is_overflow=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:1932
#13 0x00007f05528a5d53 in sgen_perform_collection_inner (requested_size=<optimized out>, generation_to_collect=<optimized out>, reason=<optimized out>, forced_serial=<optimized out>, stw=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2656
#14 sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x7f0553e8e28f "Nursery full", forced_serial=0, stw=1) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2768
#15 0x00007f05528a5b53 in sgen_ensure_free_space (size=4096, generation=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2622
#16 0x00007f0552899d17 in sgen_alloc_obj_nolock (vtable=0x5651afbea200, size=1056) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:279
#17 0x00007f05528699e7 in mono_gc_alloc_vector (vtable=0x5651afbea200, size=1056, max_length=1024) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:1340
#18 0x00000000413ebd6b in ?? ()
#19 0x00007f055183f3a0 in ?? ()
#20 0x00007f0551bf7708 in ?? ()
#21 0x00005651b67c6ca0 in ?? ()
#22 0x0000000000000400 in ?? ()
#23 0x00005651afbea200 in ?? ()
#24 0x00005651af2eebf0 in ?? ()
#25 0x00007ffce57e6a40 in ?? ()
#26 0x00007ffce57e6900 in ?? ()
#27 0x00005651b6afb990 in ?? ()
#28 0x00000000413ebed3 in ?? ()
#29 0x00005651b6a3aaa0 in ?? ()
#30 0x00007f05518c5580 in ?? ()
#31 0x00007f055183ef58 in ?? ()
#32 0x00007f05507ff000 in ?? ()
#33 0x00007f05518c5580 in ?? ()
#34 0x00000000415ef583 in ?? ()
#35 0x00005651b67c6cb8 in ?? ()
#36 0x00007f055183ef58 in ?? ()
#37 0x00007f05507ff000 in ?? ()
#38 0x00000000416a2653 in ?? ()
#39 0x0000000000000000 in ?? ()
/datadisks/disk1/work/B459093D/w/A04D08B8/e/JIT/Performance/CodeQuality/Roslyn/CscBench/CscBench.sh: line 355: 18012 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /datadisks/disk1/work/B459093D/w/A04D08B8/uploads/Reports/JIT.Performance/CodeQuality/Roslyn/CscBench/CscBench.output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/B459093D/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false CscBench.dll ''

=================================================================
Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
Native stacktrace:
=================================================================
0x7f05529f0e32 - Unknown
0x7f0552993c8e - Unknown
0x7f05528f4d61 - Unknown
0x7f0555313980 - Unknown
0x7f05528d4ee4 - Unknown
0x7f05528d3e00 - Unknown
0x7f05528d4ea7 - Unknown
0x7f05528aa89b - Unknown
0x7f05528a9a09 - Unknown
0x7f05528a5d53 - Unknown
0x7f05528a5b53 - Unknown
0x7f0552899d17 - Unknown
0x7f05528699e7 - Unknown
0x413ebd6b - Unknown

=================================================================
Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f05528d4ee4):0x7f05528d4ed4  56 41 55 41 54 53 50 49 89 f6 49 89 ff 4c 8b 27  VAUATSPI..I..L.'
0x7f05528d4ee4  49 8b 44 24 08 89 c1 83 e1 03 45 31 ed 48 83 f9  I.D$......E1.H..
0x7f05528d4ef4  03 74 10 25 07 c0 00 00 45 31 ed 48 83 f8 05 41  .t.%....E1.H...A
0x7f05528d4f04  0f 95 c5 4c 89 e7 4c 89 fe e8 1e 01 00 00 48 89  ...L..L.......H.

=================================================================
Managed Stacktrace:
=================================================================
  at <unknown> <0xffffffff>
  at System.Object:__icall_wrapper_mono_gc_alloc_vector <0x0008a>
  at System.Object:AllocVector <0x000f2>
  at Microsoft.Cci.BlobBuilder:.ctor <0x00062>
  at Microsoft.Cci.MetadataHeapsBuilder:SerializeStringHeap <0x000e2>
  at Microsoft.Cci.MetadataHeapsBuilder:Complete <0x00020>
  at Microsoft.Cci.MetadataWriter:SerializeMetadataAndIL <0x002ba>
  at Microsoft.Cci.PeWriter:WritePeToStream <0x0035f>
  at Microsoft.Cci.PeWriter:WritePeToStream <0x00142>
  at Microsoft.CodeAnalysis.Compilation:SerializeToPeStream <0x0061b>
  at Microsoft.CodeAnalysis.Compilation:Emit <0x00716>
  at Microsoft.CodeAnalysis.Compilation:Emit <0x001a7>
  at Microsoft.CodeAnalysis.Compilation:Emit <0x000bf>
  at CscBench:CompileBench <0x0028b>
  at CscBench:Main <0x00025>
  at <Module>:runtime_invoke_int <0x0005f>
=================================================================