Dotnet-GitSync-Bot
commented
4 years ago I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.
Closed am11 closed 4 years ago
Dotnet-GitSync-Bot
commented
4 years ago I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.
am11
commented
4 years ago cc @janvorli, @jclulow
jclulow
commented
4 years ago Looking at the truss output in the non-root case, it seems like the process could not lock the memory because it does not have the privilege to do so:
...
56362/1: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEF010000
56362/1: memcntl(0xFFFFFC7FEF010000, 4096, MC_LOCK, 0, 0, 0) Err#1 EPERM [proc_lock_memory]
...Note that our mlock(3C) is defined to return EPERM if you do not have the appropriate privilege:
ERRORS
The mlock() and munlock() functions will fail if:
...
EPERM
The {PRIV_PROC_LOCK_MEMORY} privilege is not asserted in the
effective set of the calling process.You don't see an mlock() call in the truss output because it's looking at system calls, and our mlock() is a small wrapper around the memcntl(2) system call:
/*
* Function to lock address range in memory.
*/
int
mlock(caddr_t addr, size_t len)
{
return (memcntl(addr, len, MC_LOCK, 0, 0, 0));
}In your ppriv output you can see that you inheritable/permitted/effective set is just basic, and the proc_lock_memory privilege is not part of the basic set that non-root processes generally start with.
jclulow
commented
4 years ago Note that the privilege model is described in privileges(5), including which privileges are in the basic set.
I'm not sure what's going on with the segfault you're seeing. I suspect you'll need to look at the program text in the compiled binary to see what's going on there -- what kind of relocation it's using to find the external variable, etc.
am11
commented
4 years ago Thank you @jclulow! Adding the proc_lock_memory privilege fixed the non-sudo issue.
sudo usermod -K defaultpriv='basic,proc_lock_memory' am11
# logout and `zlogin -lam11` back to the zone
ppriv $$outputs:
56702: -bash
flags = <none>
E: basic,proc_lock_memory
I: basic,proc_lock_memory
P: basic,proc_lock_memory
L: basic,contract_event,contract_identity,contract_observer,dtrace_proc,dtrace_user,file_chown,file_chown_self,file_dac_execute,file_dac_read,file_dac_search,file_dac_write,file_owner,file_setid,ipc_dac_read,ipc_dac_write,ipc_owner,net_bindmlp,net_icmpaccess,net_mac_aware,net_observability,net_privaddr,net_rawaccess,proc_audit,proc_chroot,proc_lock_memory,proc_owner,proc_prioup,proc_setid,proc_taskid,sys_acct,sys_admin,sys_audit,sys_fs_import,sys_ip_config,sys_iptun_config,sys_mount,sys_nfs,sys_ppp_config,sys_resource,sys_smbNow, both sudo and non-sudo invocations are aligned; fail with the same sigsegv.
@janvorli, can we devise some other mechanism to ensure mprotect calls will not desync, in case mlock fails? Otherwise, I think a more specific error message would help the user in this case.
am11
commented
4 years ago you'll need to look at the program text in the compiled binary to see what's going on there
with objdump -D, found this information in libcoreclr.so (there is nothing in corerun executable related to .*spin.*, so my assumption was wrong about extern, global is initialized and used within libcoreclr.so):
$ objdump -D artifacts/bin/coreclr/$(uname).x64.Debug/libcoreclr.so | grep -A15 -i initializespinconst.*:
0000000000add1f4 <_Z23InitializeSpinConstantsv>:
add1f4: 55 push %rbp
add1f5: 48 89 e5 mov %rsp,%rbp
add1f8: 53 push %rbx
add1f9: 48 83 ec 08 sub $0x8,%rsp
add1fd: 48 8b 05 fc 63 92 00 mov 0x9263fc(%rip),%rax # 1403600 <_GLOBAL_OFFSET_TABLE_+0x3c8>
add204: 48 8b 00 mov (%rax),%rax
add207: 48 89 c7 mov %rax,%rdi
add20a: e8 f1 fe ff ff callq add100 <_ZNK8EEConfig19SpinInitialDurationEv>
add20f: 89 c2 mov %eax,%edx
add211: 48 8b 05 60 79 92 00 mov 0x927960(%rip),%rax # 1404b78 <_GLOBAL_OFFSET_TABLE_+0x1940>
add218: 89 10 mov %edx,(%rax)
add21a: 48 8b 05 df 63 92 00 mov 0x9263df(%rip),%rax # 1403600 <_GLOBAL_OFFSET_TABLE_+0x3c8>
add221: 48 8b 00 mov (%rax),%rax
add224: 48 89 c7 mov %rax,%rdi
add227: e8 fc fe ff ff callq add128 <_ZNK8EEConfig16SpinLimitProcCapEv>
$ objdump -D artifacts/bin/coreclr/$(uname).x64.Debug/libcoreclr.so| grep -A10 -i g_spin
000000000140d880 <g_SpinConstants>:
140d880: 32 00 xor (%rax),%al
140d882: 00 00 add %al,(%rax)
140d884: 40 9c rex pushfq
140d886: 00 00 add %al,(%rax)
140d888: 03 00 add (%rax),%eax
140d88a: 00 00 add %al,(%rax)
140d88c: 0a 00 or (%rax),%al
140d88e: 00 00 add %al,(%rax)
140d890: 00 00 add %al,(%rax)from the bt above (0xfffffc7fe864d218 ending with 218), looks like sigsegv is raised at:
add218: 89 10 mov %edx,(%rax).
uploaded some files at: https://github.com/am11/runtime/releases/tag/sunos-wip
janvorli
commented
4 years ago can we devise some other mechanism to ensure
mprotectcalls will not desync, in casemlockfails
I am not sure I understand what you are asking for. If the membarrier syscall with MEMBARRIER_CMD_PRIVATE_EXPEDITED is not supported and the mlock fails, it is a dead end. We have no way to flush process write buffers in that case and runtime would crash intermittently during / after a GC.
janvorli
commented
4 years ago If SmartOS has some other mean to do this flushing, we can obviously add that.
am11
commented
4 years ago Thank you @janvorli. I did not realize that the mlock was already in a fallback path. It makes sense to keep it as is, unless we find some alternative way on SmartOS/Illumos to flush the write buffer.
So far, I could not figure out the global address issue from text search. The variables defined in .got section should be writable. We can read this variable (e.g. printf(g_SpinConstants.dwInitialDuration); just before the assignment prints the value of 50 correctly), but value assignment (to even a constant) fails. So it is still a mystery.
am11
commented
4 years ago Tracked it down to an mprotect call: in InitGSCookie() after we set protection back to READONLY (see line 534 below), assignment g_SpinConstants.dwInitialDuration = 50 fails:
one line above (L533) the assignment works. @janvorli, does this mean we do not expect g_SpinConstants etc. to be affected by these protection changes and somehow it is happening on SmartOS but not the others? or something else?
am11
commented
4 years ago A bit more context: assignment before InitGSCookie() function call also works. It seems like after GSCookie val = (GSCookie)GetTickCount();, val refers to the region of memory that should not have global variables such as g_SpinConstants, but in case of SmartOS, it does, hence renders the global variables readonly. Perhaps, it is an indication to address this review comment:
// REVIEW: Need something better for PAL...
GSCookie val = (GSCookie)GetTickCount(); One workaround to unblock this situation is to change the protection to PAGE_READWRITE in the beginning of InitializeSpinConstants(), then change it back to PAGE_READONLY at the end, after the assignment to global variable is completed.
janvorli
commented
4 years ago It seems that this means that the GSCookie is already stored in read-write section. On other Unixes, the fact that it is marked as const results in it being placed in read only section. There are two options there:
__attribute__((section("sectionname)). But maybe there is some readonly data section already that we could add the cookie to.am11
commented
4 years ago Thank you. I will attempt method #1.
Meanwhile, I have tried to unprotect the cookie before the global variable assignment and protect it afterwards. This patch has fixed two SIGSEGV issues (another one happened in threads.cpp) and we now get a graceful 0x80004005 error (due to the missing mscorlib for Illumos): http://sprunge.us/uFPUg4. Since we have lost the old mechanism to cross-compile mscorlib for other OS on Windows (due to the deletion of build.cmd script), I am currently blocked; opened #36684.
janvorli
commented
4 years ago I think that using a System.Private.CoreLib.dll built for Linux should work, you just need to take the IL version (from the IL subfolder of the build target folder of coreclr). The regular one should work too, but you'd need to patch the machine type code to match the one expected for sunos (0x8664 ^ 0x1992 instead of 0x8664 ^ 0x7B79).
am11
commented
4 years ago Thanks, I transferred the IL version of System.Private.CoreLib.dll from Ubuntu.
In consecutive runs, the result is bit flaky:
map/virtual.cpp are failingInterlockedCompareExchange64 (backtrace shown below).0x80004005.(I added printf statements before two assertions in virtual.cpp to capture the values, therefore the line numbers in assertion messages shown below are one/two lines off)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= -276561920
18446740222383226880 <= 18446740225133506560
18446740222383882240 <= 18446740225133506560
18446740222659010560 <= 18446740222383161344
{1-fffffc7fd6389510} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.696: Expression: endAddress <= (SIZE_T)pRight->startBoundary
Abort (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= -644022272
18446740222015766528 <= 18446740225133506560
18446740222016421888 <= 18446740225133506560
18446740222659010560 <= 18446740225133506560
18446740225119621120 <= 18446740225133506560
coreclr_initialize failed - status: 0x80004005
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 63635456
18446740222723424256 <= 18446740225133572096
18446740222724079616 <= 18446740225133572096
18446740222659010560 <= 18446740222723358720
18446740225119621120 <= 18446740225133572096
18446740222724407296 <= 18446740225114767360
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 771817472
18446740223431606272 <= 18446740225133506560
18446740223432261632 <= 18446740225133506560
18446740222659010560 <= 18446740223431540736
18446740225119621120 <= 18446740225133506560
18446740223432589312 <= 18446740225114767360
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 1480065024
18446740224139853824 <= 18446740225133572096
18446740224140509184 <= 18446740225133572096
18446740222659010560 <= 18446740224139788288
18446740225119621120 <= 18446740225133572096
18446740224140836864 <= 18446740225114767360
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= -2106720256
18446740220553068544 <= 18446740225133506560
18446740220553723904 <= 18446740225133506560
18446740222659010560 <= 18446740225133506560
18446740225119621120 <= 18446740225133506560
coreclr_initialize failed - status: 0x80004005
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= -1398538240
18446740221261250560 <= 18446740225133506560
18446740221261905920 <= 18446740225133506560
18446740222659010560 <= 18446740225133506560
18446740225119621120 <= 18446740225133506560
coreclr_initialize failed - status: 0x80004005
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= -690356224
18446740221969432576 <= 18446740225133506560
18446740221970087936 <= 18446740225133506560
18446740222659010560 <= 18446740225133506560
18446740225119621120 <= 18446740225133506560
coreclr_initialize failed - status: 0x80004005
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 17825792
18446740222677614592 <= 18446740225133506560
18446740222678269952 <= 18446740225133506560
18446740222659010560 <= 18446740222677549056
18446740225119621120 <= 18446740225133506560
18446740222678597632 <= 18446740225114767360
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 726007808
18446740223385796608 <= 18446740225133572096
18446740223386451968 <= 18446740225133572096
18446740222659010560 <= 18446740223385731072
18446740225119621120 <= 18446740225133572096
18446740223386779648 <= 18446740225114767360
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 1434255360
18446740224094044160 <= 18446740225133572096
18446740224094699520 <= 18446740225133572096
18446740222659010560 <= 18446740224093978624
18446740225119621120 <= 18446740225133572096
18446740224095027200 <= 18446740225114767360
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
2042560512 >= 2142437376
{1-fffffc7fd6389510} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.2182: Expression: sizeOfAllocation >= (int32_t)((UINT_PTR)m_nextFreeAddress - (UINT_PTR)m_startAddress)
Trace/Breakpoint Trap (core dumped)Under the debugger, the sigsegv appears to be coming from InterlockedCompareExchange64:
(gdb) r
Starting program: /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
2042560512 >= 1553661952
[New LWP 2 ]
18446740224213450752 <= 18446740225133572096
18446740224214106112 <= 18446740225133572096
[New LWP 3 ]
[New LWP 4 ]
18446740222659010560 <= 18446740224213385216
18446740225119621120 <= 18446740225133572096
[New LWP 5 ]
18446740224214433792 <= 18446740225114767360
[New Thread 2 (LWP 2)]
[New Thread 3 (LWP 3)]
[New Thread 4 ]
[New Thread 5 ]
Thread 2 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0xfffffc7fd5cfff9b in InterlockedCompareExchange64 () from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
(gdb) bt
#0 0xfffffc7fd5cfff9b in InterlockedCompareExchange64 () from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
#1 0xfffffc7fd5f72ccc in _FILE* InterlockedCompareExchangeT<_FILE>(_FILE* volatile*, _FILE*, _FILE*) () from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
#2 0xfffffc7fd5f6fe75 in LogR2r (msg=0xfffffc7fd65edd30 "Ready to Run disabled - no loaded IL image", pFile=0x608010) at /home/am11/runtime/src/coreclr/src/vm/readytoruninfo.cpp:422
#3 0xfffffc7fd5f7038f in ReadyToRunInfo::Initialize (pModule=0xfffffc7fb8264020, pamTracker=0xfffffc7fffdfed20) at /home/am11/runtime/src/coreclr/src/vm/readytoruninfo.cpp:538
#4 0xfffffc7fd5dbc8fa in Module::Initialize (this=0xfffffc7fb8264020, pamTracker=0xfffffc7fffdfed20, szName=0x0) at /home/am11/runtime/src/coreclr/src/vm/ceeload.cpp:598
#5 0xfffffc7fd5dbb651 in Module::DoInit (this=0xfffffc7fb8264020, pamTracker=0xfffffc7fffdfed20, szName=0x0) at /home/am11/runtime/src/coreclr/src/vm/ceeload.cpp:176
#6 0xfffffc7fd5dbd3bd in Module::Create (pAssembly=0x608d20, moduleRef=637534208, file=0x608010, pamTracker=0xfffffc7fffdfed20) at /home/am11/runtime/src/coreclr/src/vm/ceeload.cpp:1269
#7 0xfffffc7fd5d9f4c0 in Assembly::Init (this=0x608d20, pamTracker=0xfffffc7fffdfed20, pLoaderAllocator=0x0) at /home/am11/runtime/src/coreclr/src/vm/assembly.cpp:181
#8 0xfffffc7fd5d9fb2f in Assembly::Create (pDomain=0x46ce50, pFile=0x608010, debuggerFlags=36, fIsCollectible=0, pamTracker=0xfffffc7fffdfed20, pLoaderAllocator=0x0)
at /home/am11/runtime/src/coreclr/src/vm/assembly.cpp:379
#9 0xfffffc7fd5e3f5a5 in DomainAssembly::Allocate (this=0x608b00) at /home/am11/runtime/src/coreclr/src/vm/domainfile.cpp:1479
#10 0xfffffc7fd5e3e71c in DomainFile::DoIncrementalLoad (this=0x608b00, level=FILE_LOAD_ALLOCATE) at /home/am11/runtime/src/coreclr/src/vm/domainfile.cpp:494
#11 0xfffffc7fd5d8a01a in AppDomain::TryIncrementalLoad (this=0x46ce50, pFile=0x608b00, workLevel=FILE_LOAD_ALLOCATE, lockHolder=...) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3961
#12 0xfffffc7fd5d89d6e in AppDomain::LoadDomainFile (this=0x46ce50, pLock=0x608c00, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3895
#13 0xfffffc7fd5d897f4 in AppDomain::LoadDomainAssemblyInternal (this=0x46ce50, pIdentity=0x0, pFile=0x608010, targetLevel=FILE_LOAD_POST_LOADLIBRARY)
at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3759
#14 0xfffffc7fd5d89020 in AppDomain::LoadDomainAssembly (this=0x46ce50, pSpec=0x0, pFile=0x608010, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3621
#15 0xfffffc7fd5d85c13 in SystemDomain::LoadBaseSystemClasses (this=0xfffffc7fd69178e0 <g_pSystemDomainMemory>) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:1936
#16 0xfffffc7fd5d858c2 in SystemDomain::Init (this=0xfffffc7fd69178e0 <g_pSystemDomainMemory>) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:1802
#17 0xfffffc7fd5f7af78 in EEStartupHelper () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:963
#18 0xfffffc7fd5f7b532 in <lambda(PVOID)>::operator()(PVOID) const (__closure=0xfffffc7fffdff60a, p=0x0) at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1132
#19 0xfffffc7fd5f7b68c in EEStartup () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1134
#20 0xfffffc7fd5f7a034 in EnsureEEStarted () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:320
#21 0xfffffc7fd5e217c9 in CorHost2::Start (this=0x4318a0) at /home/am11/runtime/src/coreclr/src/vm/corhost.cpp:106
#22 0xfffffc7fd5cf64be in coreclr_initialize (exePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun", appDomainFriendlyName=0x40a84f "unixcorerun", propertyCount=6,
propertyKeys=0xfffffc7fffdff940, propertyValues=0xfffffc7fffdff970, hostHandle=0xfffffc7fffdff868, domainId=0xfffffc7fffdff85c) at /home/am11/runtime/src/coreclr/src/dlls/mscoree/unixinterface.cpp:202
#23 0x0000000000408ebf in ExecuteManagedAssembly (currentExeAbsolutePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun",
clrFilesAbsolutePath=0x42cde0 "/home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug", managedAssemblyAbsolutePath=0x42cd70 "/home/am11/runtime/src/libraries/Common/tests/Data/TinyAssembly.dll",
managedAssemblyArgc=0, managedAssemblyArgv=0x0) at /home/am11/runtime/src/coreclr/src/hosts/unixcoreruncommon/coreruncommon.cpp:498
#24 0x0000000000408043 in main (argc=2, argv=0xfffffc7fffdffb68) at /home/am11/runtime/src/coreclr/src/hosts/unixcorerun/corerun.cpp:154at this point, I tried to disassemble frame 0, and that raised sigsegv and crashed gdb:
(gdb) f 0
#0 0xfffffc7fd5cfff9b in InterlockedCompareExchange64 () from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
(gdb) disassemble
Dump of assembler code for function InterlockedCompareExchange64:
0xfffffc7fd5cfff7b <+0>: push %rbp
0xfffffc7fd5cfff7c <+1>: mov %rsp,%rbp
Segmentation Fault (core dumped)The asserts verify that the VIRTUALStoreAllocationInfo has stored the entry correctly into the linked list with head pointed to by pVirtualMemory. The only thing I can imagine could go wrong is that mmap would return overlapping regions somehow and we would end up trying to store them in this list.
Could you try to print the startBoundary and memSize at the beginning of VIRTUALStoreAllocationInfo too? And can you please print them in hex (use the %p and cast them to void*). And also print the ones we are trying to compare with in hex. It is difficult for me to compare the large decimal values.
am11
commented
4 years ago Added print statement in VIRTUALStoreAllocationInfo and printed all three with %p (after the reinterpret_cast<void*>. Ran it in 1-200 loop with 0.5 seconds of delay; and majority of them are sigsegv cases, only few assertion failures in TryReserveInitialMemory, the assertion around line 695 in VerifyRightEntry is not violated in this run.
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
79bf0000 >= 51d90000
startBoundary: fffffc7feefef000, memSize: 20000
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
79bf0000 >= 7c0f0000
{1-fffffc7fd4eb9534} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.2183: Expression: sizeOfAllocation >= (int32_t)((UINT_PTR)m_nextFreeAddress - (UINT_PTR)m_startAddress)
Trace/Breakpoint Trap (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
79bf0000 >= 55430000
startBoundary: fffffc7feefef000, memSize: 20000
Segmentation Fault (core dumped)
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
79bf0000 >= 7f790000
{1-fffffc7fd4eb9534} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.2183: Expression: sizeOfAllocation >= (int32_t)((UINT_PTR)m_nextFreeAddress - (UINT_PTR)m_startAddress)
Trace/Breakpoint Trap (core dumped)Hmm, as for the sigsegv, have you tried to view the first argument to the InterlockedCompareExchange64? That one is the target memory address, so I guess that would be the one causing the crash.
janvorli
commented
4 years ago Also, regarding the other assert, can you please add printing (in hex) of the following:
if (m_startAddress == nullptr))ifam11
commented
4 years ago I have printed InterlockedCompareExchange64's Destination value and the rest. The m_startAddress right after the body of if does not get hit:
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
coreclrLoadAddress: fffffc7fd0160000
before if sizeOfAllocation: 79bf0000
randomOffset: 7a8b000
79bf0000 >= 7a90000
InterlockedCompareExchange64 Destination: FFFFFC7FD15D6998
startBoundary: fffffc7feefef000, memSize: 20000
startBoundary: fffffc7f5e000000, memSize: 10000
fffffc7f5e010000 <= fffffc7feefef000
startBoundary: fffffc7f5e010000, memSize: a0000
fffffc7f5e0b0000 <= fffffc7feefef000
startBoundary: fffffc7f36400000, memSize: 20002000
fffffc7f56402000 <= fffffc7f5e000000
startBoundary: fffffc7fede00000, memSize: 4a1000
fffffc7fee2a1000 <= fffffc7feefef000
InterlockedCompareExchange64 Destination: 00000000005FFB38
startBoundary: fffffc7f5e0b0000, memSize: 50000
fffffc7f5e100000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FD156E588
Segmentation Fault (core dumped)I would need to see the print for the case when the assert failed. As for the InterlockedCompareExchange64 case, it seems it could be caused by the cookie protection too - have you already disabled the protection changing of the cookie?
am11
commented
4 years ago Ah, sorry I missed it. Here is the assertion case:
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
coreclrLoadAddress: fffffc7fd0160000
before if sizeOfAllocation: 79bf0000
randomOffset: fffffffffed1e000
79bf0000 >= fffffffffed20000
InterlockedCompareExchange64 Destination: FFFFFC7FD15D6998
startBoundary: fffffc7feefef000, memSize: 20000
startBoundary: fffffc7f55290000, memSize: 10000
fffffc7f552a0000 <= fffffc7feefef000
startBoundary: fffffc7f552a0000, memSize: a0000
fffffc7f55340000 <= fffffc7feefef000
startBoundary: fffffc7f36400000, memSize: 20002000
fffffc7f56402000 <= fffffc7f55290000
{1-fffffc7fd10498d8} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.696: Expression: endAddress <= (SIZE_T)pRight->startBoundary
Abort (core dumped)and here is the third case where everything goes fine, and we get 0x80004005 and ICE64 does not throw (perhaps due to a lower address: 00000000005FFBC8):
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
coreclrLoadAddress: fffffc7fd0160000
before if sizeOfAllocation: 79bf0000
randomOffset: ffffffffd49bb000
79bf0000 >= ffffffffd49c0000
InterlockedCompareExchange64 Destination: FFFFFC7FD15D6998
startBoundary: fffffc7feefef000, memSize: 20000
startBoundary: fffffc7f2af30000, memSize: 10000
fffffc7f2af40000 <= fffffc7feefef000
startBoundary: fffffc7f2af40000, memSize: a0000
fffffc7f2afe0000 <= fffffc7feefef000
startBoundary: fffffc7f36400000, memSize: 20002000
fffffc7f56402000 <= fffffc7feefef000
startBoundary: fffffc7fede00000, memSize: 4a1000
fffffc7fee2a1000 <= fffffc7feefef000
InterlockedCompareExchange64 Destination: 00000000005FFBC8
coreclr_initialize failed - status: 0x80004005I will try disabling the protection in ICE64.
am11
commented
4 years ago Unprotecting memory before ICE64 and implementing a missing "get clock_id" get me to:
System.MissingMethodException: Entry point not found in assembly 'TinyAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.which is exactly what we get from Ubuntu, so happy path upto entry point is working. I will try with real hello world next.
When the MissingMethodException happens, it abnormally terminates the process with core dumped. here is the gdb bt (with printf() statements in place):
$ gdb --args artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-sun-solaris2.11".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from artifacts/bin/coreclr/SunOS.x64.Debug/corerun...Reading symbols from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun.dbg...done.
done.
(gdb) r
Starting program: /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
coreclrLoadAddress: fffffc7fcd0a0000
before if sizeOfAllocation: 79bf0000
randomOffset: 4f581000
79bf0000 >= 4f590000
[New LWP 2 ]
InterlockedCompareExchange64 Destination: FFFFFC7FCE516B18
startBoundary: fffffc7feefef000, memSize: 20000
startBoundary: fffffc7fa2a40000, memSize: 10000
fffffc7fa2a50000 <= fffffc7feefef000
startBoundary: fffffc7fa2a50000, memSize: a0000
fffffc7fa2af0000 <= fffffc7feefef000
[New LWP 3 ]
[New LWP 4 ]
startBoundary: fffffc7f33400000, memSize: 20002000
fffffc7f53402000 <= fffffc7fa2a40000
startBoundary: fffffc7fede00000, memSize: 4a1000
fffffc7fee2a1000 <= fffffc7feefef000
[New LWP 5 ]
InterlockedCompareExchange64 Destination: 00000000005FFAA8
startBoundary: fffffc7fa2af0000, memSize: 50000
fffffc7fa2b40000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FCE4AE708
startBoundary: fffffc7fa2b40000, memSize: 10000
fffffc7fa2b50000 <= fffffc7fede00000
startBoundary: fffffc7fa2b50000, memSize: 10000
fffffc7fa2b60000 <= fffffc7fede00000
startBoundary: fffffc7fa2b60000, memSize: 10000
fffffc7fa2b70000 <= fffffc7fede00000
startBoundary: fffffc7fa2b70000, memSize: 10000
fffffc7fa2b80000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FCE519260
startBoundary: fffffc7fa2b80000, memSize: 80000
fffffc7fa2c00000 <= fffffc7fede00000
startBoundary: fffffc7fa2c00000, memSize: 10000
fffffc7fa2c10000 <= fffffc7fede00000
startBoundary: fffffc7fa2c10000, memSize: 10000
fffffc7fa2c20000 <= fffffc7fede00000
startBoundary: fffffc7fa2c20000, memSize: 10000
fffffc7fa2c30000 <= fffffc7fede00000
startBoundary: fffffc7fa2c30000, memSize: 10000
fffffc7fa2c40000 <= fffffc7fede00000
startBoundary: fffffc7fa2c40000, memSize: 10000
fffffc7fa2c50000 <= fffffc7fede00000
startBoundary: fffffc7fa2c50000, memSize: 10000
fffffc7fa2c60000 <= fffffc7fede00000
startBoundary: fffffc7fa2c60000, memSize: 10000
fffffc7fa2c70000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FED4F0B58
startBoundary: fffffc7fa2c70000, memSize: 10000
fffffc7fa2c80000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FED4F8A98
InterlockedCompareExchange64 Destination: FFFFFC7FED4EE9E8
InterlockedCompareExchange64 Destination: FFFFFC7FED4F0B68
startBoundary: fffffc7fa2c80000, memSize: 10000
fffffc7fa2c90000 <= fffffc7fede00000
startBoundary: fffffc7fa2c90000, memSize: 10000
fffffc7fa2ca0000 <= fffffc7fede00000
startBoundary: fffffc7fa2ca0000, memSize: 10000
fffffc7fa2cb0000 <= fffffc7fede00000
startBoundary: fffffc7fa2cb0000, memSize: 10000
fffffc7fa2cc0000 <= fffffc7fede00000
startBoundary: fffffc7fa2cc0000, memSize: 10000
fffffc7fa2cd0000 <= fffffc7fede00000
startBoundary: fffffc7fa2cd0000, memSize: 10000
fffffc7fa2ce0000 <= fffffc7fede00000
startBoundary: fffffc7fa2ce0000, memSize: 10000
fffffc7fa2cf0000 <= fffffc7fede00000
startBoundary: fffffc7feefa0000, memSize: 10000
fffffc7feefb0000 <= fffffc7feefef000
[New LWP 6 ]
startBoundary: fffffc7fa2cf0000, memSize: 10000
fffffc7fa2d00000 <= fffffc7fede00000
startBoundary: fffffc7fa2d00000, memSize: 10000
fffffc7fa2d10000 <= fffffc7fede00000
startBoundary: fffffc7fa2d10000, memSize: 10000
fffffc7fa2d20000 <= fffffc7fede00000
startBoundary: fffffc7fa2d20000, memSize: 10000
fffffc7fa2d30000 <= fffffc7fede00000
startBoundary: fffffc7fa2d30000, memSize: 10000
fffffc7fa2d40000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FCE52B0A0
startBoundary: fffffc7fa2d40000, memSize: 10000
fffffc7fa2d50000 <= fffffc7fede00000
startBoundary: fffffc7fa2d50000, memSize: 10000
fffffc7fa2d60000 <= fffffc7fede00000
startBoundary: fffffc7fa2d60000, memSize: 10000
fffffc7fa2d70000 <= fffffc7fede00000
startBoundary: fffffc7fa2d70000, memSize: 10000
fffffc7fa2d80000 <= fffffc7fede00000
startBoundary: fffffc7fa2d80000, memSize: 10000
fffffc7fa2d90000 <= fffffc7fede00000
startBoundary: fffffc7fa2d90000, memSize: 10000
fffffc7fa2da0000 <= fffffc7fede00000
startBoundary: fffffc7fa2da0000, memSize: 10000
fffffc7fa2db0000 <= fffffc7fede00000
startBoundary: fffffc7fa2db0000, memSize: 10000
fffffc7fa2dc0000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FA2D6FDF8
InterlockedCompareExchange64 Destination: FFFFFC7FA2DB2EA8
startBoundary: fffffc7fa2dc0000, memSize: 10000
fffffc7fa2dd0000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FA2DBA8D8
InterlockedCompareExchange64 Destination: FFFFFC7FCE519368
startBoundary: fffffc7fa2dd0000, memSize: 10000
fffffc7fa2de0000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: 0000000000622DD8
InterlockedCompareExchange64 Destination: FFFFFC7FA2A44400
startBoundary: fffffc7fa2de0000, memSize: 10000
fffffc7fa2df0000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7F33402810
startBoundary: fffffc7fa2df0000, memSize: 10000
fffffc7fa2e00000 <= fffffc7fede00000
startBoundary: fffffc7fa2e00000, memSize: 10000
fffffc7fa2e10000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FA2DF74A0
startBoundary: fffffc7fa2e10000, memSize: 10000
fffffc7fa2e20000 <= fffffc7fede00000
startBoundary: fffffc7fa2e20000, memSize: 10000
fffffc7fa2e30000 <= fffffc7fede00000
startBoundary: fffffc7fa2e30000, memSize: 10000
fffffc7fa2e40000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7F33402C20
startBoundary: fffffc7fa2e40000, memSize: 10000
fffffc7fa2e50000 <= fffffc7fede00000
startBoundary: fffffc7fa2e50000, memSize: 10000
fffffc7fa2e60000 <= fffffc7fede00000
startBoundary: fffffc7fa2e60000, memSize: 10000
fffffc7fa2e70000 <= fffffc7fede00000
startBoundary: fffffc7fa2e70000, memSize: 10000
fffffc7fa2e80000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7F43402410
InterlockedCompareExchange64 Destination: FFFFFC7F43402418
startBoundary: fffffc7fa2e80000, memSize: 10000
fffffc7fa2e90000 <= fffffc7fede00000
startBoundary: fffffc7fa2e90000, memSize: 10000
fffffc7fa2ea0000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FA2D702F0
InterlockedCompareExchange64 Destination: FFFFFC7FA2D717C8
InterlockedCompareExchange64 Destination: FFFFFC7FA2DB3470
InterlockedCompareExchange64 Destination: FFFFFC7FA2E9C110
startBoundary: fffffc7fa2ea0000, memSize: 10000
fffffc7fa2eb0000 <= fffffc7fede00000
startBoundary: fffffc7fa2eb0000, memSize: 10000
fffffc7fa2ec0000 <= fffffc7fede00000
startBoundary: fffffc7fa2ec0000, memSize: 10000
fffffc7fa2ed0000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FA2D39D28
startBoundary: fffffc7fa2ed0000, memSize: 10000
fffffc7fa2ee0000 <= fffffc7fede00000
startBoundary: fffffc7fa2ee0000, memSize: 10000
fffffc7fa2ef0000 <= fffffc7fede00000
startBoundary: fffffc7fa2ef0000, memSize: 10000
fffffc7fa2f00000 <= fffffc7fede00000
startBoundary: fffffc7fa2f00000, memSize: 10000
fffffc7fa2f10000 <= fffffc7fede00000
InterlockedCompareExchange64 Destination: FFFFFC7FCE517138
InterlockedCompareExchange64 Destination: FFFFFC7FCE517130
Unhandled exception. InterlockedCompareExchange64 Destination: FFFFFC7F33408820
System.MissingMethodException: Entry point not found in assembly 'TinyAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.
[New Thread 2 (LWP 2)]
[New Thread 3 (LWP 3)]
[New Thread 4 ]
[New Thread 5 ]
[New Thread 6 (LWP 6)]
Thread 2 received signal SIGABRT, Aborted.
[Switching to Thread 1 (LWP 1)]
0xfffffc7fef25cd7a in _lwp_kill () from /lib/64/libc.so.1
(gdb) bt
#0 0xfffffc7fef25cd7a in _lwp_kill () from /lib/64/libc.so.1
#1 0xfffffc7fef253700 in thr_kill () from /lib/64/libc.so.1
#2 0xfffffc7fef1f0cee in raise () from /lib/64/libc.so.1
#3 0xfffffc7fef1cab58 in abort () from /lib/64/libc.so.1
#4 0xfffffc7fcdfe1050 in PROCAbort () at /home/am11/runtime/src/coreclr/src/pal/src/thread/process.cpp:3475
#5 0xfffffc7fcdfde14a in PROCEndProcess (hProcess=0xffffff01, uExitCode=1, bTerminateUnconditionally=1) at /home/am11/runtime/src/coreclr/src/pal/src/thread/process.cpp:1472
#6 0xfffffc7fcdfddc4d in TerminateProcess (hProcess=0xffffff01, uExitCode=1) at /home/am11/runtime/src/coreclr/src/pal/src/thread/process.cpp:1371
#7 0xfffffc7fcda53b5e in CrashDumpAndTerminateProcess (exitCode=1) at /home/am11/runtime/src/coreclr/src/vm/excep.cpp:4260
#8 0xfffffc7fcda22638 in CorHost2::ExecuteAssembly (this=0x4318a0, dwAppDomainId=1, pwzAssemblyPath=0x623b20 u"/home/am11/runtime/src/libraries/Common/tests/Data/TinyAssembly.dll", argc=0, argv=0x0,
pReturnValue=0xfffffc7fffdff858) at /home/am11/runtime/src/coreclr/src/vm/corhost.cpp:400
#9 0xfffffc7fcd8f6ae5 in coreclr_execute_assembly (hostHandle=0x4318a0, domainId=1, argc=0, argv=0x0, managedAssemblyPath=0x42cd70 "/home/am11/runtime/src/libraries/Common/tests/Data/TinyAssembly.dll",
exitCode=0xfffffc7fffdff858) at /home/am11/runtime/src/coreclr/src/dlls/mscoree/unixinterface.cpp:397
#10 0x0000000000408f39 in ExecuteManagedAssembly (currentExeAbsolutePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun",
clrFilesAbsolutePath=0x42cde0 "/home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug", managedAssemblyAbsolutePath=0x42cd70 "/home/am11/runtime/src/libraries/Common/tests/Data/TinyAssembly.dll",
managedAssemblyArgc=0, managedAssemblyArgv=0x0) at /home/am11/runtime/src/coreclr/src/hosts/unixcoreruncommon/coreruncommon.cpp:507
#11 0x0000000000408043 in main (argc=2, argv=0xfffffc7fffdffb68) at /home/am11/runtime/src/coreclr/src/hosts/unixcorerun/corerun.cpp:154Assertion cases from virtual.cpp continue to persist on executing corerun command repeatedly:
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
coreclrLoadAddress: fffffc7fcd0a0000
before if sizeOfAllocation: 79bf0000
randomOffset: ffffffffe8768000
79bf0000 >= ffffffffe8770000
InterlockedCompareExchange64 Destination: FFFFFC7FCE516B18
startBoundary: fffffc7feefdf000, memSize: 20000
startBoundary: fffffc7f3bc20000, memSize: 10000
fffffc7f3bc30000 <= fffffc7feefdf000
startBoundary: fffffc7f3bc30000, memSize: a0000
fffffc7f3bcd0000 <= fffffc7feefdf000
startBoundary: fffffc7f33400000, memSize: 20002000
fffffc7f53402000 <= fffffc7f3bc20000
{1-fffffc7fcdf89988} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.696: Expression: endAddress <= (SIZE_T)pRight->startBoundary
Abort (core dumped)Also seeing the unknown 0x80004005 cases in repeated runs:
$ artifacts/bin/coreclr/$(uname).x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
coreclrLoadAddress: fffffc7fcd0a0000
before if sizeOfAllocation: 79bf0000
randomOffset: ffffffffc933c000
79bf0000 >= ffffffffc9340000
InterlockedCompareExchange64 Destination: FFFFFC7FCE516B18
startBoundary: fffffc7feefdf000, memSize: 20000
startBoundary: fffffc7f1c7f0000, memSize: 10000
fffffc7f1c800000 <= fffffc7feefdf000
startBoundary: fffffc7f1c800000, memSize: a0000
fffffc7f1c8a0000 <= fffffc7feefdf000
startBoundary: fffffc7f33400000, memSize: 20002000
fffffc7f53402000 <= fffffc7feefdf000
startBoundary: fffffc7fede00000, memSize: 4a1000
fffffc7fee2a1000 <= fffffc7feefdf000
InterlockedCompareExchange64 Destination: 00000000005FFBC8
coreclr_initialize failed - status: 0x80004005"get clock_id"
Oracle Solaris has pthread_getcpuclockid but Illumos does not.
Addendum: The analogy of difference between these two OpenSolaris forks could be made with macOS vs. FreeBSD; both used to share same base, but not anymore (for over a decade now). Due to this and other facts such as; binary compiled on Illumos distros run on other Illumos distros but not on Oracle Solaris due to libc symbols differences, as well as syscall that are made by libc itself are different/missing -- I think we would need to create two platform deriving from (abstract) SunOS: illumos and solaris. This work is progress in a separate branch.
This is how i got clock_id in case of illumos:
--- a/src/coreclr/src/pal/src/thread/thread.cpp
+++ b/src/coreclr/src/pal/src/thread/thread.cpp
@@ -1463,13 +1463,17 @@ CorUnix::GetThreadTimesInternal(
#if HAVE_PTHREAD_GETCPUCLOCKID
if (pthread_getcpuclockid(pTargetThread->GetPThreadSelf(), &cid) != 0)
-#endif
{
ASSERT("Unable to get clock from thread\n", hThread);
SetLastError(ERROR_INTERNAL_ERROR);
pTargetThread->Unlock(pThread);
goto SetTimesToZero;
}
+#elif __sun
+ cid = CLOCK_REALTIME;
+#else
+#error "Don't know how to obtain CPU clock ID on this platform."
+#endif
struct timespec ts;
if (clock_gettime(cid, &ts) != 0)@jclulow, fyi, i read through Illumos libc sources to make the above patch (and hope i've get it right), in particular this fallback:
if (clock_id != CLOCK_REALTIME && clock_id != CLOCK_HIGHRES)
clock_id = CLOCK_REALTIME;randomOffset: fffffffffed1e000
So here is a problem. The randomOffset should be in the range of 0..64*4096. Since it is out of that range, it seems it could cause both of the asserts depending on its value. When it is negative, it causes the ExecutableMemoryAllocator to provide memory out of the range it has reserved, so the ranges it allocates can overlap with ranges allocated later by the VirtualAlloc (boils down to mmap).
Looking at how we compute the offset, it seems it could overflow if the random() function was returning values longer than 56 bits, so multiplying it by 64 that we do would make the number negative. It seems we were just lucky that on the Unixes we've seen so far, the value was in 32 bit range.
am11
commented
4 years ago When it overflows, values look like this:
random(): 1178565238
RAND_MAX: 32767
pageCount: 2301955
GenerateRandomStartOffset returns: ffffffff89350bb8happy path examples:
random(): 1090043636
RAND_MAX: 32767
pageCount: 2129056
GenerateRandomStartOffset returns: 7ee6d100random(): 1001522033
RAND_MAX: 32767
pageCount: 1956157
GenerateRandomStartOffset returns: 74989648should we round off the random() result in this case?
@@ -2286,5 +2292,7 @@ int32_t ExecutableMemoryAllocator::GenerateRandomStartOffset()
- return pageCount * GetVirtualPageSize();
+ uint32_t result = pageCount * GetVirtualPageSize();
+ return result > INT_MAX ? INT_MAX : result;With the above patch applied, still seeing the assertion failure intermittently during the repeated runs:
$ artifacts/bin/coreclr/Illumos.x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
coreclrLoadAddress: fffffc7fc9f80000
before if sizeOfAllocation: 79bf0000
randomOffset: 7d7ce000
79bf0000 >= 7d7d0000
{1-fffffc7fcae699a8} ASSERT [VIRTUAL] at /home/am11/runtime/src/coreclr/src/pal/src/map/virtual.cpp.2186: Expression: sizeOfAllocation >= (int32_t)((UINT_PTR)m_nextFreeAddress - (UINT_PTR)m_startAddress)
Trace/Breakpoint Trap (core dumped)Ah, so it seems the RAND_MAX from pal.h has leaked here somehow on sunos or RAND_MAX definition on sunos is wrong. The RAND_MAX for Unix should be 23^32-1. Generating a preprocessed version of the virtual.cpp on Linux, I can see that the method becomes:
int32_t ExecutableMemoryAllocator::GenerateRandomStartOffset()
{
int32_t pageCount;
const int32_t MaxStartPageOffset = 64;
srandom(time(__null));
pageCount = (int32_t)(MaxStartPageOffset * (int64_t)random() / (0x7fffffff));
return pageCount * GetVirtualPageSize();
}
The change you've made is not helpful, the pageCount needs to be in range 0..MaxStartPageOffset-1 (0..63).
PrivatePuffin
commented
4 years ago I experience a similair failure on FreeBSD 11.3 (both jail and host). Although it might not be related, it is interesting to see three Unix based operating systems having the same issue currently:
I don't know if the OSX error might be related, but I expect at least SmartOS and FreeBSD are.
janvorli
commented
4 years ago I experience a similair failure on FreeBSD 11.3
Can you please share what are the similarities? We've been discussing a couple of different issues here, so I am not sure which ones you are referring to.
PrivatePuffin
commented
4 years ago @janvorli I'm not too versed into this projects codestack so excuse me if i'm wrong, but all issues specify the same error:
coreclr_initialize failed - status: 0x8007ff02 and mlock issues.
edit Correction: OSX seems to be another error all together, sorry... confused the error codes.
However:
My experience with FreeBSD 11.3 is actually Failed to create CoreCLR, HRESULT: 0x8007FF02
am11
commented
4 years ago @janvorli, it looks like the value of RAND_MAX provided by the system is same:
$ sudo grep -R RAND_MAX /usr
/usr/include/iso/stdlib_iso.h:#define RAND_MAX 32767and hasn't changed in past 15 years: https://github.com/illumos/illumos-gate/blob/ee8ae3f/usr/src/head/iso/stdlib_iso.h#L86. I special cased for __sun and set it to 0x7fffffff:
--- a/src/coreclr/src/pal/inc/pal.h
+++ b/src/coreclr/src/pal/inc/pal.h
#ifndef PAL_STDCPP_COMPAT
#define RAND_MAX 0x7fff
+#elif defined(__sun)
+#undef RAND_MAX
+#define RAND_MAX 0x7fffffff
#endif // !PAL_STDCPP_COMPATNow the assertion is not violation (in 100K runs of TinyAssembly).
The other case where it is intermittently failing with coreclr_initialize failed - status: 0x80004005, I have tracked it down to AppDomain::TryIncrementalLoad:
Thread 2 hit Breakpoint 1, AppDomain::TryIncrementalLoad (this=0x46ceb0, pFile=0x608c30, workLevel=FILE_LOAD_ALLOCATE, lockHolder=...) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3988
3988 Exception *pEx = GET_EXCEPTION();
(gdb) next
3993 if (!pEx->IsTransient() && !pFile->IsLoaded())
(gdb) print pEx
$1 = (Exception *) 0xfffffc7fb5796b00 <g_OOMExceptionInstance>
(gdb) print pFile
$2 = (DomainFile *) 0x608c30
(gdb) print pFile->GetFile()->GetDebugName()
$3 = (const WCHAR *) 0x608b58 u"/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/System.Private.CoreLib.dll"
(gdb) st
Ambiguous command "st": stack, start, status, step, stepi, stepping, stop, strace.
(gdb) bt
#0 AppDomain::TryIncrementalLoad (this=0x46ceb0, pFile=0x608c30, workLevel=FILE_LOAD_ALLOCATE, lockHolder=...) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3993
#1 0xfffffc7fb4c09e16 in AppDomain::LoadDomainFile (this=0x46ceb0, pLock=0x608d30, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3895
#2 0xfffffc7fb4c0989c in AppDomain::LoadDomainAssemblyInternal (this=0x46ceb0, pIdentity=0x0, pFile=0x608130, targetLevel=FILE_LOAD_POST_LOADLIBRARY)
at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3759
#3 0xfffffc7fb4c090c8 in AppDomain::LoadDomainAssembly (this=0x46ceb0, pSpec=0x0, pFile=0x608130, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3621
#4 0xfffffc7fb4c05cbb in SystemDomain::LoadBaseSystemClasses (this=0xfffffc7fb57985e0 <g_pSystemDomainMemory>) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:1936
#5 0xfffffc7fb4c0596a in SystemDomain::Init (this=0xfffffc7fb57985e0 <g_pSystemDomainMemory>) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:1802
#6 0xfffffc7fb4dfb450 in EEStartupHelper () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:964
#7 0xfffffc7fb4dfbafc in <lambda(PVOID)>::operator()(PVOID) const (__closure=0xfffffc7fffdff54a, p=0x0) at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1135
#8 0xfffffc7fb4dfbc56 in EEStartup () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1137
#9 0xfffffc7fb4dfa403 in EnsureEEStarted () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:320
#10 0xfffffc7fb4ca1959 in CorHost2::Start (this=0x4318e0) at /home/am11/runtime/src/coreclr/src/vm/corhost.cpp:106
#11 0xfffffc7fb4b764ee in coreclr_initialize (exePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/corerun", appDomainFriendlyName=0x40a84f "unixcorerun", propertyCount=6,
propertyKeys=0xfffffc7fffdff880, propertyValues=0xfffffc7fffdff8b0, hostHandle=0xfffffc7fffdff7a8, domainId=0xfffffc7fffdff79c) at /home/am11/runtime/src/coreclr/src/dlls/mscoree/unixinterface.cpp:202
#12 0x0000000000408ebf in ExecuteManagedAssembly (currentExeAbsolutePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/corerun",
clrFilesAbsolutePath=0x42cde0 "/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug", managedAssemblyAbsolutePath=0x42cd70 "/home/am11/runtime/src/libraries/Common/tests/Data/TinyAssembly.dll",
managedAssemblyArgc=0, managedAssemblyArgv=0x0) at /home/am11/runtime/src/coreclr/src/hosts/unixcoreruncommon/coreruncommon.cpp:498
#13 0x0000000000408043 in main (argc=2, argv=0xfffffc7fffdffaa8) at /home/am11/runtime/src/coreclr/src/hosts/unixcorerun/corerun.cpp:154it raises OOM exception when loading System.Private.CoreLib.dll.
am11
commented
4 years ago better stacktrace:
Reading symbols from artifacts/bin/coreclr/Illumos.x64.Debug/corerun...Reading symbols from /home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/corerun.dbg...done.
done.
(gdb) catch throw g_OOMExceptionInstance
Catchpoint 1 (throw)
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
coreclrLoadAddress: fffffc7fb4320000
before if sizeOfAllocation: 79bf0000
randomOffset: ffffffff9125c000
79bf0000 >= ffffffff91260000
[New LWP 2 ]
InterlockedCompareExchange64 Destination: FFFFFC7FB5796D98
startBoundary: fffffc7feefef000, memSize: 20000
startBoundary: fffffc7ecb990000, memSize: 10000
fffffc7ecb9a0000 <= fffffc7feefef000
startBoundary: fffffc7ecb9a0000, memSize: a0000
fffffc7ecba40000 <= fffffc7feefef000
[New LWP 3 ]
[New LWP 4 ]
startBoundary: fffffc7fcb200000, memSize: 20002000
fffffc7feb202000 <= fffffc7feefef000
startBoundary: fffffc7fcac00000, memSize: 4a1000
fffffc7fcb0a1000 <= fffffc7fcb200000
[New LWP 5 ]
InterlockedCompareExchange64 Destination: 00000000005FFB48
not stopped at a C++ exception catchpoint
[New Thread 2 (LWP 2)]
[New Thread 3 (LWP 3)]
[New Thread 4 ]
[New Thread 5 ]
[Switching to Thread 1 (LWP 1)]
Thread 2 hit Catchpoint 1 (exception thrown), 0xfffffc7feb5162a4 in __cxa_throw () from /opt/local/gcc7/lib/amd64/libstdc++.so.6
(gdb) bt
#0 0xfffffc7feb5162a4 in __cxa_throw () from /opt/local/gcc7/lib/amd64/libstdc++.so.6
#1 0xfffffc7fb4b7a38a in ThrowOutOfMemory () at /home/am11/runtime/src/coreclr/src/utilcode/ex.cpp:1045
#2 0xfffffc7fb518fd64 in UnlockedLoaderHeap::UnlockedAllocMem (this=0xfffffc7fb5799140 <g_pSystemDomainMemory+2912>, dwSize=80,
szFile=0xfffffc7fb551e860 "/home/am11/runtime/src/coreclr/src/vm/pendingload.cpp", lineNum=43) at /home/am11/runtime/src/coreclr/src/utilcode/loaderheap.cpp:1257
#3 0xfffffc7fb4c11c9a in LoaderHeap::RealAllocMemUnsafe(unsigned long, char const*, int) () from /home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/libcoreclr.so
#4 0xfffffc7fb4c11c1c in LoaderHeap::RealAllocMem(ClrSafeInt<unsigned long>, char const*, int) () from /home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/libcoreclr.so
#5 0xfffffc7fb4ec6dc4 in PendingTypeLoadTable::Create (pHeap=0xfffffc7fb5799138 <g_pSystemDomainMemory+2904>, dwNumBuckets=8, pamTracker=0xfffffc7fffdfec60)
at /home/am11/runtime/src/coreclr/src/vm/pendingload.cpp:43
#6 0xfffffc7fb4c6c137 in ClassLoader::Init (this=0x608ea0, pamTracker=0xfffffc7fffdfec60) at /home/am11/runtime/src/coreclr/src/vm/clsload.cpp:2310
#7 0xfffffc7fb4c1f4e8 in Assembly::Init (this=0x608e20, pamTracker=0xfffffc7fffdfec60, pLoaderAllocator=0x0) at /home/am11/runtime/src/coreclr/src/vm/assembly.cpp:173
#8 0xfffffc7fb4c1fbd7 in Assembly::Create (pDomain=0x46ceb0, pFile=0x5fff70, debuggerFlags=36, fIsCollectible=0, pamTracker=0xfffffc7fffdfec60, pLoaderAllocator=0x0)
at /home/am11/runtime/src/coreclr/src/vm/assembly.cpp:379
#9 0xfffffc7fb4cbf735 in DomainAssembly::Allocate (this=0x608b80) at /home/am11/runtime/src/coreclr/src/vm/domainfile.cpp:1479
#10 0xfffffc7fb4cbe8ac in DomainFile::DoIncrementalLoad (this=0x608b80, level=FILE_LOAD_ALLOCATE) at /home/am11/runtime/src/coreclr/src/vm/domainfile.cpp:494
#11 0xfffffc7fb4c0a0c2 in AppDomain::TryIncrementalLoad (this=0x46ceb0, pFile=0x608b80, workLevel=FILE_LOAD_ALLOCATE, lockHolder=...) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3961
#12 0xfffffc7fb4c09e16 in AppDomain::LoadDomainFile (this=0x46ceb0, pLock=0x608c80, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3895
#13 0xfffffc7fb4c0989c in AppDomain::LoadDomainAssemblyInternal (this=0x46ceb0, pIdentity=0x0, pFile=0x5fff70, targetLevel=FILE_LOAD_POST_LOADLIBRARY)
at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3759
#14 0xfffffc7fb4c090c8 in AppDomain::LoadDomainAssembly (this=0x46ceb0, pSpec=0x0, pFile=0x5fff70, targetLevel=FILE_LOAD_POST_LOADLIBRARY) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:3621
#15 0xfffffc7fb4c05cbb in SystemDomain::LoadBaseSystemClasses (this=0xfffffc7fb57985e0 <g_pSystemDomainMemory>) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:1936
#16 0xfffffc7fb4c0596a in SystemDomain::Init (this=0xfffffc7fb57985e0 <g_pSystemDomainMemory>) at /home/am11/runtime/src/coreclr/src/vm/appdomain.cpp:1802
#17 0xfffffc7fb4dfb450 in EEStartupHelper () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:964
#18 0xfffffc7fb4dfbafc in <lambda(PVOID)>::operator()(PVOID) const (__closure=0xfffffc7fffdff54a, p=0x0) at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1135
#19 0xfffffc7fb4dfbc56 in EEStartup () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1137
#20 0xfffffc7fb4dfa403 in EnsureEEStarted () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:320
#21 0xfffffc7fb4ca1959 in CorHost2::Start (this=0x4318e0) at /home/am11/runtime/src/coreclr/src/vm/corhost.cpp:106
#22 0xfffffc7fb4b764ee in coreclr_initialize (exePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/corerun", appDomainFriendlyName=0x40a84f "unixcorerun", propertyCount=6,
propertyKeys=0xfffffc7fffdff880, propertyValues=0xfffffc7fffdff8b0, hostHandle=0xfffffc7fffdff7a8, domainId=0xfffffc7fffdff79c) at /home/am11/runtime/src/coreclr/src/dlls/mscoree/unixinterface.cpp:202
#23 0x0000000000408ebf in ExecuteManagedAssembly (currentExeAbsolutePath=0x42cd10 "/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug/corerun",
clrFilesAbsolutePath=0x42cde0 "/home/am11/runtime/artifacts/bin/coreclr/Illumos.x64.Debug", managedAssemblyAbsolutePath=0x42cd70 "/home/am11/runtime/src/libraries/Common/tests/Data/TinyAssembly.dll",
managedAssemblyArgc=0, managedAssemblyArgv=0x0) at /home/am11/runtime/src/coreclr/src/hosts/unixcoreruncommon/coreruncommon.cpp:498
#24 0x0000000000408043 in main (argc=2, argv=0xfffffc7fffdffaa8) at /home/am11/runtime/src/coreclr/src/hosts/unixcorerun/corerun.cpp:154@am11 we should not change the RAND_MAX value in the pal.h. That value is meant for the runtime to represent max value of the rand() function that we implement in the PAL (named PAL_rand()). The value from the pal.h should not leak into PAL implementation (and it clearly doesn't on Linux), so I wonder how come it fixed the issue for you.
Also, I've noticed there is a difference between the doc of random between linux and sunos. On SunOS, it says it generates values from 0 to 2^31-1. (https://www.unix.com/man-page/sunos/3C/random/) On Linux, it says that it generates values from 0..RAND_MAX.
Edit: So I think that on sunos, we should use 0x7FFFFFFF in the GenerateRandomStartOffset method instead of RAND_MAX.
janvorli
commented
4 years ago Btw, in your second stack trace, the randomOffset is still wrong - ffffffff9125c000.
am11
commented
4 years ago I added this value definition in an #elif __sun case for #ifndef PAL_STDCPP_COMPAT in pal.h, I thought that branch is applicable for stdcpp and it will apply to GenerateRandomStartOffset(). For some reason, assertion in virtual.cpp stopped occurring after that. I will move it to GenerateRandomStartOffset() now (maybe define a local macro OFFSET_RAND_MAX for clarity).
janvorli
commented
4 years ago @Ornias1993 the failure to mlock (HRESULT: 0x8007FF02) is a resources issue. It may be that you don't have more pages that can be "wired" pages available on your system (not sure how FreeBSD works with those, but e.g. on OSX, there is a system-wide maximum of these pages)
am11
commented
4 years ago @janvorli, thank you for clarification. With this patch there was no coreclr_initialize status found in 100K executions of TinyAssembly.dll:
+#ifdef __sun
+#define OFFSET_RAND_MAX 0x7FFFFFFF
+#else
+#define OFFSET_RAND_MAX RAND_MAX
+#endif
int32_t ExecutableMemoryAllocator::GenerateRandomStartOffset()
{
int32_t pageCount;
@@ -2284,7 +2295,7 @@ int32_t ExecutableMemoryAllocator::GenerateRandomStartOffset()
// This code is similar to what coreclr runtime does on Windows.
// It generates a random number of pages to skip between 0...MaxStartPageOffset.
srandom(time(NULL));
- pageCount = (int32_t)(MaxStartPageOffset * (int64_t)random() / RAND_MAX);
+ pageCount = (int32_t)(MaxStartPageOffset * (int64_t)random() / OFFSET_RAND_MAX);
return pageCount * GetVirtualPageSize();
}The hwapp that I built with dotnet 3.1 on macOS and then copied over to artifacts/bin/coreclr/Illumos.x64.Debug failed with System.BadImageFormatException. I will work on rootfs script later, so we can cross-compile libraries for Illumos.
Could you please clarify this part:
Force the GSCookie to get placed into a readonly section on SmartOS somehow. I've played with this concept in the past on OSX and Linux. IIRC, then I've created a new section using a linker options and added
__attribute__((section("sectionname)). But maybe there is some readonly data section already that we could add the cookie to.
I have just tested with:
objdump -s -j .rodata artifacts/bin/coreclr/Illumos.x64.Debug/libcoreclr.so | grep -i cookieit shows me these values in .rodata section on SmartOS:
106f0b0 47657456 41536967 436f6f6b 69652829 GetVASigCookie()
1084710 20564153 6967436f 6f6b6965 4f666673 VASigCookieOffs
109b130 726f6365 73734753 436f6f6b 69652829 rocessGSCookie()
109b400 50726f63 65737347 53436f6f 6b696528 ProcessGSCookie(
109b500 74475343 6f6f6b69 65507472 2829203d tGSCookiePtr() =
109b960 47657447 53436f6f 6b696550 74722829 GetGSCookiePtr()
109b970 203d3d20 28282847 53436f6f 6b696520 == (((GSCookie
11c3bd0 74725f74 2920636f 6f6b6965 29202620 tr_t) cookie) &
11de2f0 73747243 6f6f6b69 65202626 2021686e strCookie && !hn
1336080 61727368 616c436f 6f6b6965 00496964 arshalCookie.Iidand on Ubuntu:
ad3000 436f6f6b 69655074 72282920 3d3d2047 CookiePtr() == G
ad32a0 25730a00 2a476574 4753436f 6f6b6965 %s..*GetGSCookie
ad32c0 65737347 53436f6f 6b696528 29002870 essGSCookie().(p
ad33b0 4753436f 6f6b6965 50747228 29203d3d GSCookiePtr() ==
ad37d0 47657447 53436f6f 6b696550 74722829 GetGSCookiePtr()
ad37e0 203d3d20 28282847 53436f6f 6b696520 == (((GSCookie
ad3940 4753436f 6f6b6965 50747228 29002f68 GSCookiePtr()./h
b96dd0 20636f6f 6b696529 20262050 4f49534f cookie) & POISO
baa140 73747243 6f6f6b69 65202626 2021686e strCookie && !hn
c5c780 41536967 436f6f6b 6965203d 3d207046 ASigCookie == pF
d43ee0 72736861 6c436f6f 6b696500 49696450 rshalCookie.IidPAs I understood it, the issue is two-fold:
InitGSCookie, the cookie should be in read-only mode; therefore we mark it read-write explicitly.
InitGSCookie, we mark the (renewed) cookie as read-only.
InitGSCookie, all attempts to write to g_SpinConstants, dwHashCodeSeed etc. require unprotect, then write, then re-protect.g_SpinConstants and friends without unprotecting it first in, for example, InitializeSpinConstants?
# smartos
$ objdump -s -j .rodata artifacts/bin/coreclr/Illumos.x64.Debug/libcoreclr.so | grep -i spinc
1114ed0 6174696f 6e203c20 675f5370 696e436f ation < g_SpinCo
1114ef0 72537069 6e436f75 6e740000 00000000 rSpinCount......
13a0320 20203d20 25750a73 70696e63 6f756e74 = %u.spincount
13a0710 6e416e64 5370696e 436f756e 74286c70 nAndSpinCount(lp
13a0730 25702c20 64775370 696e436f 756e743d %p, dwSpinCount=
13a0760 6e416e64 5370696e 436f756e 74207265 nAndSpinCount re
13a12e0 2025700a 09537069 6e436f75 6e742009 %p..SpinCount .
13a1540 6e416e64 5370696e 436f756e 74000000 nAndSpinCount...vs.
# ubuntu
$ objdump -s -j .rodata artifacts/bin/coreclr/Linux.x64.Debug/libcoreclr.so | grep -i spinc
b1a140 5f537069 6e436f6e 7374616e 74732e64 _SpinConstants.d
b1a150 774d6f6e 69746f72 5370696e 436f756e wMonitorSpinCoun
cd8fc0 63740042 47435370 696e436f 756e7400 ct.BGCSpinCount.
d99770 2020203d 2025750a 7370696e 636f756e = %u.spincoun
d99a40 696f6e3d 25702c20 64775370 696e436f ion=%p, dwSpinCo
d99aa0 65637469 6f6e416e 64537069 6e436f75 ectionAndSpinCou
d99ac0 74696f6e 3d25702c 20647753 70696e43 tion=%p, dwSpinC
d99af0 696f6e41 6e645370 696e436f 756e7420 ionAndSpinCount
d99ef0 53656374 696f6e41 6e645370 696e436f SectionAndSpinCo
d9a360 0a095370 696e436f 756e7420 093d2025 ..SpinCount .= %The thing is that on other OSes, the GS cookie is placed in a readonly section, as it is defined as "const" in the code. The g_SpinConstants is not const, so it is in a writable section. Sections are placed on memory page boundaries / with memory page granularity, so when we change a protection of the GS cookie that's in readonly section and then change it back, we cannot influence the g_SpinConstants that are in readwrite section. However on sunos, for some reason the GS cookie is stored in a writable section and shares the memory page with possibly many other globals. So changing that page to readonly is not possible. That's why I've suggested placing the GS cookie into a readonly section. But I am not sure how that can be done on the Illumos. The attribute I've mentioned can be used to force the variable to a specific section. It might be possible to place it to the .rodata explicitly or we could create an extra section for it in the linker script. Until it is figured out, you can just stop changing the protection of the page with GS cookie on Illumos. That would fix the problem.
jclulow
commented
4 years ago Also, I've noticed there is a difference between the doc of random between linux and sunos. On SunOS, it says it generates values from 0 to 2^31-1. (https://www.unix.com/man-page/sunos/3C/random/) On Linux, it says that it generates values from 0..RAND_MAX.
Edit: So I think that on sunos, we should use 0x7FFFFFFF in the GenerateRandomStartOffset method instead of RAND_MAX.
As an aside, there isn't really one "sunos" anymore -- there is Oracle Solaris, and there is illumos. That manual page you have linked appears to be for (Oracle) Solaris 10. Our pages are up on the web as well; e.g.,
Note that we have a much better random number generator, arc4random(3C), as mentioned in the NOTES section of random(3C):
NOTES
The random() and srandom() functions are unsafe in multithreaded
applications.
Use of these functions in multithreaded applications is unsupported.
arc4random(3C) is a newer and better performing random number generator.
Use it instead.We also have an implementation of getrandom(2) if that helps.
janvorli
commented
4 years ago @am11 actually, I've just found that the GSCookie is stored in the writable .data section on Linux too. I need to figure out what's going on there.
janvorli
commented
4 years ago there isn't really one "sunos" anymore
@jclulow thank you for the clarification, I was aware of that from the previous posts in this issue, I was just using sunos as a shortcut to refer to the current port.
am11
commented
4 years ago +1, sorry for not making the clear distinction from the start. Learned by making a mistake recently, when i copied the dotnet(1) compiled on SmartOS 2020 to Solaris 11 and execution failed due to the wrong version of libc (it was explicitly looking for libc version ILLUMOS xy). After that i even copied the libc from SmartOS 2020 to Solaris 11, added to LD_LIBRARY_PATH; that failed to make syscalls (by libc during its initialization). Then, I tried the same donet(1) binary on other Illumos-based distros: OpenIndiana and OmniOS and that get executed (after installing libstdc++, libgcc etc.), without problems. (name changes in scripts will be covered by RID story listed in https://github.com/dotnet/runtime/issues/34944)
PrivatePuffin
commented
4 years ago @janvorli I understand it's a resources issue (and at least in the case of OSX a OS issue, thats why I removed it from my comment).
In my case it looks more like this kind of issue: A fake allocation issue. I forcefully freed 5GB worth of memory on my 32GB, still same issue and these 5GB where formally ZFS cache hence I just freed up 5GB worth of wired memory and still it tells me It isn't having enough wire-able memory. (and thats byond the fact that most of the remaining "consumed" memory is an ZFS cache that is auto-freed when required)
That sounds suspicieusly similar to this issue, not?
janvorli
commented
4 years ago @am11, actually, I was wrong w.r.t. the GSCookie being stored in the writable .data section on Linux too. It is in the .rodata as it should:
objdump -x artifacts/bin/coreclr/Linux.x64.Debug/libcoreclr.so | grep "s_gsCookie"
0000000000b31728 l O .rodata 0000000000000008 s_gsCookie@Ornias1993 looking at the mlock FreeBSD doc, the issue in your case could also be caused by:
[EPERM] security.bsd.unprivileged_mlock is set to 0 and the
caller is not the super-user.@janvorli Well it can be one of 4 thing:
[EPERM] security.bsd.unprivileged_mlock is set to 0 and the
caller is not the super-user.
[EINVAL] The address range given wraps around zero.
[EAGAIN] Locking the indicated range would exceed the system
limit for locked memory.
[ENOMEM] Some portion of the indicated address range is not al-
located. There was an error faulting/mapping a page.
Locking the indicated range would exceed the per-
process limit for locked memory.I'm root, I made sure the application was ran as root as best as I can, I even made sure it was owned by root and set to 777.
Also, unpriv. mlock is enabled:
security.bsd.unprivileged_mlock: 1
So I highly doubt it is a permissions issue.
I also checked both on the host and in a 11.3 iocage jail (which requires mlock access to even start). Both same issues. So [EPERM] seems unlikely.
That leaves the other 3.
Being actually out of slots to wire per-application seems unlikely. Being actually out of slots globally seems, VERY unlikely.
I've attatched a complete log of all memory settings. Is there a way I can get more verbose output out of the SDK?
janvorli
commented
4 years ago @Ornias1993 can you please run under strace and share the log (ideally via a gist)? The error code should be visible in that log.
am11
commented
4 years ago strace -f dotnet <args> equivalent on FreeBSD is also truss -f dotnet <args> (same as Illumos/Solaris). I am piping the output to sprunge to get sharable gist on the internets:
truss -f artifacts/bin/illumos-x64.Debug/corehost/dotnet helloworld.dll | \
curl -F"sprunge=<-" http://sprunge.usfor some reason the GS cookie is stored in a writable section
@janvorli, it turned out to be a gcc behavior. I built coreclr with gcc7 on Ubuntu, the GSCookie was placed in .data section:
$ src/coreclr/build-runtime.sh -skipgenerateversion -nopgooptimize -gcc
# snip
$ objdump -x artifacts/bin/coreclr/Linux.x64.Debug/libcoreclr.so | grep "s_gsCookie"
000000000107cac8 l O .data 0000000000000008 s_gsCookiewith (default) clang9, it places it in .rodata section:
$ src/coreclr/build-runtime.sh -skipgenerateversion -nopgooptimize
# snip
$ objdump -x artifacts/bin/coreclr/Linux.x64.Debug/libcoreclr.so | grep "s_gsCookie"
0000000000b312e8 l O .rodata 0000000000000008 s_gsCookieit seems like it is considered as UB, and gcc places volatile consts in data section if they are not initialized in the header.
I implemented your idea as:
--- a/src/coreclr/src/vm/vars.hpp
+++ b/src/coreclr/src/vm/vars.hpp
@@ -690,12 +690,18 @@ struct ModuleIndex
typedef DPTR(GSCookie) PTR_GSCookie;
+#ifdef _MSC_VER
+#define READONLY_DATA
+#else
+#define READONLY_DATA __attribute__((section(".rodata")))
+#endif
+
#ifndef DACCESS_COMPILE
// const is so that it gets placed in the .text section (which is read-only)
// volatile is so that accesses to it do not get optimized away because of the const
//
-extern "C" RAW_KEYWORD(volatile) const GSCookie s_gsCookie;
+extern "C" RAW_KEYWORD(volatile) READONLY_DATA const GSCookie s_gsCookie;which explicitly tells gcc (and clang) to place the cookie in .rodata section. After that, I removed all the other unprotect-write-protect patches and no OOM or sigsegv in repeated run of TinyAssembly with corerun host.
Thank you for helping with troubleshooting this issue. I will create a PR with these few fixes pertaining to coreclr initialization, and then focus on chroot on Ubuntu for rootfs work next; so we can test the actual applications with illumos-flavored dotnet libraries. There might be other problems which will come up once the managed entrypoint is called and thereafter.
ps: @jeffschwMSFT I am not sure about the question tag, as this is a real issue with current master and requires a PR to two to fix etc. :)
PrivatePuffin
commented
4 years ago @janvorli and @am11 Thanks for explaining the debug process.
I figured it out and although it isn;t related to this issue, this should be documented (also for sunOS!), There where two issues at play here:
iocage on FreeBSD <12 (so 11.3 for example here) does NOT allow mlock in jail. See: https://github.com/iocage/iocage/pull/619/files
ZFS eats up more wired memory than the system allows, leading to any and every mlock to fail:
sysctl vm.stats.vm.v_wire_count
Will be higher than:
sysctl vm.max_wired
Setting the sysctl vm.max_wired higher than the current sysctl vm.stats.vm.v_wire_count solves the issue.
I think it's worth while looking into not using mlock on FreeBSD somehow. Because I don't expect it to be fixed soon and ZFS on FreeBSD is pretty usual.
Without
sudoIt fails earlier with
coreclr_initialize failed - status: 0x8007ff02and returns false from: https://github.com/dotnet/runtime/blob/bacef403fbfce0d9e3d49368bfd36dec6dea01fc/src/coreclr/src/pal/src/thread/process.cpp#L3517-L3524 (similar situation as reported in #34793)truss -fresult: http://sprunge.us/Uwi2jZ.ulimit -a:kstat -p -m memory_cap:kstat -p -m vmem:Output collapsed
```sh vmem:1:heap:alloc 22368 vmem:1:heap:class vmem vmem:1:heap:contains 0 vmem:1:heap:contains_search 0 vmem:1:heap:crtime 0 vmem:1:heap:fail 0 vmem:1:heap:free 5922 vmem:1:heap:lookup 228 vmem:1:heap:mem_import 0 vmem:1:heap:mem_inuse 1637310464 vmem:1:heap:mem_total 2060927893504 vmem:1:heap:populate_fail 0 vmem:1:heap:populate_wait 0 vmem:1:heap:search 7792 vmem:1:heap:snaptime 7041.815844948 vmem:1:heap:vmem_source 0 vmem:1:heap:wait 0 vmem:2:vmem_metadata:alloc 63121 vmem:2:vmem_metadata:class vmem vmem:2:vmem_metadata:contains 0 vmem:2:vmem_metadata:contains_search 0 vmem:2:vmem_metadata:crtime 0 vmem:2:vmem_metadata:fail 0 vmem:2:vmem_metadata:free 37 vmem:2:vmem_metadata:lookup 42 vmem:2:vmem_metadata:mem_import 278265856 vmem:2:vmem_metadata:mem_inuse 278212608 vmem:2:vmem_metadata:mem_total 278265856 vmem:2:vmem_metadata:populate_fail 0 vmem:2:vmem_metadata:populate_wait 0 vmem:2:vmem_metadata:search 0 vmem:2:vmem_metadata:snaptime 7041.815858463 vmem:2:vmem_metadata:vmem_source 1 vmem:2:vmem_metadata:wait 0 vmem:3:vmem_seg:alloc 62998 vmem:3:vmem_seg:class vmem vmem:3:vmem_seg:contains 0 vmem:3:vmem_seg:contains_search 0 vmem:3:vmem_seg:crtime 0 vmem:3:vmem_seg:fail 0 vmem:3:vmem_seg:free 0 vmem:3:vmem_seg:lookup 0 vmem:3:vmem_seg:mem_import 258101248 vmem:3:vmem_seg:mem_inuse 258101248 vmem:3:vmem_seg:mem_total 258101248 vmem:3:vmem_seg:populate_fail 0 vmem:3:vmem_seg:populate_wait 0 vmem:3:vmem_seg:search 0 vmem:3:vmem_seg:snaptime 7041.815869908 vmem:3:vmem_seg:vmem_source 2 vmem:3:vmem_seg:wait 0 vmem:4:vmem_hash:alloc 66 vmem:4:vmem_hash:class vmem vmem:4:vmem_hash:contains 0 vmem:4:vmem_hash:contains_search 0 vmem:4:vmem_hash:crtime 0 vmem:4:vmem_hash:fail 0 vmem:4:vmem_hash:free 35 vmem:4:vmem_hash:lookup 37 vmem:4:vmem_hash:mem_import 19832832 vmem:4:vmem_hash:mem_inuse 19830784 vmem:4:vmem_hash:mem_total 19832832 vmem:4:vmem_hash:populate_fail 0 vmem:4:vmem_hash:populate_wait 0 vmem:4:vmem_hash:search 0 vmem:4:vmem_hash:snaptime 7041.815881051 vmem:4:vmem_hash:vmem_source 2 vmem:4:vmem_hash:wait 0 vmem:5:vmem_vmem:alloc 91 vmem:5:vmem_vmem:class vmem vmem:5:vmem_vmem:contains 0 vmem:5:vmem_vmem:contains_search 0 vmem:5:vmem_vmem:crtime 0 vmem:5:vmem_vmem:fail 0 vmem:5:vmem_vmem:free 13 vmem:5:vmem_vmem:lookup 5 vmem:5:vmem_vmem:mem_import 278528 vmem:5:vmem_vmem:mem_inuse 283920 vmem:5:vmem_vmem:mem_total 314928 vmem:5:vmem_vmem:populate_fail 0 vmem:5:vmem_vmem:populate_wait 0 vmem:5:vmem_vmem:search 11 vmem:5:vmem_vmem:snaptime 7041.815892208 vmem:5:vmem_vmem:vmem_source 2 vmem:5:vmem_vmem:wait 0 vmem:6:heap_core:alloc 52 vmem:6:heap_core:class vmem vmem:6:heap_core:contains 0 vmem:6:heap_core:contains_search 0 vmem:6:heap_core:crtime 0 vmem:6:heap_core:fail 0 vmem:6:heap_core:free 7 vmem:6:heap_core:lookup 4 vmem:6:heap_core:mem_import 0 vmem:6:heap_core:mem_inuse 2265088 vmem:6:heap_core:mem_total 931049472 vmem:6:heap_core:populate_fail 0 vmem:6:heap_core:populate_wait 0 vmem:6:heap_core:search 0 vmem:6:heap_core:snaptime 7041.815903496 vmem:6:heap_core:vmem_source 0 vmem:6:heap_core:wait 0 vmem:7:heaptext:alloc 155 vmem:7:heaptext:class vmem vmem:7:heaptext:contains 263 vmem:7:heaptext:contains_search 263 vmem:7:heaptext:crtime 0 vmem:7:heaptext:fail 0 vmem:7:heaptext:free 62 vmem:7:heaptext:lookup 21 vmem:7:heaptext:mem_import 0 vmem:7:heaptext:mem_inuse 7954432 vmem:7:heaptext:mem_total 67108864 vmem:7:heaptext:populate_fail 0 vmem:7:heaptext:populate_wait 0 vmem:7:heaptext:search 0 vmem:7:heaptext:snaptime 7041.815914727 vmem:7:heaptext:vmem_source 6 vmem:7:heaptext:wait 0 vmem:8:static:alloc 0 vmem:8:static:class vmem vmem:8:static:contains 0 vmem:8:static:contains_search 0 vmem:8:static:crtime 0 vmem:8:static:fail 0 vmem:8:static:free 0 vmem:8:static:lookup 0 vmem:8:static:mem_import 0 vmem:8:static:mem_inuse 0 vmem:8:static:mem_total 0 vmem:8:static:populate_fail 0 vmem:8:static:populate_wait 0 vmem:8:static:search 0 vmem:8:static:snaptime 7041.815925952 vmem:8:static:vmem_source 1 vmem:8:static:wait 0 vmem:9:static_alloc:alloc 0 vmem:9:static_alloc:class vmem vmem:9:static_alloc:contains 0 vmem:9:static_alloc:contains_search 0 vmem:9:static_alloc:crtime 0 vmem:9:static_alloc:fail 0 vmem:9:static_alloc:free 0 vmem:9:static_alloc:lookup 0 vmem:9:static_alloc:mem_import 0 vmem:9:static_alloc:mem_inuse 0 vmem:9:static_alloc:mem_total 0 vmem:9:static_alloc:populate_fail 0 vmem:9:static_alloc:populate_wait 0 vmem:9:static_alloc:search 0 vmem:9:static_alloc:snaptime 7041.815937300 vmem:9:static_alloc:vmem_source 8 vmem:9:static_alloc:wait 0 vmem:10:hat_memload:alloc 1348 vmem:10:hat_memload:class vmem vmem:10:hat_memload:contains 0 vmem:10:hat_memload:contains_search 0 vmem:10:hat_memload:crtime 0 vmem:10:hat_memload:fail 0 vmem:10:hat_memload:free 76 vmem:10:hat_memload:lookup 2 vmem:10:hat_memload:mem_import 5210112 vmem:10:hat_memload:mem_inuse 5210112 vmem:10:hat_memload:mem_total 5210112 vmem:10:hat_memload:populate_fail 0 vmem:10:hat_memload:populate_wait 0 vmem:10:hat_memload:search 0 vmem:10:hat_memload:snaptime 7041.815948590 vmem:10:hat_memload:vmem_source 1 vmem:10:hat_memload:wait 0 vmem:11:kstat:alloc 1476 vmem:11:kstat:class vmem vmem:11:kstat:contains 0 vmem:11:kstat:contains_search 0 vmem:11:kstat:crtime 0 vmem:11:kstat:fail 0 vmem:11:kstat:free 403 vmem:11:kstat:lookup 32 vmem:11:kstat:mem_import 364544 vmem:11:kstat:mem_inuse 404192 vmem:11:kstat:mem_total 430080 vmem:11:kstat:populate_fail 0 vmem:11:kstat:populate_wait 0 vmem:11:kstat:search 240 vmem:11:kstat:snaptime 7041.815960001 vmem:11:kstat:vmem_source 1 vmem:11:kstat:wait 0 vmem:12:kmem_metadata:alloc 65871 vmem:12:kmem_metadata:class vmem vmem:12:kmem_metadata:contains 0 vmem:12:kmem_metadata:contains_search 0 vmem:12:kmem_metadata:crtime 0 vmem:12:kmem_metadata:fail 0 vmem:12:kmem_metadata:free 36 vmem:12:kmem_metadata:lookup 40 vmem:12:kmem_metadata:mem_import 288489472 vmem:12:kmem_metadata:mem_inuse 288411648 vmem:12:kmem_metadata:mem_total 288489472 vmem:12:kmem_metadata:populate_fail 0 vmem:12:kmem_metadata:populate_wait 0 vmem:12:kmem_metadata:search 17 vmem:12:kmem_metadata:snaptime 7041.815980704 vmem:12:kmem_metadata:vmem_source 1 vmem:12:kmem_metadata:wait 0 vmem:13:kmem_msb:alloc 65663 vmem:13:kmem_msb:class vmem vmem:13:kmem_msb:contains 0 vmem:13:kmem_msb:contains_search 0 vmem:13:kmem_msb:crtime 0 vmem:13:kmem_msb:fail 0 vmem:13:kmem_msb:free 10 vmem:13:kmem_msb:lookup 0 vmem:13:kmem_msb:mem_import 268914688 vmem:13:kmem_msb:mem_inuse 268914688 vmem:13:kmem_msb:mem_total 268914688 vmem:13:kmem_msb:populate_fail 0 vmem:13:kmem_msb:populate_wait 0 vmem:13:kmem_msb:search 0 vmem:13:kmem_msb:snaptime 7041.815992342 vmem:13:kmem_msb:vmem_source 12 vmem:13:kmem_msb:wait 0 vmem:14:kmem_cache:alloc 559 vmem:14:kmem_cache:class vmem vmem:14:kmem_cache:contains 0 vmem:14:kmem_cache:contains_search 0 vmem:14:kmem_cache:crtime 0 vmem:14:kmem_cache:fail 0 vmem:14:kmem_cache:free 105 vmem:14:kmem_cache:lookup 0 vmem:14:kmem_cache:mem_import 471040 vmem:14:kmem_cache:mem_inuse 461264 vmem:14:kmem_cache:mem_total 471040 vmem:14:kmem_cache:populate_fail 0 vmem:14:kmem_cache:populate_wait 0 vmem:14:kmem_cache:search 428 vmem:14:kmem_cache:snaptime 7041.816052250 vmem:14:kmem_cache:vmem_source 12 vmem:14:kmem_cache:wait 0 vmem:15:kmem_hash:alloc 374 vmem:15:kmem_hash:class vmem vmem:15:kmem_hash:contains 0 vmem:15:kmem_hash:contains_search 0 vmem:15:kmem_hash:crtime 0 vmem:15:kmem_hash:fail 0 vmem:15:kmem_hash:free 117 vmem:15:kmem_hash:lookup 62 vmem:15:kmem_hash:mem_import 19025920 vmem:15:kmem_hash:mem_inuse 19006464 vmem:15:kmem_hash:mem_total 19025920 vmem:15:kmem_hash:populate_fail 0 vmem:15:kmem_hash:populate_wait 0 vmem:15:kmem_hash:search 0 vmem:15:kmem_hash:snaptime 7041.816066098 vmem:15:kmem_hash:vmem_source 12 vmem:15:kmem_hash:wait 0 vmem:16:kmem_log:alloc 9 vmem:16:kmem_log:class vmem vmem:16:kmem_log:contains 0 vmem:16:kmem_log:contains_search 0 vmem:16:kmem_log:crtime 0 vmem:16:kmem_log:fail 0 vmem:16:kmem_log:free 0 vmem:16:kmem_log:lookup 0 vmem:16:kmem_log:mem_import 348160 vmem:16:kmem_log:mem_inuse 345888 vmem:16:kmem_log:mem_total 348160 vmem:16:kmem_log:populate_fail 0 vmem:16:kmem_log:populate_wait 0 vmem:16:kmem_log:search 2 vmem:16:kmem_log:snaptime 7041.816077952 vmem:16:kmem_log:vmem_source 1 vmem:16:kmem_log:wait 0 vmem:17:kmem_firewall_va:alloc 297 vmem:17:kmem_firewall_va:class vmem vmem:17:kmem_firewall_va:contains 0 vmem:17:kmem_firewall_va:contains_search 0 vmem:17:kmem_firewall_va:crtime 0 vmem:17:kmem_firewall_va:fail 0 vmem:17:kmem_firewall_va:free 244 vmem:17:kmem_firewall_va:lookup 3 vmem:17:kmem_firewall_va:mem_import 230109184 vmem:17:kmem_firewall_va:mem_inuse 230109184 vmem:17:kmem_firewall_va:mem_total 230109184 vmem:17:kmem_firewall_va:populate_fail 0 vmem:17:kmem_firewall_va:populate_wait 0 vmem:17:kmem_firewall_va:search 0 vmem:17:kmem_firewall_va:snaptime 7041.816089715 vmem:17:kmem_firewall_va:vmem_source 1 vmem:17:kmem_firewall_va:wait 0 vmem:18:kmem_firewall:alloc 0 vmem:18:kmem_firewall:class vmem vmem:18:kmem_firewall:contains 0 vmem:18:kmem_firewall:contains_search 0 vmem:18:kmem_firewall:crtime 0 vmem:18:kmem_firewall:fail 0 vmem:18:kmem_firewall:free 0 vmem:18:kmem_firewall:lookup 0 vmem:18:kmem_firewall:mem_import 0 vmem:18:kmem_firewall:mem_inuse 0 vmem:18:kmem_firewall:mem_total 0 vmem:18:kmem_firewall:populate_fail 0 vmem:18:kmem_firewall:populate_wait 0 vmem:18:kmem_firewall:search 0 vmem:18:kmem_firewall:snaptime 7041.816101378 vmem:18:kmem_firewall:vmem_source 17 vmem:18:kmem_firewall:wait 0 vmem:20:mod_sysfile:alloc 39 vmem:20:mod_sysfile:class vmem vmem:20:mod_sysfile:contains 0 vmem:20:mod_sysfile:contains_search 0 vmem:20:mod_sysfile:crtime 0 vmem:20:mod_sysfile:fail 0 vmem:20:mod_sysfile:free 0 vmem:20:mod_sysfile:lookup 0 vmem:20:mod_sysfile:mem_import 4096 vmem:20:mod_sysfile:mem_inuse 1469 vmem:20:mod_sysfile:mem_total 4096 vmem:20:mod_sysfile:populate_fail 0 vmem:20:mod_sysfile:populate_wait 0 vmem:20:mod_sysfile:search 0 vmem:20:mod_sysfile:snaptime 7041.816113101 vmem:20:mod_sysfile:vmem_source 1 vmem:20:mod_sysfile:wait 0 vmem:21:kmem_oversize:alloc 297 vmem:21:kmem_oversize:class vmem vmem:21:kmem_oversize:contains 0 vmem:21:kmem_oversize:contains_search 0 vmem:21:kmem_oversize:crtime 0 vmem:21:kmem_oversize:fail 0 vmem:21:kmem_oversize:free 244 vmem:21:kmem_oversize:lookup 3 vmem:21:kmem_oversize:mem_import 230109184 vmem:21:kmem_oversize:mem_inuse 230077567 vmem:21:kmem_oversize:mem_total 230109184 vmem:21:kmem_oversize:populate_fail 0 vmem:21:kmem_oversize:populate_wait 0 vmem:21:kmem_oversize:search 0 vmem:21:kmem_oversize:snaptime 7041.816124911 vmem:21:kmem_oversize:vmem_source 17 vmem:21:kmem_oversize:wait 0 vmem:22:kmem_va:alloc 5813 vmem:22:kmem_va:class vmem vmem:22:kmem_va:contains 0 vmem:22:kmem_va:contains_search 0 vmem:22:kmem_va:crtime 0 vmem:22:kmem_va:fail 0 vmem:22:kmem_va:free 0 vmem:22:kmem_va:lookup 0 vmem:22:kmem_va:mem_import 799567872 vmem:22:kmem_va:mem_inuse 799567872 vmem:22:kmem_va:mem_total 799567872 vmem:22:kmem_va:populate_fail 0 vmem:22:kmem_va:populate_wait 0 vmem:22:kmem_va:search 0 vmem:22:kmem_va:snaptime 7041.816140750 vmem:22:kmem_va:vmem_source 1 vmem:22:kmem_va:wait 0 vmem:23:kmem_default:alloc 121408 vmem:23:kmem_default:class vmem vmem:23:kmem_default:contains 0 vmem:23:kmem_default:contains_search 0 vmem:23:kmem_default:crtime 0 vmem:23:kmem_default:fail 0 vmem:23:kmem_default:free 7219 vmem:23:kmem_default:lookup 248 vmem:23:kmem_default:mem_import 797302784 vmem:23:kmem_default:mem_inuse 797302784 vmem:23:kmem_default:mem_total 797302784 vmem:23:kmem_default:populate_fail 0 vmem:23:kmem_default:populate_wait 0 vmem:23:kmem_default:search 0 vmem:23:kmem_default:snaptime 7041.816152623 vmem:23:kmem_default:vmem_source 22 vmem:23:kmem_default:wait 0 vmem:24:logminor_space:alloc 86 vmem:24:logminor_space:class vmem vmem:24:logminor_space:contains 0 vmem:24:logminor_space:contains_search 0 vmem:24:logminor_space:crtime 0 vmem:24:logminor_space:fail 0 vmem:24:logminor_space:free 44 vmem:24:logminor_space:lookup 0 vmem:24:logminor_space:mem_import 0 vmem:24:logminor_space:mem_inuse 42 vmem:24:logminor_space:mem_total 262137 vmem:24:logminor_space:populate_fail 0 vmem:24:logminor_space:populate_wait 0 vmem:24:logminor_space:search 2 vmem:24:logminor_space:snaptime 7041.816179867 vmem:24:logminor_space:vmem_source 0 vmem:24:logminor_space:wait 0 vmem:25:taskq_id_arena:alloc 147 vmem:25:taskq_id_arena:class vmem vmem:25:taskq_id_arena:contains 0 vmem:25:taskq_id_arena:contains_search 0 vmem:25:taskq_id_arena:crtime 0 vmem:25:taskq_id_arena:fail 0 vmem:25:taskq_id_arena:free 86 vmem:25:taskq_id_arena:lookup 28 vmem:25:taskq_id_arena:mem_import 0 vmem:25:taskq_id_arena:mem_inuse 61 vmem:25:taskq_id_arena:mem_total 2147483647 vmem:25:taskq_id_arena:populate_fail 0 vmem:25:taskq_id_arena:populate_wait 0 vmem:25:taskq_id_arena:search 0 vmem:25:taskq_id_arena:snaptime 7041.816192472 vmem:25:taskq_id_arena:vmem_source 0 vmem:25:taskq_id_arena:wait 0 vmem:26:kmem_io_64G:alloc 4867 vmem:26:kmem_io_64G:class vmem vmem:26:kmem_io_64G:contains 0 vmem:26:kmem_io_64G:contains_search 0 vmem:26:kmem_io_64G:crtime 0 vmem:26:kmem_io_64G:fail 0 vmem:26:kmem_io_64G:free 0 vmem:26:kmem_io_64G:lookup 0 vmem:26:kmem_io_64G:mem_import 19935232 vmem:26:kmem_io_64G:mem_inuse 19935232 vmem:26:kmem_io_64G:mem_total 19935232 vmem:26:kmem_io_64G:populate_fail 0 vmem:26:kmem_io_64G:populate_wait 0 vmem:26:kmem_io_64G:search 0 vmem:26:kmem_io_64G:snaptime 7041.816204460 vmem:26:kmem_io_64G:vmem_source 1 vmem:26:kmem_io_64G:wait 0 vmem:27:kmem_io_4G:alloc 4 vmem:27:kmem_io_4G:class vmem vmem:27:kmem_io_4G:contains 0 vmem:27:kmem_io_4G:contains_search 0 vmem:27:kmem_io_4G:crtime 0 vmem:27:kmem_io_4G:fail 0 vmem:27:kmem_io_4G:free 0 vmem:27:kmem_io_4G:lookup 0 vmem:27:kmem_io_4G:mem_import 16384 vmem:27:kmem_io_4G:mem_inuse 16384 vmem:27:kmem_io_4G:mem_total 16384 vmem:27:kmem_io_4G:populate_fail 0 vmem:27:kmem_io_4G:populate_wait 0 vmem:27:kmem_io_4G:search 0 vmem:27:kmem_io_4G:snaptime 7041.816217703 vmem:27:kmem_io_4G:vmem_source 1 vmem:27:kmem_io_4G:wait 0 vmem:28:kmem_io_2G:alloc 14 vmem:28:kmem_io_2G:class vmem vmem:28:kmem_io_2G:contains 0 vmem:28:kmem_io_2G:contains_search 0 vmem:28:kmem_io_2G:crtime 0 vmem:28:kmem_io_2G:fail 0 vmem:28:kmem_io_2G:free 12 vmem:28:kmem_io_2G:lookup 0 vmem:28:kmem_io_2G:mem_import 12288 vmem:28:kmem_io_2G:mem_inuse 12288 vmem:28:kmem_io_2G:mem_total 12288 vmem:28:kmem_io_2G:populate_fail 0 vmem:28:kmem_io_2G:populate_wait 0 vmem:28:kmem_io_2G:search 0 vmem:28:kmem_io_2G:snaptime 7041.816230911 vmem:28:kmem_io_2G:vmem_source 1 vmem:28:kmem_io_2G:wait 0 vmem:29:kmem_io_16M:alloc 0 vmem:29:kmem_io_16M:class vmem vmem:29:kmem_io_16M:contains 0 vmem:29:kmem_io_16M:contains_search 0 vmem:29:kmem_io_16M:crtime 0 vmem:29:kmem_io_16M:fail 0 vmem:29:kmem_io_16M:free 0 vmem:29:kmem_io_16M:lookup 0 vmem:29:kmem_io_16M:mem_import 0 vmem:29:kmem_io_16M:mem_inuse 0 vmem:29:kmem_io_16M:mem_total 0 vmem:29:kmem_io_16M:populate_fail 0 vmem:29:kmem_io_16M:populate_wait 0 vmem:29:kmem_io_16M:search 0 vmem:29:kmem_io_16M:snaptime 7041.816244297 vmem:29:kmem_io_16M:vmem_source 1 vmem:29:kmem_io_16M:wait 0 vmem:30:id32:alloc 1 vmem:30:id32:class vmem vmem:30:id32:contains 0 vmem:30:id32:contains_search 0 vmem:30:id32:crtime 0 vmem:30:id32:fail 0 vmem:30:id32:free 0 vmem:30:id32:lookup 0 vmem:30:id32:mem_import 4096 vmem:30:id32:mem_inuse 4096 vmem:30:id32:mem_total 4096 vmem:30:id32:populate_fail 0 vmem:30:id32:populate_wait 0 vmem:30:id32:search 0 vmem:30:id32:snaptime 7041.816257721 vmem:30:id32:vmem_source 6 vmem:30:id32:wait 0 vmem:31:bp_map:alloc 172 vmem:31:bp_map:class vmem vmem:31:bp_map:contains 0 vmem:31:bp_map:contains_search 0 vmem:31:bp_map:crtime 0 vmem:31:bp_map:fail 0 vmem:31:bp_map:free 168 vmem:31:bp_map:lookup 9 vmem:31:bp_map:mem_import 524288 vmem:31:bp_map:mem_inuse 524288 vmem:31:bp_map:mem_total 524288 vmem:31:bp_map:populate_fail 0 vmem:31:bp_map:populate_wait 0 vmem:31:bp_map:search 0 vmem:31:bp_map:snaptime 7041.816271868 vmem:31:bp_map:vmem_source 1 vmem:31:bp_map:wait 0 vmem:32:kvmm_arena:alloc 0 vmem:32:kvmm_arena:class vmem vmem:32:kvmm_arena:contains 0 vmem:32:kvmm_arena:contains_search 0 vmem:32:kvmm_arena:crtime 0 vmem:32:kvmm_arena:fail 0 vmem:32:kvmm_arena:free 0 vmem:32:kvmm_arena:lookup 0 vmem:32:kvmm_arena:mem_import 0 vmem:32:kvmm_arena:mem_inuse 0 vmem:32:kvmm_arena:mem_total 96733233152 vmem:32:kvmm_arena:populate_fail 0 vmem:32:kvmm_arena:populate_wait 0 vmem:32:kvmm_arena:search 0 vmem:32:kvmm_arena:snaptime 7041.816285066 vmem:32:kvmm_arena:vmem_source 0 vmem:32:kvmm_arena:wait 0 vmem:33:zfs_file_data:alloc 69527 vmem:33:zfs_file_data:class vmem vmem:33:zfs_file_data:contains 0 vmem:33:zfs_file_data:contains_search 0 vmem:33:zfs_file_data:crtime 0 vmem:33:zfs_file_data:fail 0 vmem:33:zfs_file_data:free 0 vmem:33:zfs_file_data:lookup 0 vmem:33:zfs_file_data:mem_import 0 vmem:33:zfs_file_data:mem_inuse 9084755968 vmem:33:zfs_file_data:mem_total 36274438144 vmem:33:zfs_file_data:populate_fail 0 vmem:33:zfs_file_data:populate_wait 0 vmem:33:zfs_file_data:search 0 vmem:33:zfs_file_data:snaptime 7041.816299059 vmem:33:zfs_file_data:vmem_source 0 vmem:33:zfs_file_data:wait 0 vmem:34:zfs_file_data_buf:alloc 1933785 vmem:34:zfs_file_data_buf:class vmem vmem:34:zfs_file_data_buf:contains 0 vmem:34:zfs_file_data_buf:contains_search 0 vmem:34:zfs_file_data_buf:crtime 0 vmem:34:zfs_file_data_buf:fail 0 vmem:34:zfs_file_data_buf:free 11015 vmem:34:zfs_file_data_buf:lookup 791 vmem:34:zfs_file_data_buf:mem_import 9081225216 vmem:34:zfs_file_data_buf:mem_inuse 9081225216 vmem:34:zfs_file_data_buf:mem_total 9081225216 vmem:34:zfs_file_data_buf:populate_fail 0 vmem:34:zfs_file_data_buf:populate_wait 0 vmem:34:zfs_file_data_buf:search 0 vmem:34:zfs_file_data_buf:snaptime 7041.816311451 vmem:34:zfs_file_data_buf:vmem_source 33 vmem:34:zfs_file_data_buf:wait 0 vmem:35:device:alloc 50 vmem:35:device:class vmem vmem:35:device:contains 0 vmem:35:device:contains_search 0 vmem:35:device:crtime 0 vmem:35:device:fail 0 vmem:35:device:free 39 vmem:35:device:lookup 0 vmem:35:device:mem_import 0 vmem:35:device:mem_inuse 1228800 vmem:35:device:mem_total 1073741824 vmem:35:device:populate_fail 0 vmem:35:device:populate_wait 0 vmem:35:device:search 0 vmem:35:device:snaptime 7041.816323822 vmem:35:device:vmem_source 0 vmem:35:device:wait 0 vmem:36:segkp:alloc 4878 vmem:36:segkp:class vmem vmem:36:segkp:contains 0 vmem:36:segkp:contains_search 0 vmem:36:segkp:crtime 0 vmem:36:segkp:fail 0 vmem:36:segkp:free 3912 vmem:36:segkp:lookup 2658 vmem:36:segkp:mem_import 0 vmem:36:segkp:mem_inuse 24936448 vmem:36:segkp:mem_total 2147483648 vmem:36:segkp:populate_fail 0 vmem:36:segkp:populate_wait 0 vmem:36:segkp:search 104925 vmem:36:segkp:snaptime 7041.816337311 vmem:36:segkp:vmem_source 0 vmem:36:segkp:wait 0 vmem:37:umem_np:alloc 94 vmem:37:umem_np:class vmem vmem:37:umem_np:contains 0 vmem:37:umem_np:contains_search 0 vmem:37:umem_np:crtime 0 vmem:37:umem_np:fail 0 vmem:37:umem_np:free 87 vmem:37:umem_np:lookup 0 vmem:37:umem_np:mem_import 917504 vmem:37:umem_np:mem_inuse 917504 vmem:37:umem_np:mem_total 917504 vmem:37:umem_np:populate_fail 0 vmem:37:umem_np:populate_wait 0 vmem:37:umem_np:search 0 vmem:37:umem_np:snaptime 7041.816351432 vmem:37:umem_np:vmem_source 1 vmem:37:umem_np:wait 0 vmem:38:ksyms:alloc 264 vmem:38:ksyms:class vmem vmem:38:ksyms:contains 93 vmem:38:ksyms:contains_search 5854 vmem:38:ksyms:crtime 0 vmem:38:ksyms:fail 0 vmem:38:ksyms:free 92 vmem:38:ksyms:lookup 39 vmem:38:ksyms:mem_import 2420736 vmem:38:ksyms:mem_inuse 2145612 vmem:38:ksyms:mem_total 2420736 vmem:38:ksyms:populate_fail 0 vmem:38:ksyms:populate_wait 0 vmem:38:ksyms:search 1060 vmem:38:ksyms:snaptime 7041.816364030 vmem:38:ksyms:vmem_source 1 vmem:38:ksyms:wait 0 vmem:39:ctf:alloc 256 vmem:39:ctf:class vmem vmem:39:ctf:contains 89 vmem:39:ctf:contains_search 4588 vmem:39:ctf:crtime 0 vmem:39:ctf:fail 0 vmem:39:ctf:free 89 vmem:39:ctf:lookup 28 vmem:39:ctf:mem_import 1662976 vmem:39:ctf:mem_inuse 1404781 vmem:39:ctf:mem_total 1662976 vmem:39:ctf:populate_fail 0 vmem:39:ctf:populate_wait 0 vmem:39:ctf:search 1009 vmem:39:ctf:snaptime 7041.816376785 vmem:39:ctf:vmem_source 1 vmem:39:ctf:wait 0 vmem:40:module_text:alloc 263 vmem:40:module_text:class vmem vmem:40:module_text:contains 0 vmem:40:module_text:contains_search 0 vmem:40:module_text:crtime 0 vmem:40:module_text:fail 0 vmem:40:module_text:free 93 vmem:40:module_text:lookup 29 vmem:40:module_text:mem_import 7954432 vmem:40:module_text:mem_inuse 9799736 vmem:40:module_text:mem_total 10051584 vmem:40:module_text:populate_fail 0 vmem:40:module_text:populate_wait 0 vmem:40:module_text:search 1271 vmem:40:module_text:snaptime 7041.816390787 vmem:40:module_text:vmem_source 7 vmem:40:module_text:wait 0 vmem:41:module_data:alloc 365 vmem:41:module_data:class vmem vmem:41:module_data:contains 0 vmem:41:module_data:contains_search 0 vmem:41:module_data:crtime 0 vmem:41:module_data:fail 0 vmem:41:module_data:free 125 vmem:41:module_data:lookup 64 vmem:41:module_data:mem_import 2260992 vmem:41:module_data:mem_inuse 1176783 vmem:41:module_data:mem_total 2568192 vmem:41:module_data:populate_fail 0 vmem:41:module_data:populate_wait 0 vmem:41:module_data:search 940 vmem:41:module_data:snaptime 7041.816403702 vmem:41:module_data:vmem_source 6 vmem:41:module_data:wait 0 vmem:42:mac_minor_ids:alloc 13 vmem:42:mac_minor_ids:class vmem vmem:42:mac_minor_ids:contains 0 vmem:42:mac_minor_ids:contains_search 0 vmem:42:mac_minor_ids:crtime 0 vmem:42:mac_minor_ids:fail 0 vmem:42:mac_minor_ids:free 1 vmem:42:mac_minor_ids:lookup 0 vmem:42:mac_minor_ids:mem_import 0 vmem:42:mac_minor_ids:mem_inuse 12 vmem:42:mac_minor_ids:mem_total 130070 vmem:42:mac_minor_ids:populate_fail 0 vmem:42:mac_minor_ids:populate_wait 0 vmem:42:mac_minor_ids:search 2 vmem:42:mac_minor_ids:snaptime 7041.816419137 vmem:42:mac_minor_ids:vmem_source 0 vmem:42:mac_minor_ids:wait 0 vmem:43:namefs_inodes:alloc 1 vmem:43:namefs_inodes:class vmem vmem:43:namefs_inodes:contains 0 vmem:43:namefs_inodes:contains_search 0 vmem:43:namefs_inodes:crtime 0 vmem:43:namefs_inodes:fail 0 vmem:43:namefs_inodes:free 0 vmem:43:namefs_inodes:lookup 0 vmem:43:namefs_inodes:mem_import 0 vmem:43:namefs_inodes:mem_inuse 64 vmem:43:namefs_inodes:mem_total 65536 vmem:43:namefs_inodes:populate_fail 0 vmem:43:namefs_inodes:populate_wait 0 vmem:43:namefs_inodes:search 0 vmem:43:namefs_inodes:snaptime 7041.816433042 vmem:43:namefs_inodes:vmem_source 0 vmem:43:namefs_inodes:wait 0 vmem:44:rctl_ids:alloc 45 vmem:44:rctl_ids:class vmem vmem:44:rctl_ids:contains 0 vmem:44:rctl_ids:contains_search 0 vmem:44:rctl_ids:crtime 0 vmem:44:rctl_ids:fail 0 vmem:44:rctl_ids:free 0 vmem:44:rctl_ids:lookup 0 vmem:44:rctl_ids:mem_import 0 vmem:44:rctl_ids:mem_inuse 45 vmem:44:rctl_ids:mem_total 32767 vmem:44:rctl_ids:populate_fail 0 vmem:44:rctl_ids:populate_wait 0 vmem:44:rctl_ids:search 2 vmem:44:rctl_ids:snaptime 7041.816448729 vmem:44:rctl_ids:vmem_source 0 vmem:44:rctl_ids:wait 0 vmem:45:zoneid_space:alloc 1 vmem:45:zoneid_space:class vmem vmem:45:zoneid_space:contains 0 vmem:45:zoneid_space:contains_search 0 vmem:45:zoneid_space:crtime 0 vmem:45:zoneid_space:fail 0 vmem:45:zoneid_space:free 0 vmem:45:zoneid_space:lookup 0 vmem:45:zoneid_space:mem_import 0 vmem:45:zoneid_space:mem_inuse 1 vmem:45:zoneid_space:mem_total 8190 vmem:45:zoneid_space:populate_fail 0 vmem:45:zoneid_space:populate_wait 0 vmem:45:zoneid_space:search 2 vmem:45:zoneid_space:snaptime 7041.816461754 vmem:45:zoneid_space:vmem_source 0 vmem:45:zoneid_space:wait 0 vmem:46:taskid_space:alloc 169 vmem:46:taskid_space:class vmem vmem:46:taskid_space:contains 0 vmem:46:taskid_space:contains_search 0 vmem:46:taskid_space:crtime 0 vmem:46:taskid_space:fail 0 vmem:46:taskid_space:free 112 vmem:46:taskid_space:lookup 1 vmem:46:taskid_space:mem_import 0 vmem:46:taskid_space:mem_inuse 57 vmem:46:taskid_space:mem_total 999999 vmem:46:taskid_space:populate_fail 0 vmem:46:taskid_space:populate_wait 0 vmem:46:taskid_space:search 2 vmem:46:taskid_space:snaptime 7041.816474890 vmem:46:taskid_space:vmem_source 0 vmem:46:taskid_space:wait 0 vmem:47:pool_ids:alloc 0 vmem:47:pool_ids:class vmem vmem:47:pool_ids:contains 0 vmem:47:pool_ids:contains_search 0 vmem:47:pool_ids:crtime 0 vmem:47:pool_ids:fail 0 vmem:47:pool_ids:free 0 vmem:47:pool_ids:lookup 0 vmem:47:pool_ids:mem_import 0 vmem:47:pool_ids:mem_inuse 0 vmem:47:pool_ids:mem_total 999998 vmem:47:pool_ids:populate_fail 0 vmem:47:pool_ids:populate_wait 0 vmem:47:pool_ids:search 0 vmem:47:pool_ids:snaptime 7041.816488031 vmem:47:pool_ids:vmem_source 0 vmem:47:pool_ids:wait 0 vmem:48:contracts:alloc 181 vmem:48:contracts:class vmem vmem:48:contracts:contains 0 vmem:48:contracts:contains_search 0 vmem:48:contracts:crtime 0 vmem:48:contracts:fail 0 vmem:48:contracts:free 113 vmem:48:contracts:lookup 2 vmem:48:contracts:mem_import 0 vmem:48:contracts:mem_inuse 68 vmem:48:contracts:mem_total 2147483646 vmem:48:contracts:populate_fail 0 vmem:48:contracts:populate_wait 0 vmem:48:contracts:search 2 vmem:48:contracts:snaptime 7041.816501776 vmem:48:contracts:vmem_source 0 vmem:48:contracts:wait 0 vmem:49:ddi_periodic:alloc 0 vmem:49:ddi_periodic:class vmem vmem:49:ddi_periodic:contains 0 vmem:49:ddi_periodic:contains_search 0 vmem:49:ddi_periodic:crtime 19.026255796 vmem:49:ddi_periodic:fail 0 vmem:49:ddi_periodic:free 0 vmem:49:ddi_periodic:lookup 0 vmem:49:ddi_periodic:mem_import 0 vmem:49:ddi_periodic:mem_inuse 0 vmem:49:ddi_periodic:mem_total 1023 vmem:49:ddi_periodic:populate_fail 0 vmem:49:ddi_periodic:populate_wait 0 vmem:49:ddi_periodic:search 0 vmem:49:ddi_periodic:snaptime 7041.816517611 vmem:49:ddi_periodic:vmem_source 0 vmem:49:ddi_periodic:wait 0 vmem:50:bootfs_minors:alloc 1 vmem:50:bootfs_minors:class vmem vmem:50:bootfs_minors:contains 0 vmem:50:bootfs_minors:contains_search 0 vmem:50:bootfs_minors:crtime 19.057419741 vmem:50:bootfs_minors:fail 0 vmem:50:bootfs_minors:free 0 vmem:50:bootfs_minors:lookup 0 vmem:50:bootfs_minors:mem_import 0 vmem:50:bootfs_minors:mem_inuse 1 vmem:50:bootfs_minors:mem_total 2147483646 vmem:50:bootfs_minors:populate_fail 0 vmem:50:bootfs_minors:populate_wait 0 vmem:50:bootfs_minors:search 2 vmem:50:bootfs_minors:snaptime 7041.816542178 vmem:50:bootfs_minors:vmem_source 0 vmem:50:bootfs_minors:wait 0 vmem:51:ip_minor_arena_sa:alloc 1 vmem:51:ip_minor_arena_sa:class vmem vmem:51:ip_minor_arena_sa:contains 0 vmem:51:ip_minor_arena_sa:contains_search 0 vmem:51:ip_minor_arena_sa:crtime 19.075826158 vmem:51:ip_minor_arena_sa:fail 0 vmem:51:ip_minor_arena_sa:free 0 vmem:51:ip_minor_arena_sa:lookup 0 vmem:51:ip_minor_arena_sa:mem_import 0 vmem:51:ip_minor_arena_sa:mem_inuse 64 vmem:51:ip_minor_arena_sa:mem_total 262140 vmem:51:ip_minor_arena_sa:populate_fail 0 vmem:51:ip_minor_arena_sa:populate_wait 0 vmem:51:ip_minor_arena_sa:search 0 vmem:51:ip_minor_arena_sa:snaptime 7041.816559828 vmem:51:ip_minor_arena_sa:vmem_source 0 vmem:51:ip_minor_arena_sa:wait 0 vmem:52:ip_minor_arena_la:alloc 1 vmem:52:ip_minor_arena_la:class vmem vmem:52:ip_minor_arena_la:contains 0 vmem:52:ip_minor_arena_la:contains_search 0 vmem:52:ip_minor_arena_la:crtime 19.075837698 vmem:52:ip_minor_arena_la:fail 0 vmem:52:ip_minor_arena_la:free 0 vmem:52:ip_minor_arena_la:lookup 0 vmem:52:ip_minor_arena_la:mem_import 0 vmem:52:ip_minor_arena_la:mem_inuse 64 vmem:52:ip_minor_arena_la:mem_total 4294705152 vmem:52:ip_minor_arena_la:populate_fail 0 vmem:52:ip_minor_arena_la:populate_wait 0 vmem:52:ip_minor_arena_la:search 0 vmem:52:ip_minor_arena_la:snaptime 7041.816573339 vmem:52:ip_minor_arena_la:vmem_source 0 vmem:52:ip_minor_arena_la:wait 0 vmem:53:lib_va_32:alloc 20 vmem:53:lib_va_32:class vmem vmem:53:lib_va_32:contains 0 vmem:53:lib_va_32:contains_search 0 vmem:53:lib_va_32:crtime 20.567654148 vmem:53:lib_va_32:fail 0 vmem:53:lib_va_32:free 0 vmem:53:lib_va_32:lookup 0 vmem:53:lib_va_32:mem_import 0 vmem:53:lib_va_32:mem_inuse 8347648 vmem:53:lib_va_32:mem_total 2138767360 vmem:53:lib_va_32:populate_fail 0 vmem:53:lib_va_32:populate_wait 0 vmem:53:lib_va_32:search 20 vmem:53:lib_va_32:snaptime 7041.816610354 vmem:53:lib_va_32:vmem_source 0 vmem:53:lib_va_32:wait 0 vmem:54:lport-instances:alloc 0 vmem:54:lport-instances:class vmem vmem:54:lport-instances:contains 0 vmem:54:lport-instances:contains_search 0 vmem:54:lport-instances:crtime 20.621196542 vmem:54:lport-instances:fail 0 vmem:54:lport-instances:free 0 vmem:54:lport-instances:lookup 0 vmem:54:lport-instances:mem_import 0 vmem:54:lport-instances:mem_inuse 0 vmem:54:lport-instances:mem_total 65536 vmem:54:lport-instances:populate_fail 0 vmem:54:lport-instances:populate_wait 0 vmem:54:lport-instances:search 0 vmem:54:lport-instances:snaptime 7041.816624197 vmem:54:lport-instances:vmem_source 0 vmem:54:lport-instances:wait 0 vmem:55:rport-instances:alloc 0 vmem:55:rport-instances:class vmem vmem:55:rport-instances:contains 0 vmem:55:rport-instances:contains_search 0 vmem:55:rport-instances:crtime 20.621205920 vmem:55:rport-instances:fail 0 vmem:55:rport-instances:free 0 vmem:55:rport-instances:lookup 0 vmem:55:rport-instances:mem_import 0 vmem:55:rport-instances:mem_inuse 0 vmem:55:rport-instances:mem_total 65536 vmem:55:rport-instances:populate_fail 0 vmem:55:rport-instances:populate_wait 0 vmem:55:rport-instances:search 0 vmem:55:rport-instances:snaptime 7041.816637693 vmem:55:rport-instances:vmem_source 0 vmem:55:rport-instances:wait 0 vmem:56:tl_minor_space:alloc 826 vmem:56:tl_minor_space:class vmem vmem:56:tl_minor_space:contains 0 vmem:56:tl_minor_space:contains_search 0 vmem:56:tl_minor_space:crtime 21.252066609 vmem:56:tl_minor_space:fail 0 vmem:56:tl_minor_space:free 792 vmem:56:tl_minor_space:lookup 0 vmem:56:tl_minor_space:mem_import 0 vmem:56:tl_minor_space:mem_inuse 34 vmem:56:tl_minor_space:mem_total 262138 vmem:56:tl_minor_space:populate_fail 0 vmem:56:tl_minor_space:populate_wait 0 vmem:56:tl_minor_space:search 2 vmem:56:tl_minor_space:snaptime 7041.816679510 vmem:56:tl_minor_space:vmem_source 0 vmem:56:tl_minor_space:wait 0 vmem:57:keysock:alloc 0 vmem:57:keysock:class vmem vmem:57:keysock:contains 0 vmem:57:keysock:contains_search 0 vmem:57:keysock:crtime 21.253319123 vmem:57:keysock:fail 0 vmem:57:keysock:free 0 vmem:57:keysock:lookup 0 vmem:57:keysock:mem_import 0 vmem:57:keysock:mem_inuse 0 vmem:57:keysock:mem_total 4294967295 vmem:57:keysock:populate_fail 0 vmem:57:keysock:populate_wait 0 vmem:57:keysock:search 0 vmem:57:keysock:snaptime 7041.816695124 vmem:57:keysock:vmem_source 0 vmem:57:keysock:wait 0 vmem:58:spdsock:alloc 1 vmem:58:spdsock:class vmem vmem:58:spdsock:contains 0 vmem:58:spdsock:contains_search 0 vmem:58:spdsock:crtime 21.257343685 vmem:58:spdsock:fail 0 vmem:58:spdsock:free 0 vmem:58:spdsock:lookup 0 vmem:58:spdsock:mem_import 0 vmem:58:spdsock:mem_inuse 64 vmem:58:spdsock:mem_total 4294967295 vmem:58:spdsock:populate_fail 0 vmem:58:spdsock:populate_wait 0 vmem:58:spdsock:search 0 vmem:58:spdsock:snaptime 7041.816709123 vmem:58:spdsock:vmem_source 0 vmem:58:spdsock:wait 0 vmem:59:lofi_id:alloc 1 vmem:59:lofi_id:class vmem vmem:59:lofi_id:contains 0 vmem:59:lofi_id:contains_search 0 vmem:59:lofi_id:crtime 23.588251440 vmem:59:lofi_id:fail 0 vmem:59:lofi_id:free 0 vmem:59:lofi_id:lookup 0 vmem:59:lofi_id:mem_import 0 vmem:59:lofi_id:mem_inuse 1 vmem:59:lofi_id:mem_total 67108863 vmem:59:lofi_id:populate_fail 0 vmem:59:lofi_id:populate_wait 0 vmem:59:lofi_id:search 1 vmem:59:lofi_id:snaptime 7041.816724322 vmem:59:lofi_id:vmem_source 0 vmem:59:lofi_id:wait 0 vmem:60:lib_va_64:alloc 134 vmem:60:lib_va_64:class vmem vmem:60:lib_va_64:contains 0 vmem:60:lib_va_64:contains_search 0 vmem:60:lib_va_64:crtime 24.026199225 vmem:60:lib_va_64:fail 0 vmem:60:lib_va_64:free 0 vmem:60:lib_va_64:lookup 0 vmem:60:lib_va_64:mem_import 0 vmem:60:lib_va_64:mem_inuse 124862464 vmem:60:lib_va_64:mem_total 136888915828736 vmem:60:lib_va_64:populate_fail 0 vmem:60:lib_va_64:populate_wait 0 vmem:60:lib_va_64:search 134 vmem:60:lib_va_64:snaptime 7041.816739191 vmem:60:lib_va_64:vmem_source 0 vmem:60:lib_va_64:wait 0 vmem:61:dtrace:alloc 18550 vmem:61:dtrace:class vmem vmem:61:dtrace:contains 0 vmem:61:dtrace:contains_search 0 vmem:61:dtrace:crtime 25.188633613 vmem:61:dtrace:fail 0 vmem:61:dtrace:free 18461 vmem:61:dtrace:lookup 32 vmem:61:dtrace:mem_import 0 vmem:61:dtrace:mem_inuse 89 vmem:61:dtrace:mem_total 4294967295 vmem:61:dtrace:populate_fail 0 vmem:61:dtrace:populate_wait 0 vmem:61:dtrace:search 37010 vmem:61:dtrace:snaptime 7041.816753478 vmem:61:dtrace:vmem_source 0 vmem:61:dtrace:wait 0 vmem:62:dtrace_minor:alloc 0 vmem:62:dtrace_minor:class vmem vmem:62:dtrace_minor:contains 0 vmem:62:dtrace_minor:contains_search 0 vmem:62:dtrace_minor:crtime 25.188649726 vmem:62:dtrace_minor:fail 0 vmem:62:dtrace_minor:free 0 vmem:62:dtrace_minor:lookup 0 vmem:62:dtrace_minor:mem_import 0 vmem:62:dtrace_minor:mem_inuse 0 vmem:62:dtrace_minor:mem_total 4294967293 vmem:62:dtrace_minor:populate_fail 0 vmem:62:dtrace_minor:populate_wait 0 vmem:62:dtrace_minor:search 0 vmem:62:dtrace_minor:snaptime 7041.816767295 vmem:62:dtrace_minor:vmem_source 0 vmem:62:dtrace_minor:wait 0 vmem:68:overlay_target_minors:alloc 1 vmem:68:overlay_target_minors:class vmem vmem:68:overlay_target_minors:contains 0 vmem:68:overlay_target_minors:contains_search 0 vmem:68:overlay_target_minors:crtime 26.544120711 vmem:68:overlay_target_minors:fail 0 vmem:68:overlay_target_minors:free 0 vmem:68:overlay_target_minors:lookup 0 vmem:68:overlay_target_minors:mem_import 0 vmem:68:overlay_target_minors:mem_inuse 1 vmem:68:overlay_target_minors:mem_total 2147483646 vmem:68:overlay_target_minors:populate_fail 0 vmem:68:overlay_target_minors:populate_wait 0 vmem:68:overlay_target_minors:search 2 vmem:68:overlay_target_minors:snaptime 7041.816804778 vmem:68:overlay_target_minors:vmem_source 0 vmem:68:overlay_target_minors:wait 0 vmem:69:vnd_minors:alloc 0 vmem:69:vnd_minors:class vmem vmem:69:vnd_minors:contains 0 vmem:69:vnd_minors:contains_search 0 vmem:69:vnd_minors:crtime 26.550900013 vmem:69:vnd_minors:fail 0 vmem:69:vnd_minors:free 0 vmem:69:vnd_minors:lookup 0 vmem:69:vnd_minors:mem_import 0 vmem:69:vnd_minors:mem_inuse 0 vmem:69:vnd_minors:mem_total 2147483646 vmem:69:vnd_minors:populate_fail 0 vmem:69:vnd_minors:populate_wait 0 vmem:69:vnd_minors:search 0 vmem:69:vnd_minors:snaptime 7041.816820000 vmem:69:vnd_minors:vmem_source 0 vmem:69:vnd_minors:wait 0 vmem:70:logdmux_minor:alloc 0 vmem:70:logdmux_minor:class vmem vmem:70:logdmux_minor:contains 0 vmem:70:logdmux_minor:contains_search 0 vmem:70:logdmux_minor:crtime 26.571093664 vmem:70:logdmux_minor:fail 0 vmem:70:logdmux_minor:free 0 vmem:70:logdmux_minor:lookup 0 vmem:70:logdmux_minor:mem_import 0 vmem:70:logdmux_minor:mem_inuse 0 vmem:70:logdmux_minor:mem_total 256 vmem:70:logdmux_minor:populate_fail 0 vmem:70:logdmux_minor:populate_wait 0 vmem:70:logdmux_minor:search 0 vmem:70:logdmux_minor:snaptime 7041.816833730 vmem:70:logdmux_minor:vmem_source 0 vmem:70:logdmux_minor:wait 0 vmem:71:ptms_minor:alloc 3 vmem:71:ptms_minor:class vmem vmem:71:ptms_minor:contains 0 vmem:71:ptms_minor:contains_search 0 vmem:71:ptms_minor:crtime 26.574400673 vmem:71:ptms_minor:fail 0 vmem:71:ptms_minor:free 1 vmem:71:ptms_minor:lookup 0 vmem:71:ptms_minor:mem_import 0 vmem:71:ptms_minor:mem_inuse 2 vmem:71:ptms_minor:mem_total 16 vmem:71:ptms_minor:populate_fail 0 vmem:71:ptms_minor:populate_wait 0 vmem:71:ptms_minor:search 0 vmem:71:ptms_minor:snaptime 7041.816847406 vmem:71:ptms_minor:vmem_source 0 vmem:71:ptms_minor:wait 0 vmem:73:ipf_minor:alloc 31 vmem:73:ipf_minor:class vmem vmem:73:ipf_minor:contains 0 vmem:73:ipf_minor:contains_search 0 vmem:73:ipf_minor:crtime 26.667819089 vmem:73:ipf_minor:fail 0 vmem:73:ipf_minor:free 30 vmem:73:ipf_minor:lookup 0 vmem:73:ipf_minor:mem_import 0 vmem:73:ipf_minor:mem_inuse 1 vmem:73:ipf_minor:mem_total 4294967294 vmem:73:ipf_minor:populate_fail 0 vmem:73:ipf_minor:populate_wait 0 vmem:73:ipf_minor:search 62 vmem:73:ipf_minor:snaptime 7041.816860961 vmem:73:ipf_minor:vmem_source 0 vmem:73:ipf_minor:wait 0 vmem:74:sppptun_minor:alloc 0 vmem:74:sppptun_minor:class vmem vmem:74:sppptun_minor:contains 0 vmem:74:sppptun_minor:contains_search 0 vmem:74:sppptun_minor:crtime 26.690987998 vmem:74:sppptun_minor:fail 0 vmem:74:sppptun_minor:free 0 vmem:74:sppptun_minor:lookup 0 vmem:74:sppptun_minor:mem_import 0 vmem:74:sppptun_minor:mem_inuse 0 vmem:74:sppptun_minor:mem_total 16 vmem:74:sppptun_minor:populate_fail 0 vmem:74:sppptun_minor:populate_wait 0 vmem:74:sppptun_minor:search 0 vmem:74:sppptun_minor:snaptime 7041.816874679 vmem:74:sppptun_minor:vmem_source 0 vmem:74:sppptun_minor:wait 0 vmem:79:vmm_minors:alloc 0 vmem:79:vmm_minors:class vmem vmem:79:vmm_minors:contains 0 vmem:79:vmm_minors:contains_search 0 vmem:79:vmm_minors:crtime 26.909226220 vmem:79:vmm_minors:fail 0 vmem:79:vmm_minors:free 0 vmem:79:vmm_minors:lookup 0 vmem:79:vmm_minors:mem_import 0 vmem:79:vmm_minors:mem_inuse 0 vmem:79:vmm_minors:mem_total 262142 vmem:79:vmm_minors:populate_fail 0 vmem:79:vmm_minors:populate_wait 0 vmem:79:vmm_minors:search 0 vmem:79:vmm_minors:snaptime 7041.816892260 vmem:79:vmm_minors:vmem_source 0 vmem:79:vmm_minors:wait 0 vmem:82:devfsadm_event_channel:alloc 1 vmem:82:devfsadm_event_channel:class vmem vmem:82:devfsadm_event_channel:contains 0 vmem:82:devfsadm_event_channel:contains_search 0 vmem:82:devfsadm_event_channel:crtime 30.424926041 vmem:82:devfsadm_event_channel:fail 0 vmem:82:devfsadm_event_channel:free 0 vmem:82:devfsadm_event_channel:lookup 0 vmem:82:devfsadm_event_channel:mem_import 0 vmem:82:devfsadm_event_channel:mem_inuse 1 vmem:82:devfsadm_event_channel:mem_total 101 vmem:82:devfsadm_event_channel:populate_fail 0 vmem:82:devfsadm_event_channel:populate_wait 0 vmem:82:devfsadm_event_channel:search 2 vmem:82:devfsadm_event_channel:snaptime 7041.816915033 vmem:82:devfsadm_event_channel:vmem_source 0 vmem:82:devfsadm_event_channel:wait 0 vmem:83:devfsadm_event_channel:alloc 1 vmem:83:devfsadm_event_channel:class vmem vmem:83:devfsadm_event_channel:contains 0 vmem:83:devfsadm_event_channel:contains_search 0 vmem:83:devfsadm_event_channel:crtime 30.424966800 vmem:83:devfsadm_event_channel:fail 0 vmem:83:devfsadm_event_channel:free 0 vmem:83:devfsadm_event_channel:lookup 0 vmem:83:devfsadm_event_channel:mem_import 0 vmem:83:devfsadm_event_channel:mem_inuse 1 vmem:83:devfsadm_event_channel:mem_total 2 vmem:83:devfsadm_event_channel:populate_fail 0 vmem:83:devfsadm_event_channel:populate_wait 0 vmem:83:devfsadm_event_channel:search 2 vmem:83:devfsadm_event_channel:snaptime 7041.816928983 vmem:83:devfsadm_event_channel:vmem_source 0 vmem:83:devfsadm_event_channel:wait 0 vmem:84:syseventconfd_door:alloc 0 vmem:84:syseventconfd_door:class vmem vmem:84:syseventconfd_door:contains 0 vmem:84:syseventconfd_door:contains_search 0 vmem:84:syseventconfd_door:crtime 30.441343572 vmem:84:syseventconfd_door:fail 0 vmem:84:syseventconfd_door:free 0 vmem:84:syseventconfd_door:lookup 0 vmem:84:syseventconfd_door:mem_import 0 vmem:84:syseventconfd_door:mem_inuse 0 vmem:84:syseventconfd_door:mem_total 101 vmem:84:syseventconfd_door:populate_fail 0 vmem:84:syseventconfd_door:populate_wait 0 vmem:84:syseventconfd_door:search 0 vmem:84:syseventconfd_door:snaptime 7041.816942773 vmem:84:syseventconfd_door:vmem_source 0 vmem:84:syseventconfd_door:wait 0 vmem:85:syseventconfd_door:alloc 1 vmem:85:syseventconfd_door:class vmem vmem:85:syseventconfd_door:contains 0 vmem:85:syseventconfd_door:contains_search 0 vmem:85:syseventconfd_door:crtime 30.441360871 vmem:85:syseventconfd_door:fail 0 vmem:85:syseventconfd_door:free 0 vmem:85:syseventconfd_door:lookup 0 vmem:85:syseventconfd_door:mem_import 0 vmem:85:syseventconfd_door:mem_inuse 1 vmem:85:syseventconfd_door:mem_total 2 vmem:85:syseventconfd_door:populate_fail 0 vmem:85:syseventconfd_door:populate_wait 0 vmem:85:syseventconfd_door:search 2 vmem:85:syseventconfd_door:snaptime 7041.816956539 vmem:85:syseventconfd_door:vmem_source 0 vmem:85:syseventconfd_door:wait 0 vmem:86:syseventd_channel:alloc 2 vmem:86:syseventd_channel:class vmem vmem:86:syseventd_channel:contains 0 vmem:86:syseventd_channel:contains_search 0 vmem:86:syseventd_channel:crtime 30.444381239 vmem:86:syseventd_channel:fail 0 vmem:86:syseventd_channel:free 0 vmem:86:syseventd_channel:lookup 0 vmem:86:syseventd_channel:mem_import 0 vmem:86:syseventd_channel:mem_inuse 2 vmem:86:syseventd_channel:mem_total 101 vmem:86:syseventd_channel:populate_fail 0 vmem:86:syseventd_channel:populate_wait 0 vmem:86:syseventd_channel:search 2 vmem:86:syseventd_channel:snaptime 7041.816970462 vmem:86:syseventd_channel:vmem_source 0 vmem:86:syseventd_channel:wait 0 vmem:87:syseventd_channel:alloc 1 vmem:87:syseventd_channel:class vmem vmem:87:syseventd_channel:contains 0 vmem:87:syseventd_channel:contains_search 0 vmem:87:syseventd_channel:crtime 30.444392367 vmem:87:syseventd_channel:fail 0 vmem:87:syseventd_channel:free 0 vmem:87:syseventd_channel:lookup 0 vmem:87:syseventd_channel:mem_import 0 vmem:87:syseventd_channel:mem_inuse 1 vmem:87:syseventd_channel:mem_total 2 vmem:87:syseventd_channel:populate_fail 0 vmem:87:syseventd_channel:populate_wait 0 vmem:87:syseventd_channel:search 2 vmem:87:syseventd_channel:snaptime 7041.816984513 vmem:87:syseventd_channel:vmem_source 0 vmem:87:syseventd_channel:wait 0 vmem:88:Hex0xfffffe1fee182518_minor:alloc 0 vmem:88:Hex0xfffffe1fee182518_minor:class vmem vmem:88:Hex0xfffffe1fee182518_minor:contains 0 vmem:88:Hex0xfffffe1fee182518_minor:contains_search 0 vmem:88:Hex0xfffffe1fee182518_minor:crtime 36.762355537 vmem:88:Hex0xfffffe1fee182518_minor:fail 0 vmem:88:Hex0xfffffe1fee182518_minor:free 0 vmem:88:Hex0xfffffe1fee182518_minor:lookup 0 vmem:88:Hex0xfffffe1fee182518_minor:mem_import 0 vmem:88:Hex0xfffffe1fee182518_minor:mem_inuse 0 vmem:88:Hex0xfffffe1fee182518_minor:mem_total 4294967294 vmem:88:Hex0xfffffe1fee182518_minor:populate_fail 0 vmem:88:Hex0xfffffe1fee182518_minor:populate_wait 0 vmem:88:Hex0xfffffe1fee182518_minor:search 0 vmem:88:Hex0xfffffe1fee182518_minor:snaptime 7041.816998728 vmem:88:Hex0xfffffe1fee182518_minor:vmem_source 0 vmem:88:Hex0xfffffe1fee182518_minor:wait 0 vmem:89:Hex0xfffffe1fee182520_minor:alloc 0 vmem:89:Hex0xfffffe1fee182520_minor:class vmem vmem:89:Hex0xfffffe1fee182520_minor:contains 0 vmem:89:Hex0xfffffe1fee182520_minor:contains_search 0 vmem:89:Hex0xfffffe1fee182520_minor:crtime 36.764578965 vmem:89:Hex0xfffffe1fee182520_minor:fail 0 vmem:89:Hex0xfffffe1fee182520_minor:free 0 vmem:89:Hex0xfffffe1fee182520_minor:lookup 0 vmem:89:Hex0xfffffe1fee182520_minor:mem_import 0 vmem:89:Hex0xfffffe1fee182520_minor:mem_inuse 0 vmem:89:Hex0xfffffe1fee182520_minor:mem_total 4294967294 vmem:89:Hex0xfffffe1fee182520_minor:populate_fail 0 vmem:89:Hex0xfffffe1fee182520_minor:populate_wait 0 vmem:89:Hex0xfffffe1fee182520_minor:search 0 vmem:89:Hex0xfffffe1fee182520_minor:snaptime 7041.817072438 vmem:89:Hex0xfffffe1fee182520_minor:vmem_source 0 vmem:89:Hex0xfffffe1fee182520_minor:wait 0 vmem:90:Hex0xfffffe200d9d6518_minor:alloc 0 vmem:90:Hex0xfffffe200d9d6518_minor:class vmem vmem:90:Hex0xfffffe200d9d6518_minor:contains 0 vmem:90:Hex0xfffffe200d9d6518_minor:contains_search 0 vmem:90:Hex0xfffffe200d9d6518_minor:crtime 71.280637066 vmem:90:Hex0xfffffe200d9d6518_minor:fail 0 vmem:90:Hex0xfffffe200d9d6518_minor:free 0 vmem:90:Hex0xfffffe200d9d6518_minor:lookup 0 vmem:90:Hex0xfffffe200d9d6518_minor:mem_import 0 vmem:90:Hex0xfffffe200d9d6518_minor:mem_inuse 0 vmem:90:Hex0xfffffe200d9d6518_minor:mem_total 4294967294 vmem:90:Hex0xfffffe200d9d6518_minor:populate_fail 0 vmem:90:Hex0xfffffe200d9d6518_minor:populate_wait 0 vmem:90:Hex0xfffffe200d9d6518_minor:search 0 vmem:90:Hex0xfffffe200d9d6518_minor:snaptime 7041.817104515 vmem:90:Hex0xfffffe200d9d6518_minor:vmem_source 0 vmem:90:Hex0xfffffe200d9d6518_minor:wait 0 vmem:91:Hex0xfffffe200d9d6520_minor:alloc 0 vmem:91:Hex0xfffffe200d9d6520_minor:class vmem vmem:91:Hex0xfffffe200d9d6520_minor:contains 0 vmem:91:Hex0xfffffe200d9d6520_minor:contains_search 0 vmem:91:Hex0xfffffe200d9d6520_minor:crtime 71.282533072 vmem:91:Hex0xfffffe200d9d6520_minor:fail 0 vmem:91:Hex0xfffffe200d9d6520_minor:free 0 vmem:91:Hex0xfffffe200d9d6520_minor:lookup 0 vmem:91:Hex0xfffffe200d9d6520_minor:mem_import 0 vmem:91:Hex0xfffffe200d9d6520_minor:mem_inuse 0 vmem:91:Hex0xfffffe200d9d6520_minor:mem_total 4294967294 vmem:91:Hex0xfffffe200d9d6520_minor:populate_fail 0 vmem:91:Hex0xfffffe200d9d6520_minor:populate_wait 0 vmem:91:Hex0xfffffe200d9d6520_minor:search 0 vmem:91:Hex0xfffffe200d9d6520_minor:snaptime 7041.817120442 vmem:91:Hex0xfffffe200d9d6520_minor:vmem_source 0 vmem:91:Hex0xfffffe200d9d6520_minor:wait 0 ```ppriv $$(current process privileges):With
sudowe cross the memory bridge and fail later:
(for some reason gdb doesn't load libcoreclr.so symbols when running with corerun)
It fails in libcoreclr, where we try to assign value to a global variable defined in other binary (corerun): https://github.com/dotnet/runtime/blob/b41296a37f4bfead66db234e9d14b81fff302567/src/coreclr/src/vm/syncblk.h#L146
it happens at the assignment; after
g_pConfig->SpinInitialDuration()returns (value of 50). Before this line it successfully prints the initial value ofg_SpinConstants.dwInitialDuration, which is also 50, so the memory is not corrupt.truss -fresult: http://sprunge.us/PfoIZl.