search

rr-debugger / rr

Record and Replay Framework
http://rr-project.org/
Other
9.21k stars 587 forks source link

Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SYSCALL: munmap'; expected 106734163, got 106734145 #2417

Open jrmuizel opened 4 years ago

jrmuizel commented 4 years ago

I get the following when trying to replay a recording of Firefox on Ubuntu 19.10

FATAL /build/rr-Stwazj/rr-5.2.0/src/ReplaySession.cc:1023:check_ticks_consistency()] 
 (task 20251 (rec:18138) at time 5571)
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SYSCALL: munmap'; expected 106734163, got 106734145
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x41)[0x55cc30ebb581]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x605)[0x55cc30debdd5]
rr(_ZN2rr21EmergencyDebugOstreamD2Ev+0x122)[0x55cc30dfbe82]
rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0xaa)[0x55cc30e65f2a]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x84e)[0x55cc30e6bf1e]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0xce)[0x55cc30e828de]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x85)[0x55cc30deec25]
rr(+0x10c7a7)[0x55cc30e627a7]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x750)[0x55cc30e63310]
rr(main+0x282)[0x55cc30da6992]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f205f1bfb6b]
rr(_start+0x2a)[0x55cc30da6aaa]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:20251' /home/jrmuizel/.rr/firefox-0/mmap_hardlink_3_firefox
khuey commented 4 years ago

Can this trace be packed and uploaded somewhere?

jrmuizel commented 4 years ago

Yep: https://drive.google.com/open?id=1ZYYkykQnHXFWUB8UqXB3hE9T7sOYAhrg

jrmuizel commented 4 years ago

I ran rr replay -p 18823 to cause the problem.

khuey commented 4 years ago

Yeah I see it too. Is it just this one recording or is everything you record busted this way?

jrmuizel commented 4 years ago

I tried multiple recordings of the same ./mach reftest run. It reproduced. I don't see it when recording other things.

khuey commented 4 years ago

What m-c rev and reftest command did you use?

mstange commented 4 years ago

I'm running into the same problem now, on Jeff's machine, also when running a reftest. I'm using rr revision 298232cab234de889072bd0996815612642a8989 (current master). My mozilla-central hg revision is 98dddb2a4c4d, and I applied this patch on top of it. My mozconfig is

CC=/home/mstange/.mozbuild/clang/bin/clang
CXX=/home/mstange/.mozbuild/clang/bin/clang++
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../obj-m-debuig
mk_add_options MOZ_MAKE_FLAGS="-s -j8"
mk_add_options AUTOCLOBBER=1
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --enable-profiling
ac_add_options --enable-dump-painting
ac_add_options --with-ccache=/home/mstange/.mozbuild/sccache/sccache

I recorded with the command mach reftest ../mozilla-central/layout/reftests/bugs/1546856-1.html --setpref layout.display-list.dump-content=true --debugger=rr (executed from ~/code/obj-m-debuig/, with mach copied to ~/bin/ which is in my path, and ~/bin/rr being a symlink to ~/code/rr/obj/bin/rr).

I get the following output when running rr replay:

mstange@hawkman:~/code/rr/obj$ rr replay
GNU gdb (Ubuntu 8.3-0ubuntu1) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mstange/.local/share/rr/firefox-0/mmap_hardlink_3_firefox...
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:30178
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.30.so...
0x00007fd255812100 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) c
Continuing.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/mstange/code/obj-m-debuig/dist/bin/libxul.so.
Use `info auto-load python-scripts [REGEXP]' to list them.
### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmp5BXMDv.mozrunner/runreftest_leaks.log
### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmp5BXMDv.mozrunner/runreftest_leaks_socket_pid10626.log
[FATAL /build/rr-kyNkfr/rr-5.2.0/src/ReplaySession.cc:1023:check_ticks_consistency()]
 (task 30179 (rec:10607) at time 7745)
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SYSCALL: munmap'; expected 185226865, got 185226852
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x5a)[0x5645562f31ea]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x635)[0x56455621d665]
rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x130)[0x56455622dea0]
rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0xad)[0x564556298e8d]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x872)[0x56455629f152]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0xd0)[0x5645562b74b0]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xfde)[0x56455622004e]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x6fb)[0x56455622104b]
rr(+0x10c519)[0x564556295519]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x7d7)[0x564556296157]
rr(main+0x27a)[0x5645561d6aaa]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fb98690b1e3]
rr(_start+0x2e)[0x5645561d6bee]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:30179' /home/mstange/.local/share/rr/firefox-0/mmap_hardlink_3_firefox

The rr make check testsuite passes in its entirety.

I have uploaded the rr pack result to Google Drive at https://drive.google.com/file/d/1YeT8zVlOt_Nq72pRe19yhriTN-mhe189/view?usp=sharing .

rocallahan commented 4 years ago

If you followe the 'Launch gdb with' instructions, and then do where in gdb, what stack trace do you get?

rocallahan commented 4 years ago

Also, if you rr record -n, does that make the problem go away?

rocallahan commented 4 years ago

I guess the related question is: can you reproduce this when you record again?

mstange commented 4 years ago

I made another recording, it produces the same error. I then followed the 'Launch gdb with' instructions on that new recording's replay, and where gives me the following stack:

#0  0x0000000070000002 in  ()
#1  0x00007fdb83394a07 in _raw_syscall () at /build/rr-kyNkfr/rr-5.2.0/src/preload/raw_syscall.S:120
#2  0x00007fdb833914bf in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:222
#3  0x00007fdb83392728 in sys_xstat64 (call=<optimized out>) at ./src/preload/syscallbuf.c:2439
#4  0x00007fdb83392728 in syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2651
#5  0x00007fdb83392728 in syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2687
#6  0x00007fdb8339139a in _syscall_hook_trampoline () at /build/rr-kyNkfr/rr-5.2.0/src/preload/syscall_hook.S:282
#7  0x00007fdb833913ca in __morestack () at /build/rr-kyNkfr/rr-5.2.0/src/preload/syscall_hook.S:417
#8  0x00007fdb833913d0 in _syscall_hook_trampoline_48_3d_01_f0_ff_ff ()
    at /build/rr-kyNkfr/rr-5.2.0/src/preload/syscall_hook.S:423
#9  0x00007fdb82f02b91 in munmap () at ../sysdeps/unix/syscall-template.S:78
#10 0x00007fdb7f6a44a3 in mdb_env_close1 (env=0x7fdb82ba8d00) at lmdb/libraries/liblmdb/mdb.c:5089
#11 0x00007fdb7f6a4585 in mdb_env_close (env=<optimized out>) at lmdb/libraries/liblmdb/mdb.c:5163
#12 0x00007fdb7de0baa9 in <lmdb::environment::Environment as core::ops::drop::Drop>::drop (self=0x7ffda4fe0ac0)
    at /home/mstange/code/mozilla-central/third_party/rust/lmdb-rkv/src/environment.rs:375
#13 0x00007fdb7d8b79a6 in core::ptr::drop_in_place ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ptr/mod.rs:174
#14 0x00007fdb7d8b8284 in core::ptr::drop_in_place ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ptr/mod.rs:174
#15 0x00007fdb7bd99b69 in core::ptr::drop_in_place ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ptr/mod.rs:174
#16 0x00007fdb7bde5c3f in xulstore::statics::cache_data () at toolkit/components/xulstore/src/statics.rs:219
#17 0x00007fdb7bd9875f in <xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__static_ref_initialize ()
    at toolkit/components/xulstore/src/statics.rs:43
#18 0x00007fdb7bd9875f in core::ops::function::FnOnce::call_once ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ops/function.rs:232
#19 0x00007fdb7bd8d2b1 in lazy_static::lazy::Lazy<T>::get::{{closure}} ()
    at /home/mstange/code/mozilla-central/third_party/rust/lazy_static/src/inline_lazy.rs:31
#20 0x00007fdb7bd9cfb7 in std::sync::once::Once::call_once::{{closure}} ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/sync/once.rs:264
#21 0x00007fdb7f6501b1 in std::sync::once::Once::call_inner () at src/libstd/sync/once.rs:417
#22 0x00007fdb7bd9cbc8 in std::sync::once::Once::call_once
    (self=0x7fdb7ffafce8 <<xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__stability::LAZY+40>, f=...) at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/sync/once.rs:264
#23 0x00007fdb7bde7617 in lazy_static::lazy::Lazy<T>::get
    (self=0x7fdb7ffafcc0 <<xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__stability::LAZY>, f=0x7fdb7ffb0f70 <log::MAX_LOG_LEVEL_FILTER>)
    at /home/mstange/code/mozilla-central/third_party/rust/lazy_static/src/inline_lazy.rs:30
#24 0x00007fdb7bde7617 in <xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__stability ()
--Type <RET> for more, q to quit, c to continue without paging--c
    at <::lazy_static::__lazy_static_internal macros>:16
#25 0x00007fdb7bde7617 in <xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref (self=0x7fdb6d161a4d) at <::lazy_static::__lazy_static_internal macros>:18
#26 0x00007fdb7bde8e7f in xulstore::get_value (doc=0x7ffda4fe1e30, id=0x7ffda4fe1e48, attr=0x7ffda4fe1e60) at toolkit/components/xulstore/src/lib.rs:108
#27 0x00007fdb7bd8983c in xulstore::ffi::XULStoreService::get_value (self=0x7fdb663fc180, doc=0x7ffda4fe1e30, id=0x7ffda4fe1e48, attr=0x7ffda4fe1e60) at toolkit/components/xulstore/src/ffi.rs:88
#28 0x00007fdb7bd8b87a in xulstore::ffi::XULStoreService::GetValue (self=0x7fdb663fc180, doc=0x7ffda4fe1e30, id=0x7ffda4fe1e48, attr=0x7ffda4fe1e60, retval=0x7ffda4fe1e78) at <::xpcom::method::xpcom_method macros>:29
#29 0x00007fdb7bd8b0b6 in xulstore::ffi::XULStoreService::allocate::GetValue (this=0x7fdb663fc180, doc=0x7ffda4fe1e30, id=0x7ffda4fe1e48, attr=0x7ffda4fe1e60, _retval=0x7ffda4fe1e78) at toolkit/components/xulstore/src/ffi.rs:23
#30 0x00007fdb70b487ba in NS_InvokeByIndex () at /home/mstange/code/mozilla-central/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:106
#31 0x00007fdb723b8cbd in CallMethodHelper::Invoke() (this=0x7ffda4fe1de8) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNative.cpp:1634
#32 0x00007fdb723b0400 in CallMethodHelper::Call() (this=0x7ffda4fe1de8) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNative.cpp:1175
#33 0x00007fdb72391326 in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) (ccx=..., mode=XPCWrappedNative::CALL_METHOD) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNative.cpp:1141
#34 0x00007fdb72392ffd in XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) (cx=0x7fdb6551b000, argc=3, vp=0x7fdb6185d270) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:947
#35 0x00007fdb79ed471e in CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), js::CallReason, JS::CallArgs const&) (cx=0x7fdb6551b000, native=0x7fdb72392b80 <XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*)>, reason=js::CallReason::Call, args=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:477
#36 0x00007fdb79ebe6d1 in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) (cx=0x7fdb6551b000, args=..., construct=js::NO_CONSTRUCT, reason=js::CallReason::Call) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:569
#37 0x00007fdb79ebee91 in InternalCall(JSContext*, js::AnyInvokeArgs const&, js::CallReason) (cx=0x7fdb6551b000, args=..., reason=js::CallReason::Call) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:632
#38 0x00007fdb79ebec52 in js::CallFromStack(JSContext*, JS::CallArgs const&) (cx=0x7fdb6551b000, args=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:636
#39 0x00007fdb79eb1d5e in Interpret(JSContext*, js::RunState&) (cx=0x7fdb6551b000, state=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:3028
#40 0x00007fdb79ea6763 in js::RunScript(JSContext*, js::RunState&) (cx=0x7fdb6551b000, state=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:449
#41 0x00007fdb79ec0122 in js::ExecuteKernel(JSContext*, JS::Handle<JSScript*>, JSObject&, JS::Value const&, js::AbstractFramePtr, JS::Value*) (cx=0x7fdb6551b000, script=..., envChainArg=..., newTargetValue=..., evalInFrame=..., result=0x7ffda4fe5440) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:825
#42 0x00007fdb79f57495 in ExecuteInExtensibleLexicalEnvironment(JSContext*, JS::Handle<JSScript*>, JS::Handle<JSObject*>) (cx=0x7fdb6551b000, scriptArg=..., env=...) at /home/mstange/code/mozilla-central/js/src/builtin/Eval.cpp:497
#43 0x00007fdb79f57b08 in js::ExecuteInJSMEnvironment(JSContext*, JS::Handle<JSScript*>, JS::Handle<JSObject*>, JS::Handle<JS::StackGCVector<JSObject*, js::TempAllocPolicy> >) (cx=0x7fdb6551b000, scriptArg=..., varEnv=..., targetObj=...) at /home/mstange/code/mozilla-central/js/src/builtin/Eval.cpp:604
#44 0x00007fdb79f577b5 in js::ExecuteInJSMEnvironment(JSContext*, JS::Handle<JSScript*>, JS::Handle<JSObject*>) (cx=0x7fdb6551b000, scriptArg=..., varEnv=...) at /home/mstange/code/mozilla-central/js/src/builtin/Eval.cpp:559
#45 0x00007fdb722ba749 in mozJSComponentLoader::ObjectForLocation(ComponentLoaderInfo&, nsIFile*, JS::MutableHandle<JSObject*>, JS::MutableHandle<JSScript*>, char**, bool, JS::MutableHandle<JS::Value>) (this=0x7fdb6631e900, aInfo=..., aComponentFile=0x7fdb663f1280, aObject=..., aTableScript=..., aLocation=0x7fdb6734c6e8, aPropagateExceptions=true, aException=...) at /home/mstange/code/mozilla-central/js/xpconnect/loader/mozJSComponentLoader.cpp:927
#46 0x00007fdb722bef00 in mozJSComponentLoader::Import(JSContext*, nsTSubstring<char> const&, JS::MutableHandle<JSObject*>, JS::MutableHandle<JSObject*>, bool) (this=0x7fdb6631e900, aCx=0x7fdb6551b000, aLocation=..., aModuleGlobal=..., aModuleExports=..., aIgnoreExports=false) at /home/mstange/code/mozilla-central/js/xpconnect/loader/mozJSComponentLoader.cpp:1341
#47 0x00007fdb70a6bf93 in mozilla::xpcom::ConstructJSMComponent(nsTSubstring<char> const&, char const*, nsISupports**) (aURI=..., aConstructor=0x7fdb6b67ecf4 "BrowserGlue", aResult=0x7ffda4fe7cf8) at StaticComponents.cpp:1609
#48 0x00007fdb70a66be3 in mozilla::xpcom::CreateInstanceImpl(mozilla::xpcom::ModuleID, nsISupports*, nsID const&, void**) (aID=mozilla::xpcom::ModuleID::Anonymous005, aOuter=0x0, aIID=..., aResult=0x7ffda4feaf20) at StaticComponents.cpp:10785
#49 0x00007fdb70a4ce45 in mozilla::xpcom::StaticModule::CreateInstance(nsISupports*, nsID const&, void**) const (this=0x7fdb6c0074c0 <mozilla::xpcom::gStaticModules+10032>, aOuter=0x0, aIID=..., aResult=0x7ffda4feaf20) at StaticComponents.cpp:11362
#50 0x00007fdb70aa7f17 in (anonymous namespace)::EntryWrapper::CreateInstance(nsISupports*, nsID const&, void**) (this=0x7ffda4feaff0, aOuter=0x0, aIID=..., aResult=0x7ffda4feaf20) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManager.cpp:219
#51 0x00007fdb70aa3126 in nsComponentManagerImpl::GetServiceLocked((anonymous namespace)::MutexLock&, (anonymous namespace)::EntryWrapper&, nsID const&, void**) (this=0x7fdb67310320, aLock=..., aEntry=..., aIID=..., aResult=0x7ffda4feb0c0) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManager.cpp:1372
#52 0x00007fdb70a9e1d9 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) (this=0x7fdb67310320, aContractID=0x7ffda4feb2dc "@mozilla.org/browser/browserglue;1", aIID=..., aResult=0x7ffda4feb0c0) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManager.cpp:1559
#53 0x00007fdb70aa6799 in CallGetService(char const*, nsID const&, void**) (aContractID=0x7ffda4feb2dc "@mozilla.org/browser/browserglue;1", aIID=..., aResult=0x7ffda4feb0c0) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManagerUtils.cpp:61
#54 0x00007fdb70aa6f03 in nsGetServiceByContractIDWithError::operator()(nsID const&, void**) const (this=0x7ffda4feb1d0, aIID=..., aInstancePtr=0x7ffda4feb0c0) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManagerUtils.cpp:253
#55 0x00007fdb70929d8b in nsCOMPtr_base::assign_from_gs_contractid_with_error(nsGetServiceByContractIDWithError const&, nsID const&) (this=0x7ffda4feb1f0, aGS=..., aIID=...) at /home/mstange/code/mozilla-central/xpcom/base/nsCOMPtr.cpp:91
#56 0x00007fdb79cd0e5e in nsCOMPtr<nsISupports>::operator=(nsGetServiceByContractIDWithError const&) (this=0x7ffda4feb1f0, aRhs=...) at /home/mstange/code/obj-m-debuig/dist/include/nsCOMPtr.h:1053
#57 0x00007fdb79cc9588 in nsAppStartupNotifier::NotifyObservers(char const*) (aCategory=0x7fdb6b9dc8f4 "app-startup") at /home/mstange/code/mozilla-central/toolkit/xre/nsAppStartupNotifier.cpp:42
#58 0x00007fdb79cb962b in XREMain::XRE_mainRun() (this=0x7ffda4febc20) at /home/mstange/code/mozilla-central/toolkit/xre/nsAppRunner.cpp:4408
#59 0x00007fdb79cbafa0 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) (this=0x7ffda4febc20, argc=4, argv=0x7ffda4fecf88, aConfig=...) at /home/mstange/code/mozilla-central/toolkit/xre/nsAppRunner.cpp:4761
#60 0x00007fdb79cbb489 in XRE_main(int, char**, mozilla::BootstrapConfig const&) (argc=4, argv=0x7ffda4fecf88, aConfig=...) at /home/mstange/code/mozilla-central/toolkit/xre/nsAppRunner.cpp:4812
#61 0x00007fdb79cd1c17 in mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) (this=0x7fdb82b026c0, argc=4, argv=0x7ffda4fecf88, aConfig=...) at /home/mstange/code/mozilla-central/toolkit/xre/Bootstrap.cpp:45
#62 0x000055c580e33a5d in do_main(int, char**, char**) (argc=4, argv=0x7ffda4fecf88, envp=0x7ffda4fecfb0) at /home/mstange/code/mozilla-central/browser/app/nsBrowserApp.cpp:217
#63 0x000055c580e33485 in main(int, char**, char**) (argc=4, argv=0x7ffda4fecf88, envp=0x7ffda4fecfb0) at /home/mstange/code/mozilla-central/browser/app/nsBrowserApp.cpp:331

I'm going to try rr record -n now.

mstange commented 4 years ago

Same error with rr record -n, slightly different stack:

mstange@hawkman:~/code/obj-m-debuig$ mach reftest ../mozilla-central/layout/reftests/bugs/1546856-1.html --setpref layout.display-list.dump-content=true --debugger=rr --debugger-args="record -n"
[...]
REFTEST INFO | Application command: /usr/bin/rr record -n /home/mstange/code/obj-m-debuig/dist/bin/firefox -marionette -profile /tmp/tmp6HYGgN.mozrunner
[...]
REFTEST SUITE-END | Shutdown
mstange@hawkman:~/code/obj-m-debuig$ rr replay
GNU gdb (Ubuntu 8.3-0ubuntu1) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mstange/.local/share/rr/firefox-3/mmap_hardlink_3_firefox...
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:19248
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.30.so...
0x00007f11ed06a100 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) c
Continuing.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/mstange/code/obj-m-debuig/dist/bin/libxul.so.
Use `info auto-load python-scripts [REGEXP]' to list them.
### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmp6HYGgN.mozrunner/runreftest_leaks.log
### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmp6HYGgN.mozrunner/runreftest_leaks_socket_pid18657.log
[FATAL /build/rr-kyNkfr/rr-5.2.0/src/ReplaySession.cc:1023:check_ticks_consistency()]
 (task 19249 (rec:18639) at time 17296)
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SYSCALL: munmap'; expected 185056965, got 185056952
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x5a)[0x55fe78a261ea]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x635)[0x55fe78950665]
rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x130)[0x55fe78960ea0]
rr(_ZN2rr13ReplaySession23check_ticks_consistencyEPNS_10ReplayTaskERKNS_5EventE+0xad)[0x55fe789cbe8d]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x872)[0x55fe789d2152]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0xd0)[0x55fe789ea4b0]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xfde)[0x55fe7895304e]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x6fb)[0x55fe7895404b]
rr(+0x10c519)[0x55fe789c8519]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x7d7)[0x55fe789c9157]
rr(main+0x27a)[0x55fe78909aaa]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fcfd071f1e3]
rr(_start+0x2e)[0x55fe78909bee]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:19249' /home/mstange/.local/share/rr/firefox-3/mmap_hardlink_3_firefox
(gdb) where
#0  0x00007f11ecbaab8b in munmap () at ../sysdeps/unix/syscall-template.S:78
#1  0x00007f11e944c4a3 in mdb_env_close1 (env=0x7f11ec8a8d00) at lmdb/libraries/liblmdb/mdb.c:5089
#2  0x00007f11e944c585 in mdb_env_close (env=<optimized out>) at lmdb/libraries/liblmdb/mdb.c:5163
#3  0x00007f11e7bb3aa9 in <lmdb::environment::Environment as core::ops::drop::Drop>::drop (self=0x7ffe38590f00)
    at /home/mstange/code/mozilla-central/third_party/rust/lmdb-rkv/src/environment.rs:375
#4  0x00007f11e765f9a6 in core::ptr::drop_in_place ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ptr/mod.rs:174
#5  0x00007f11e7660284 in core::ptr::drop_in_place ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ptr/mod.rs:174
#6  0x00007f11e5b41b69 in core::ptr::drop_in_place ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ptr/mod.rs:174
#7  0x00007f11e5b8dc3f in xulstore::statics::cache_data () at toolkit/components/xulstore/src/statics.rs:219
#8  0x00007f11e5b4075f in <xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__static_ref_initialize ()
    at toolkit/components/xulstore/src/statics.rs:43
#9  0x00007f11e5b4075f in core::ops::function::FnOnce::call_once ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libcore/ops/function.rs:232
#10 0x00007f11e5b352b1 in lazy_static::lazy::Lazy<T>::get::{{closure}} ()
    at /home/mstange/code/mozilla-central/third_party/rust/lazy_static/src/inline_lazy.rs:31
#11 0x00007f11e5b44fb7 in std::sync::once::Once::call_once::{{closure}} ()
    at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/sync/once.rs:264
#12 0x00007f11e93f81b1 in std::sync::once::Once::call_inner () at src/libstd/sync/once.rs:417
#13 0x00007f11e5b44bc8 in std::sync::once::Once::call_once
    (self=0x7f11e9d57ce8 <<xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__stability::LAZY+40>, f=...) at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447/src/libstd/sync/once.rs:264
#14 0x00007f11e5b8f617 in lazy_static::lazy::Lazy<T>::get
    (self=0x7f11e9d57cc0 <<xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__stability::LAZY>, f=0x7f11e9d58f70 <log::MAX_LOG_LEVEL_FILTER>)
    at /home/mstange/code/mozilla-central/third_party/rust/lazy_static/src/inline_lazy.rs:30
#15 0x00007f11e5b8f617 in <xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref::__stability ()
    at <::lazy_static::__lazy_static_internal macros>:16
#16 0x00007f11e5b8f617 in <xulstore::statics::DATA_CACHE as core::ops::deref::Deref>::deref (self=0x7f11d6f09a4d)
    at <::lazy_static::__lazy_static_internal macros>:18
#17 0x00007f11e5b90e7f in xulstore::get_value (doc=0x7ffe38592270, id=0x7ffe38592288, attr=0x7ffe385922a0)
    at toolkit/components/xulstore/src/lib.rs:108
#18 0x00007f11e5b3183c in xulstore::ffi::XULStoreService::get_value
    (self=0x7f11d03fc1a0, doc=0x7ffe38592270, id=0x7ffe38592288, attr=0x7ffe385922a0)
    at toolkit/components/xulstore/src/ffi.rs:88
#19 0x00007f11e5b3387a in xulstore::ffi::XULStoreService::GetValue
    (self=0x7f11d03fc1a0, doc=0x7ffe38592270, id=0x7ffe38592288, attr=0x7ffe385922a0, retval=0x7ffe385922b8)
--Type <RET> for more, q to quit, c to continue without paging--c
    at <::xpcom::method::xpcom_method macros>:29
#20 0x00007f11e5b330b6 in xulstore::ffi::XULStoreService::allocate::GetValue (this=0x7f11d03fc1a0, doc=0x7ffe38592270, id=0x7ffe38592288, attr=0x7ffe385922a0, _retval=0x7ffe385922b8) at toolkit/components/xulstore/src/ffi.rs:23
#21 0x00007f11da8f07ba in NS_InvokeByIndex () at /home/mstange/code/mozilla-central/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:106
#22 0x00007f11dc160cbd in CallMethodHelper::Invoke() (this=0x7ffe38592228) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNative.cpp:1634
#23 0x00007f11dc158400 in CallMethodHelper::Call() (this=0x7ffe38592228) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNative.cpp:1175
#24 0x00007f11dc139326 in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) (ccx=..., mode=XPCWrappedNative::CALL_METHOD) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNative.cpp:1141
#25 0x00007f11dc13affd in XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) (cx=0x7f11cf61b000, argc=3, vp=0x7f11cc05d270) at /home/mstange/code/mozilla-central/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:947
#26 0x00007f11e3c7c71e in CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), js::CallReason, JS::CallArgs const&) (cx=0x7f11cf61b000, native=0x7f11dc13ab80 <XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*)>, reason=js::CallReason::Call, args=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:477
#27 0x00007f11e3c666d1 in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) (cx=0x7f11cf61b000, args=..., construct=js::NO_CONSTRUCT, reason=js::CallReason::Call) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:569
#28 0x00007f11e3c66e91 in InternalCall(JSContext*, js::AnyInvokeArgs const&, js::CallReason) (cx=0x7f11cf61b000, args=..., reason=js::CallReason::Call) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:632
#29 0x00007f11e3c66c52 in js::CallFromStack(JSContext*, JS::CallArgs const&) (cx=0x7f11cf61b000, args=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:636
#30 0x00007f11e3c59d5e in Interpret(JSContext*, js::RunState&) (cx=0x7f11cf61b000, state=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:3028
#31 0x00007f11e3c4e763 in js::RunScript(JSContext*, js::RunState&) (cx=0x7f11cf61b000, state=...) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:449
#32 0x00007f11e3c68122 in js::ExecuteKernel(JSContext*, JS::Handle<JSScript*>, JSObject&, JS::Value const&, js::AbstractFramePtr, JS::Value*) (cx=0x7f11cf61b000, script=..., envChainArg=..., newTargetValue=..., evalInFrame=..., result=0x7ffe38595880) at /home/mstange/code/mozilla-central/js/src/vm/Interpreter.cpp:825
#33 0x00007f11e3cff495 in ExecuteInExtensibleLexicalEnvironment(JSContext*, JS::Handle<JSScript*>, JS::Handle<JSObject*>) (cx=0x7f11cf61b000, scriptArg=..., env=...) at /home/mstange/code/mozilla-central/js/src/builtin/Eval.cpp:497
#34 0x00007f11e3cffb08 in js::ExecuteInJSMEnvironment(JSContext*, JS::Handle<JSScript*>, JS::Handle<JSObject*>, JS::Handle<JS::StackGCVector<JSObject*, js::TempAllocPolicy> >) (cx=0x7f11cf61b000, scriptArg=..., varEnv=..., targetObj=...) at /home/mstange/code/mozilla-central/js/src/builtin/Eval.cpp:604
#35 0x00007f11e3cff7b5 in js::ExecuteInJSMEnvironment(JSContext*, JS::Handle<JSScript*>, JS::Handle<JSObject*>) (cx=0x7f11cf61b000, scriptArg=..., varEnv=...) at /home/mstange/code/mozilla-central/js/src/builtin/Eval.cpp:559
#36 0x00007f11dc062749 in mozJSComponentLoader::ObjectForLocation(ComponentLoaderInfo&, nsIFile*, JS::MutableHandle<JSObject*>, JS::MutableHandle<JSScript*>, char**, bool, JS::MutableHandle<JS::Value>) (this=0x7f11d031e900, aInfo=..., aComponentFile=0x7f11d11dc280, aObject=..., aTableScript=..., aLocation=0x7f11d11506e8, aPropagateExceptions=true, aException=...) at /home/mstange/code/mozilla-central/js/xpconnect/loader/mozJSComponentLoader.cpp:927
#37 0x00007f11dc066f00 in mozJSComponentLoader::Import(JSContext*, nsTSubstring<char> const&, JS::MutableHandle<JSObject*>, JS::MutableHandle<JSObject*>, bool) (this=0x7f11d031e900, aCx=0x7f11cf61b000, aLocation=..., aModuleGlobal=..., aModuleExports=..., aIgnoreExports=false) at /home/mstange/code/mozilla-central/js/xpconnect/loader/mozJSComponentLoader.cpp:1341
#38 0x00007f11da813f93 in mozilla::xpcom::ConstructJSMComponent(nsTSubstring<char> const&, char const*, nsISupports**) (aURI=..., aConstructor=0x7f11d5426cf4 "BrowserGlue", aResult=0x7ffe38598138) at StaticComponents.cpp:1609
#39 0x00007f11da80ebe3 in mozilla::xpcom::CreateInstanceImpl(mozilla::xpcom::ModuleID, nsISupports*, nsID const&, void**) (aID=mozilla::xpcom::ModuleID::Anonymous005, aOuter=0x0, aIID=..., aResult=0x7ffe3859b360) at StaticComponents.cpp:10785
#40 0x00007f11da7f4e45 in mozilla::xpcom::StaticModule::CreateInstance(nsISupports*, nsID const&, void**) const (this=0x7f11d5daf4c0 <mozilla::xpcom::gStaticModules+10032>, aOuter=0x0, aIID=..., aResult=0x7ffe3859b360) at StaticComponents.cpp:11362
#41 0x00007f11da84ff17 in (anonymous namespace)::EntryWrapper::CreateInstance(nsISupports*, nsID const&, void**) (this=0x7ffe3859b430, aOuter=0x0, aIID=..., aResult=0x7ffe3859b360) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManager.cpp:219
#42 0x00007f11da84b126 in nsComponentManagerImpl::GetServiceLocked((anonymous namespace)::MutexLock&, (anonymous namespace)::EntryWrapper&, nsID const&, void**) (this=0x7f11d1112320, aLock=..., aEntry=..., aIID=..., aResult=0x7ffe3859b500) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManager.cpp:1372
#43 0x00007f11da8461d9 in nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) (this=0x7f11d1112320, aContractID=0x7ffe3859b71c "@mozilla.org/browser/browserglue;1", aIID=..., aResult=0x7ffe3859b500) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManager.cpp:1559
#44 0x00007f11da84e799 in CallGetService(char const*, nsID const&, void**) (aContractID=0x7ffe3859b71c "@mozilla.org/browser/browserglue;1", aIID=..., aResult=0x7ffe3859b500) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManagerUtils.cpp:61
#45 0x00007f11da84ef03 in nsGetServiceByContractIDWithError::operator()(nsID const&, void**) const (this=0x7ffe3859b610, aIID=..., aInstancePtr=0x7ffe3859b500) at /home/mstange/code/mozilla-central/xpcom/components/nsComponentManagerUtils.cpp:253
#46 0x00007f11da6d1d8b in nsCOMPtr_base::assign_from_gs_contractid_with_error(nsGetServiceByContractIDWithError const&, nsID const&) (this=0x7ffe3859b630, aGS=..., aIID=...) at /home/mstange/code/mozilla-central/xpcom/base/nsCOMPtr.cpp:91
#47 0x00007f11e3a78e5e in nsCOMPtr<nsISupports>::operator=(nsGetServiceByContractIDWithError const&) (this=0x7ffe3859b630, aRhs=...) at /home/mstange/code/obj-m-debuig/dist/include/nsCOMPtr.h:1053
#48 0x00007f11e3a71588 in nsAppStartupNotifier::NotifyObservers(char const*) (aCategory=0x7f11d57848f4 "app-startup") at /home/mstange/code/mozilla-central/toolkit/xre/nsAppStartupNotifier.cpp:42
#49 0x00007f11e3a6162b in XREMain::XRE_mainRun() (this=0x7ffe3859c060) at /home/mstange/code/mozilla-central/toolkit/xre/nsAppRunner.cpp:4408
#50 0x00007f11e3a62fa0 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) (this=0x7ffe3859c060, argc=4, argv=0x7ffe3859d3c8, aConfig=...) at /home/mstange/code/mozilla-central/toolkit/xre/nsAppRunner.cpp:4761
#51 0x00007f11e3a63489 in XRE_main(int, char**, mozilla::BootstrapConfig const&) (argc=4, argv=0x7ffe3859d3c8, aConfig=...) at /home/mstange/code/mozilla-central/toolkit/xre/nsAppRunner.cpp:4812
#52 0x00007f11e3a79c17 in mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) (this=0x7f11ec8026c0, argc=4, argv=0x7ffe3859d3c8, aConfig=...) at /home/mstange/code/mozilla-central/toolkit/xre/Bootstrap.cpp:45
#53 0x000056122721fa5d in do_main(int, char**, char**) (argc=4, argv=0x7ffe3859d3c8, envp=0x7ffe3859d3f0) at /home/mstange/code/mozilla-central/browser/app/nsBrowserApp.cpp:217
#54 0x000056122721f485 in main(int, char**, char**) (argc=4, argv=0x7ffe3859d3c8, envp=0x7ffe3859d3f0) at /home/mstange/code/mozilla-central/browser/app/nsBrowserApp.cpp:331
rocallahan commented 4 years ago

Thanks. I'm guessing a problem related to LMDB then.

rocallahan commented 4 years ago

I see that the trace maps /tmp/tmp5BXMDv.mozrunner/xulstore/data.mdb shared then does a pwrite64 through that fd, but that gets reflected into the shared memory. OTOH after that there's a buffered pwrite64 that is not reflected, which I guess we should check:

{
  real_time:4960749.081723 global_time:7739, event:`SYSCALLBUF_FLUSH' tid:10607, ticks:185224539
  { syscall:'pwrite64', ret:0x78, size:0x10 }
}
rocallahan commented 4 years ago

Can you upload an rr dump -b -m -p of the rr record -n trace?

rocallahan commented 4 years ago

the 100 events before the event at which rr fails, at least

rocallahan commented 4 years ago

Make it 1000

mstange commented 4 years ago

Sure.

mstange commented 4 years ago

Here you go: https://gist.github.com/mstange/1a89a7ffa0bdd27e0d8c72ef60be8f14

rocallahan commented 4 years ago

I want to know what the filename is here:

{
  real_time:5011593.824812 global_time:17280, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:18639, ticks:185052653
rax:0xffffffffffffffda rbx:0x7ffe38590060 rcx:0xffffffffffffffff rdx:0x81001 rsi:0x7f11d032d9d0 rdi:0xffffff9c rbp:0x7f11d032d9d0 rsp:0x7ffe3858ff90 r8:0x0 r9:0x62646d2e617461 r10:0x0 r11:0x246 r12:0x81001 r13:0x0 r14:0x81001 r15:0x7f11ec8a8d00 rip:0x7f11ecff3b14 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f11eca8c780 gs_base:0x0
}

You should be able to do rr replay -g 17280, make sure you're in the process with thread ID 18639, and then print (char*)0x7f11d032d9d0.

mstange commented 4 years ago

It's /tmp/tmp6HYGgN.mozrunner/xulstore/data.mdb.

mstange@hawkman:~$ rr replay -g 17280 -p 18639
### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmp6HYGgN.mozrunner/runreftest_leaks.log
### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmp6HYGgN.mozrunner/runreftest_leaks_socket_pid18657.log
GNU gdb (Ubuntu 8.3-0ubuntu1) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mstange/.local/share/rr/firefox-3/mmap_hardlink_3_firefox...
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:21725

--------------------------------------------------
 ---> Reached target process 18639 at event 17283.
--------------------------------------------------
Reading symbols from /usr/bin/../lib/rr/librrpreload.so...
[...]
0x00007f11ecff3b14 in __libc_open64 (file=0x7f11d032d9d0 "/tmp/tmp6HYGgN.mozrunner/xulstore/data.mdb", oflag=528385) at ../sysdeps/unix/sysv/linux/open64.c:48
48  ../sysdeps/unix/sysv/linux/open64.c: No such file or directory.
(rr) print (char*)0x7f11d032d9d0
$1 = 0x7f11d032d9d0 "/tmp/tmp6HYGgN.mozrunner/xulstore/data.mdb"
(rr)
rocallahan commented 4 years ago

Thanks! It is as I feared: for whatever reason, the LMDB code is opening the data.mdb file again under a different file description and writing to it. That write should show up in the shared memory mapping, but because it's coming via a different file description, we don't recognize that.

rocallahan commented 4 years ago

(WHY they want to reopen a file they already have a file descriptor for is unclear, but doesn't really matter...)

rocallahan commented 4 years ago

Can you upload the rr record -n trace for me, please?

mstange commented 4 years ago

Will do!

rocallahan commented 4 years ago

Actually that might not be helpful so hold off if you haven't already done it.

mstange commented 4 years ago

All right :) (I have scp'd the rr pack zip to my main machine but I haven't uploaded it to Google Drive yet.)

rocallahan commented 4 years ago

Thanks. Could you re-record with -n and RR_LOG=record_syscall:debug and send the log?

rocallahan commented 4 years ago

I guess we'll also need the rr event dump like last time.

mstange commented 4 years ago

Doing so now.

mstange commented 4 years ago 
mstange@hawkman:~/code/obj-m-debuig$ RR_LOG=record_syscall:debug mach reftest ../mozilla-central/layout/reftests/bugs/1546856-1.html --setpref layout.display-list.dump-content=true --debugger=rr --debugger-args="record -n" > ~/rr-record-syscall-debug.log 2>&1

https://gist.github.com/mstange/ef75063081e8ffca5e70adc893e50e24

mstange@hawkman:~/code/obj-m-debuig$ rr dump -b -m -p > rr-record-n-dump-bmp-2.log

https://gist.github.com/mstange/ebe971b8fc137ff7abf9d9313e1b4779

[FATAL /build/rr-kyNkfr/rr-5.2.0/src/ReplaySession.cc:1023:check_ticks_consistency()]
 (task 24319 (rec:23544) at time 17036)
 -> Assertion `ticks_now == trace_ticks' failed to hold. ticks mismatch for 'SYSCALL: munmap'; expected 184992775, got 184992762
rocallahan commented 4 years ago

OK I added some more logging to rr master, can you pull it and repeat? Thanks!!!

rocallahan commented 4 years ago

another thing to do would be to repeat what you did in https://github.com/mozilla/rr/issues/2417#issuecomment-607539387 and instead of printing the file name, get a stack trace. Then, re-run under gdb (no rr) setting a breakpoint somewhere so we can actually observe that file open outside of rr. Then stat /proc/fd/<opened-fd> and cat /proc/<pid>/maps. And also stat <actual file name>.

mstange commented 4 years ago

https://gist.github.com/mstange/071388c87fd3c185a320ef4bcb1560e7 https://gist.github.com/mstange/a066ea2612a3d3b5b125c19073f20dc1 Doing the rest now.

mstange commented 4 years ago 
mstange@hawkman:~$ stat /proc/14233/fd/34
  File: /proc/14233/fd/34 -> /tmp/tmpld1HoM.mozrunner/xulstore/data.mdb
  Size: 64          Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d   Inode: 27914431    Links: 1
Access: (0300/l-wx------)  Uid: ( 1002/ mstange)   Gid: ( 1002/ mstange)
Access: 2020-04-01 22:39:28.110548405 -0400
Modify: 2020-04-01 22:39:28.110548405 -0400
Change: 2020-04-01 22:39:28.110548405 -0400
 Birth: -
mstange@hawkman:~$ stat /tmp/tmpld1HoM.mozrunner/xulstore/data.mdb
  File: /tmp/tmpld1HoM.mozrunner/xulstore/data.mdb
  Size: 8192        Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 56531228    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1002/ mstange)   Gid: ( 1002/ mstange)
Access: 2020-04-01 22:36:48.201438220 -0400
Modify: 2020-04-01 22:36:48.201438220 -0400
Change: 2020-04-01 22:36:48.201438220 -0400
 Birth: -
mstange@hawkman:~$ cat /proc/14233/maps
1aacf100c000-1aacf101c000 ---p 00000000 00:00 0
1aacf101c000-1aacf102c000 r-xp 00000000 00:00 0
1aacf102c000-1aacf103c000 ---p 00000000 00:00 0
1aacf103c000-1aacf105c000 r-xp 00000000 00:00 0
1aacf105c000-1aacf106c000 ---p 00000000 00:00 0
1aacf106c000-1aacf1078000 r-xp 00000000 00:00 0
1aacf1078000-1aacf107c000 r-xp 00000000 00:00 0
1aacf107c000-1aacf108c000 ---p 00000000 00:00 0
1aacf108c000-1aacf1092000 r-xp 00000000 00:00 0
1aacf1092000-1aacf109c000 r-xp 00000000 00:00 0
1aacf109c000-1aad70c0c000 ---p 00000000 00:00 0
263302d00000-263302e00000 rw-p 00000000 00:00 0
30e287e00000-30e287f00000 rw-p 00000000 00:00 0
555555554000-5555555ca000 r--p 00000000 08:01 92102571                   /home/mstange/code/obj-m-debuig/dist/bin/firefox
5555555ca000-55555574e000 r-xp 00076000 08:01 92102571                   /home/mstange/code/obj-m-debuig/dist/bin/firefox
55555574e000-55555574f000 r--p 001fa000 08:01 92102571                   /home/mstange/code/obj-m-debuig/dist/bin/firefox
55555574f000-555555750000 rw-p 001fb000 08:01 92102571                   /home/mstange/code/obj-m-debuig/dist/bin/firefox
555555750000-555555751000 rw-p 00000000 00:00 0                          [heap]
7fffd59fd000-7fffd5afd000 r--s 00000000 08:01 56531228                   /tmp/tmpld1HoM.mozrunner/xulstore/data.mdb
7fffd5afd000-7fffd5afe000 ---p 00000000 00:00 0
7fffd5afe000-7fffd62fe000 rw-p 00000000 00:00 0
7fffd62fe000-7fffd62ff000 ---p 00000000 00:00 0
7fffd62ff000-7fffd6aff000 rw-p 00000000 00:00 0
7fffd6aff000-7fffd6b00000 ---p 00000000 00:00 0
7fffd6b00000-7fffd7300000 rw-p 00000000 00:00 0
7fffd7300000-7fffd73ff000 rw-p 00000000 00:00 0
7fffd73ff000-7fffd7400000 ---p 00000000 00:00 0
7fffd7400000-7fffd74ff000 rw-p 00000000 00:00 0
7fffd74ff000-7fffd7600000 ---p 00000000 00:00 0
7fffd7600000-7fffd76ff000 rw-p 00000000 00:00 0
7fffd76ff000-7fffd7700000 ---p 00000000 00:00 0
7fffd7700000-7fffd77ff000 rw-p 00000000 00:00 0
7fffd77ff000-7fffd7800000 ---p 00000000 00:00 0
7fffd7800000-7fffd78ff000 rw-p 00000000 00:00 0
7fffd78ff000-7fffd7900000 ---p 00000000 00:00 0
7fffd7907000-7fffd7908000 ---p 00000000 00:00 0
7fffd7908000-7fffd7b06000 rw-p 00000000 00:00 0
7fffd7b06000-7fffd7b07000 ---p 00000000 00:00 0
7fffd7b07000-7fffd7d05000 rw-p 00000000 00:00 0
7fffd7d05000-7fffd7d06000 ---p 00000000 00:00 0
7fffd7d06000-7fffd7f04000 rw-p 00000000 00:00 0
7fffd7f04000-7fffd7f05000 ---p 00000000 00:00 0
7fffd7f05000-7fffd8103000 rw-p 00000000 00:00 0
7fffd8103000-7fffd8104000 ---p 00000000 00:00 0
7fffd8104000-7fffd8302000 rw-p 00000000 00:00 0
7fffd8302000-7fffd8303000 ---p 00000000 00:00 0
7fffd8303000-7fffd8501000 rw-p 00000000 00:00 0
7fffd8501000-7fffd8502000 ---p 00000000 00:00 0
7fffd8502000-7fffd8700000 rw-p 00000000 00:00 0
7fffd8700000-7fffd8701000 ---p 00000000 00:00 0
7fffd8701000-7fffd88ff000 rw-p 00000000 00:00 0
7fffd88ff000-7fffd8900000 ---p 00000000 00:00 0
7fffd8900000-7fffd9100000 rw-p 00000000 00:00 0
7fffd9100000-7fffd91ff000 rw-p 00000000 00:00 0
7fffd91ff000-7fffd9200000 ---p 00000000 00:00 0
7fffd92fd000-7fffd92fe000 ---p 00000000 00:00 0
7fffd92fe000-7fffd9afe000 rw-p 00000000 00:00 0
7fffd9afe000-7fffd9aff000 ---p 00000000 00:00 0
7fffd9aff000-7fffda2ff000 rw-p 00000000 00:00 0
7fffda2ff000-7fffda300000 ---p 00000000 00:00 0
7fffda300000-7fffdab00000 rw-p 00000000 00:00 0
7fffdab00000-7fffdabff000 rw-p 00000000 00:00 0
7fffdabff000-7fffdac00000 ---p 00000000 00:00 0
7fffdacff000-7fffdad00000 ---p 00000000 00:00 0
7fffdad00000-7fffdb5ff000 rw-p 00000000 00:00 0
7fffdb5ff000-7fffdb600000 ---p 00000000 00:00 0
7fffdb600000-7fffdb6ff000 rw-p 00000000 00:00 0
7fffdb6ff000-7fffdb700000 ---p 00000000 00:00 0
7fffdb772000-7fffdb7a6000 r--p 00000000 00:19 77                         /dev/shm/org.mozilla.ipc.14233.1 (deleted)
7fffdb7a6000-7fffdb7a9000 r--p 00000000 08:01 105392468                  /lib/x86_64-linux-gnu/libnss_files-2.30.so
7fffdb7a9000-7fffdb7b0000 r-xp 00003000 08:01 105392468                  /lib/x86_64-linux-gnu/libnss_files-2.30.so
7fffdb7b0000-7fffdb7b2000 r--p 0000a000 08:01 105392468                  /lib/x86_64-linux-gnu/libnss_files-2.30.so
7fffdb7b2000-7fffdb7b3000 r--p 0000b000 08:01 105392468                  /lib/x86_64-linux-gnu/libnss_files-2.30.so
7fffdb7b3000-7fffdb7b4000 rw-p 0000c000 08:01 105392468                  /lib/x86_64-linux-gnu/libnss_files-2.30.so
7fffdb7b4000-7fffdb7ba000 rw-p 00000000 00:00 0
7fffdb7ba000-7fffdb7bc000 r--p 00000000 08:01 38544992                   /usr/lib/x86_64-linux-gnu/libogg.so.0.8.4
7fffdb7bc000-7fffdb7c1000 r-xp 00002000 08:01 38544992                   /usr/lib/x86_64-linux-gnu/libogg.so.0.8.4
7fffdb7c1000-7fffdb7c4000 r--p 00007000 08:01 38544992                   /usr/lib/x86_64-linux-gnu/libogg.so.0.8.4
7fffdb7c4000-7fffdb7c5000 ---p 0000a000 08:01 38544992                   /usr/lib/x86_64-linux-gnu/libogg.so.0.8.4
7fffdb7c5000-7fffdb7c6000 r--p 0000a000 08:01 38544992                   /usr/lib/x86_64-linux-gnu/libogg.so.0.8.4
7fffdb7c6000-7fffdb7c7000 rw-p 0000b000 08:01 38544992                   /usr/lib/x86_64-linux-gnu/libogg.so.0.8.4
7fffdb7c7000-7fffdb7ca000 r--p 00000000 08:01 38537623                   /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7fffdb7ca000-7fffdb7e0000 r-xp 00003000 08:01 38537623                   /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7fffdb7e0000-7fffdb7f1000 r--p 00019000 08:01 38537623                   /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7fffdb7f1000-7fffdb7f2000 ---p 0002a000 08:01 38537623                   /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7fffdb7f2000-7fffdb7f3000 r--p 0002a000 08:01 38537623                   /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7fffdb7f3000-7fffdb7f4000 rw-p 0002b000 08:01 38537623                   /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7fffdb7f4000-7fffdb7f6000 r--p 00000000 08:01 38535371                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7fffdb7f6000-7fffdb7fb000 r-xp 00002000 08:01 38535371                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7fffdb7fb000-7fffdb7fd000 r--p 00007000 08:01 38535371                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7fffdb7fd000-7fffdb7fe000 r--p 00008000 08:01 38535371                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7fffdb7fe000-7fffdb7ff000 rw-p 00009000 08:01 38535371                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7fffdb7ff000-7fffdb800000 ---p 00000000 00:00 0
7fffdb800000-7fffdc0ff000 rw-p 00000000 00:00 0
7fffdc0ff000-7fffdc100000 ---p 00000000 00:00 0
7fffdc107000-7fffdc10b000 r--p 00000000 08:01 38545270                   /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.18
7fffdc10b000-7fffdc119000 r-xp 00004000 08:01 38545270                   /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.18
7fffdc119000-7fffdc11f000 r--p 00012000 08:01 38545270                   /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.18
7fffdc11f000-7fffdc120000 r--p 00017000 08:01 38545270                   /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.18
7fffdc120000-7fffdc121000 rw-p 00018000 08:01 38545270                   /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.18
7fffdc14b000-7fffdc14d000 rw-s 00000000 08:01 56531227                   /tmp/tmpld1HoM.mozrunner/xulstore/lock.mdb
7fffdc14d000-7fffdc14e000 ---p 00000000 00:00 0
7fffdc14e000-7fffdc156000 rw-p 00000000 00:00 0
7fffdc156000-7fffdc157000 r--p 00000000 00:19 38                         /dev/shm/org.mozilla.ipc.14233.0 (deleted)
7fffdc157000-7fffdc169000 r--p 00000000 08:01 38929279                   /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so
7fffdc169000-7fffdc183000 r-xp 00012000 08:01 38929279                   /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so
7fffdc183000-7fffdc190000 r--p 0002c000 08:01 38929279                   /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so
7fffdc190000-7fffdc196000 r--p 00038000 08:01 38929279                   /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so
7fffdc196000-7fffdc197000 rw-p 0003e000 08:01 38929279                   /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so
7fffdc197000-7fffdc1a1000 r--p 00000000 08:01 38929275                   /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
7fffdc1a1000-7fffdc1c1000 r-xp 0000a000 08:01 38929275                   /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
7fffdc1c1000-7fffdc1cc000 r--p 0002a000 08:01 38929275                   /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
7fffdc1cc000-7fffdc1cd000 ---p 00035000 08:01 38929275                   /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
7fffdc1cd000-7fffdc1cf000 r--p 00035000 08:01 38929275                   /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
7fffdc1cf000-7fffdc1d0000 rw-p 00037000 08:01 38929275                   /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
7fffdc1d0000-7fffdca62000 r--p 00000000 08:01 38542686                   /usr/lib/locale/locale-archive
7fffdca62000-7fffdca67000 r--p 00000000 08:01 38537188                   /usr/lib/x86_64-linux-gnu/libICE.so.6.3.0
7fffdca67000-7fffdca76000 r-xp 00005000 08:01 38537188                   /usr/lib/x86_64-linux-gnu/libICE.so.6.3.0
7fffdca76000-7fffdca7b000 r--p 00014000 08:01 38537188                   /usr/lib/x86_64-linux-gnu/libICE.so.6.3.0
7fffdca7b000-7fffdca7c000 r--p 00018000 08:01 38537188                   /usr/lib/x86_64-linux-gnu/libICE.so.6.3.0
7fffdca7c000-7fffdca7d000 rw-p 00019000 08:01 38537188                   /usr/lib/x86_64-linux-gnu/libICE.so.6.3.0
7fffdca7d000-7fffdca80000 rw-p 00000000 00:00 0
7fffdca80000-7fffdca82000 r--p 00000000 08:01 38543024                   /usr/lib/x86_64-linux-gnu/libSM.so.6.0.1
7fffdca82000-7fffdca87000 r-xp 00002000 08:01 38543024                   /usr/lib/x86_64-linux-gnu/libSM.so.6.0.1
7fffdca87000-7fffdca89000 r--p 00007000 08:01 38543024                   /usr/lib/x86_64-linux-gnu/libSM.so.6.0.1
7fffdca89000-7fffdca8a000 r--p 00008000 08:01 38543024                   /usr/lib/x86_64-linux-gnu/libSM.so.6.0.1
7fffdca8a000-7fffdca8b000 rw-p 00009000 08:01 38543024                   /usr/lib/x86_64-linux-gnu/libSM.so.6.0.1
7fffdca8b000-7fffdca8c000 r--p 00000000 08:01 38546471                   /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0.6200.4
7fffdca8c000-7fffdca8d000 r-xp 00001000 08:01 38546471                   /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0.6200.4
7fffdca8d000-7fffdca8e000 r--p 00002000 08:01 38546471                   /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0.6200.4
7fffdca8e000-7fffdca8f000 r--p 00002000 08:01 38546471                   /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0.6200.4
7fffdca8f000-7fffdca90000 rw-p 00003000 08:01 38546471                   /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0.6200.4
7fffdca90000-7fffdcaf2000 r-xp 00000000 08:01 38542782                   /usr/lib/x86_64-linux-gnu/libXt.so.6.0.0
7fffdcaf2000-7fffdccf2000 ---p 00062000 08:01 38542782                   /usr/lib/x86_64-linux-gnu/libXt.so.6.0.0
7fffdccf2000-7fffdccf3000 r--p 00062000 08:01 38542782                   /usr/lib/x86_64-linux-gnu/libXt.so.6.0.0
7fffdccf3000-7fffdccf8000 rw-p 00063000 08:01 38542782                   /usr/lib/x86_64-linux-gnu/libXt.so.6.0.0
7fffdccf8000-7fffdccf9000 rw-p 00000000 00:00 0
7fffdccf9000-7fffdccfb000 r--p 00000000 08:01 38537620                   /usr/lib/x86_64-linux-gnu/libvorbisfile.so.3.3.7
7fffdccfb000-7fffdcd00000 r-xp 00002000 08:01 38537620                   /usr/lib/x86_64-linux-gnu/libvorbisfile.so.3.3.7
7fffdcd00000-7fffdcd01000 r--p 00007000 08:01 38537620                   /usr/lib/x86_64-linux-gnu/libvorbisfile.so.3.3.7
7fffdcd01000-7fffdcd02000 ---p 00008000 08:01 38537620                   /usr/lib/x86_64-linux-gnu/libvorbisfile.so.3.3.7
7fffdcd02000-7fffdcd03000 r--p 00008000 08:01 38537620                   /usr/lib/x86_64-linux-gnu/libvorbisfile.so.3.3.7
7fffdcd03000-7fffdcd04000 rw-p 00009000 08:01 38537620                   /usr/lib/x86_64-linux-gnu/libvorbisfile.so.3.3.7
7fffdcd04000-7fffdcd07000 r--p 00000000 08:01 38542104                   /usr/lib/x86_64-linux-gnu/libcanberra.so.0.2.5
7fffdcd07000-7fffdcd11000 r-xp 00003000 08:01 38542104                   /usr/lib/x86_64-linux-gnu/libcanberra.so.0.2.5
7fffdcd11000-7fffdcd15000 r--p 0000d000 08:01 38542104                   /usr/lib/x86_64-linux-gnu/libcanberra.so.0.2.5
7fffdcd15000-7fffdcd16000 r--p 00010000 08:01 38542104                   /usr/lib/x86_64-linux-gnu/libcanberra.so.0.2.5
7fffdcd16000-7fffdcd17000 rw-p 00011000 08:01 38542104                   /usr/lib/x86_64-linux-gnu/libcanberra.so.0.2.5
7fffdcd17000-7fffdcd19000 r--p 00000000 08:01 38542106                   /usr/lib/x86_64-linux-gnu/libcanberra-gtk3.so.0.1.9
7fffdcd19000-7fffdcd1b000 r-xp 00002000 08:01 38542106                   /usr/lib/x86_64-linux-gnu/libcanberra-gtk3.so.0.1.9
7fffdcd1b000-7fffdcd1c000 r--p 00004000 08:01 38542106                   /usr/lib/x86_64-linux-gnu/libcanberra-gtk3.so.0.1.9
7fffdcd1c000-7fffdcd1d000 r--p 00004000 08:01 38542106                   /usr/lib/x86_64-linux-gnu/libcanberra-gtk3.so.0.1.9
7fffdcd1d000-7fffdcd1e000 rw-p 00005000 08:01 38542106                   /usr/lib/x86_64-linux-gnu/libcanberra-gtk3.so.0.1.9
7fffdcd1e000-7fffdcd20000 r--p 00000000 08:01 38928778                   /usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
7fffdcd20000-7fffdcd23000 r-xp 00002000 08:01 38928778                   /usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
7fffdcd23000-7fffdcd24000 r--p 00005000 08:01 38928778                   /usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
7fffdcd24000-7fffdcd25000 ---p 00006000 08:01 38928778                   /usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
7fffdcd25000-7fffdcd26000 r--p 00006000 08:01 38928778                   /usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
7fffdcd26000-7fffdcd27000 rw-p 00007000 08:01 38928778                   /usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
7fffdcd27000-7fffdcd28000 r--p 00000000 08:01 40510780                   /usr/share/locale-langpack/en_CA/LC_MESSAGES/glib20.mo
7fffdcd28000-7fffdcd2f000 r--s 00000000 08:01 38930978                   /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7fffdcd2f000-7fffe18e3000 r--p 00000000 08:01 91831738                   /home/mstange/code/obj-m-debuig/toolkit/library/build/libxul.so
7fffe18e3000-7fffe18e4000 r--p 04bb4000 08:01 91831738                   /home/mstange/code/obj-m-debuig/toolkit/library/build/libxul.so
7fffe18e4000-7fffe5549000 r--p 04bb5000 08:01 91831738                   /home/mstange/code/obj-m-debuig/toolkit/library/build/libxul.so
7fffe5549000-7ffff43ec000 r-xp 0881a000 08:01 91831738                   /home/mstange/code/obj-m-debuig/toolkit/library/build/libxul.so
7ffff43ec000-7ffff4bc5000 r--p 176bd000 08:01 91831738                   /home/mstange/code/obj-m-debuig/toolkit/library/build/libxul.so
7ffff4bc5000-7ffff4c0a000 rw-p 17e96000 08:01 91831738                   /home/mstange/code/obj-m-debuig/toolkit/library/build/libxul.so
7ffff4c0a000-7ffff4cf3000 rw-p 00000000 00:00 0
7ffff4cf3000-7ffff4cf7000 r--p 00000000 08:01 105386529                  /lib/x86_64-linux-gnu/libgpg-error.so.0.27.0
7ffff4cf7000-7ffff4d0a000 r-xp 00004000 08:01 105386529                  /lib/x86_64-linux-gnu/libgpg-error.so.0.27.0
7ffff4d0a000-7ffff4d14000 r--p 00017000 08:01 105386529                  /lib/x86_64-linux-gnu/libgpg-error.so.0.27.0
7ffff4d14000-7ffff4d15000 r--p 00020000 08:01 105386529                  /lib/x86_64-linux-gnu/libgpg-error.so.0.27.0
7ffff4d15000-7ffff4d16000 rw-p 00021000 08:01 105386529                  /lib/x86_64-linux-gnu/libgpg-error.so.0.27.0
7ffff4d16000-7ffff4d22000 r--p 00000000 08:01 105390944                  /lib/x86_64-linux-gnu/libgcrypt.so.20.2.4
7ffff4d22000-7ffff4df0000 r-xp 0000c000 08:01 105390944                  /lib/x86_64-linux-gnu/libgcrypt.so.20.2.4
7ffff4df0000-7ffff4e2d000 r--p 000da000 08:01 105390944                  /lib/x86_64-linux-gnu/libgcrypt.so.20.2.4
7ffff4e2d000-7ffff4e2f000 r--p 00116000 08:01 105390944                  /lib/x86_64-linux-gnu/libgcrypt.so.20.2.4
7ffff4e2f000-7ffff4e34000 rw-p 00118000 08:01 105390944                  /lib/x86_64-linux-gnu/libgcrypt.so.20.2.4
7ffff4e34000-7ffff4e36000 r--p 00000000 08:01 38535980                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.1
7ffff4e36000-7ffff4e4f000 r-xp 00002000 08:01 38535980                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.1
7ffff4e4f000-7ffff4e52000 r--p 0001b000 08:01 38535980                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.1
7ffff4e52000-7ffff4e53000 r--p 0001d000 08:01 38535980                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.1
7ffff4e53000-7ffff4e54000 rw-p 0001e000 08:01 38535980                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.1
7ffff4e54000-7ffff4e57000 r--p 00000000 08:01 105386593                  /lib/x86_64-linux-gnu/liblzma.so.5.2.4
7ffff4e57000-7ffff4e6e000 r-xp 00003000 08:01 105386593                  /lib/x86_64-linux-gnu/liblzma.so.5.2.4
7ffff4e6e000-7ffff4e79000 r--p 0001a000 08:01 105386593                  /lib/x86_64-linux-gnu/liblzma.so.5.2.4
7ffff4e79000-7ffff4e7a000 r--p 00024000 08:01 105386593                  /lib/x86_64-linux-gnu/liblzma.so.5.2.4
7ffff4e7a000-7ffff4e7b000 rw-p 00025000 08:01 105386593                  /lib/x86_64-linux-gnu/liblzma.so.5.2.4
7ffff4e7b000-7ffff4e7f000 r--p 00000000 08:01 38542742                   /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0
7ffff4e7f000-7ffff4e8e000 r-xp 00004000 08:01 38542742                   /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0
7ffff4e8e000-7ffff4e91000 r--p 00013000 08:01 38542742                   /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0
7ffff4e91000-7ffff4e92000 ---p 00016000 08:01 38542742                   /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0
7ffff4e92000-7ffff4e93000 r--p 00016000 08:01 38542742                   /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0
7ffff4e93000-7ffff4e94000 rw-p 00017000 08:01 38542742                   /usr/lib/x86_64-linux-gnu/libbsd.so.0.10.0
7ffff4e94000-7ffff4e95000 rw-p 00000000 00:00 0
7ffff4e95000-7ffff4e97000 r--p 00000000 08:01 38539681                   /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.5
7ffff4e97000-7ffff4e9b000 r-xp 00002000 08:01 38539681                   /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.5
7ffff4e9b000-7ffff4e9d000 r--p 00006000 08:01 38539681                   /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.5
7ffff4e9d000-7ffff4e9e000 r--p 00007000 08:01 38539681                   /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.5
7ffff4e9e000-7ffff4e9f000 rw-p 00008000 08:01 38539681                   /usr/lib/x86_64-linux-gnu/libdatrie.so.1.3.5
7ffff4e9f000-7ffff4ea1000 r--p 00000000 08:01 38542340                   /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.7.1
7ffff4ea1000-7ffff4efd000 r-xp 00002000 08:01 38542340                   /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.7.1
7ffff4efd000-7ffff4f22000 r--p 0005e000 08:01 38542340                   /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.7.1
7ffff4f22000-7ffff4f23000 r--p 00082000 08:01 38542340                   /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.7.1
7ffff4f23000-7ffff4f24000 rw-p 00083000 08:01 38542340                   /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.7.1
7ffff4f24000-7ffff4f2e000 r--p 00000000 08:01 105390474                  /lib/x86_64-linux-gnu/libblkid.so.1.1.0
7ffff4f2e000-7ffff4f64000 r-xp 0000a000 08:01 105390474                  /lib/x86_64-linux-gnu/libblkid.so.1.1.0
7ffff4f64000-7ffff4f74000 r--p 00040000 08:01 105390474                  /lib/x86_64-linux-gnu/libblkid.so.1.1.0
7ffff4f74000-7ffff4f75000 ---p 00050000 08:01 105390474                  /lib/x86_64-linux-gnu/libblkid.so.1.1.0
7ffff4f75000-7ffff4f7a000 r--p 00050000 08:01 105390474                  /lib/x86_64-linux-gnu/libblkid.so.1.1.0
7ffff4f7a000-7ffff4f7b000 rw-p 00055000 08:01 105390474                  /lib/x86_64-linux-gnu/libblkid.so.1.1.0
7ffff4f7b000-7ffff4f8b000 r--p 00000000 08:01 105392589                  /lib/x86_64-linux-gnu/libsystemd.so.0.26.0
7ffff4f8b000-7ffff4ffa000 r-xp 00010000 08:01 105392589                  /lib/x86_64-linux-gnu/libsystemd.so.0.26.0
7ffff4ffa000-7ffff501d000 r--p 0007f000 08:01 105392589                  /lib/x86_64-linux-gnu/libsystemd.so.0.26.0
7ffff501d000-7ffff501e000 ---p 000a2000 08:01 105392589                  /lib/x86_64-linux-gnu/libsystemd.so.0.26.0
7ffff501e000-7ffff5021000 r--p 000a2000 08:01 105392589                  /lib/x86_64-linux-gnu/libsystemd.so.0.26.0
7ffff5021000-7ffff5022000 rw-p 000a5000 08:01 105392589                  /lib/x86_64-linux-gnu/libsystemd.so.0.26.0
7ffff5022000-7ffff5023000 rw-p 00000000 00:00 0
7ffff5023000-7ffff5025000 r--p 00000000 08:01 38545313                   /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
7ffff5025000-7ffff5027000 r-xp 00002000 08:01 38545313                   /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
7ffff5027000-7ffff5029000 r--p 00004000 08:01 38545313                   /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
7ffff5029000-7ffff502a000 r--p 00005000 08:01 38545313                   /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
7ffff502a000-7ffff502b000 rw-p 00006000 08:01 38545313                   /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
7ffff502b000-7ffff502c000 r--p 00000000 08:01 38544879                   /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
7ffff502c000-7ffff502e000 r-xp 00001000 08:01 38544879                   /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
7ffff502e000-7ffff502f000 r--p 00003000 08:01 38544879                   /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
7ffff502f000-7ffff5030000 r--p 00003000 08:01 38544879                   /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
7ffff5030000-7ffff5031000 rw-p 00004000 08:01 38544879                   /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
7ffff5031000-7ffff5033000 r--p 00000000 08:01 105386484                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7ffff5033000-7ffff5037000 r-xp 00002000 08:01 105386484                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7ffff5037000-7ffff5038000 r--p 00006000 08:01 105386484                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7ffff5038000-7ffff5039000 r--p 00006000 08:01 105386484                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7ffff5039000-7ffff503a000 rw-p 00007000 08:01 105386484                  /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7ffff503a000-7ffff503e000 r--p 00000000 08:01 105385638                  /lib/x86_64-linux-gnu/libexpat.so.1.6.9
7ffff503e000-7ffff505a000 r-xp 00004000 08:01 105385638                  /lib/x86_64-linux-gnu/libexpat.so.1.6.9
7ffff505a000-7ffff5064000 r--p 00020000 08:01 105385638                  /lib/x86_64-linux-gnu/libexpat.so.1.6.9
7ffff5064000-7ffff5065000 ---p 0002a000 08:01 105385638                  /lib/x86_64-linux-gnu/libexpat.so.1.6.9
7ffff5065000-7ffff5067000 r--p 0002a000 08:01 105385638                  /lib/x86_64-linux-gnu/libexpat.so.1.6.9
7ffff5067000-7ffff5068000 rw-p 0002c000 08:01 105385638                  /lib/x86_64-linux-gnu/libexpat.so.1.6.9
7ffff5068000-7ffff506a000 r--p 00000000 08:01 105386496                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7ffff506a000-7ffff50bc000 r-xp 00002000 08:01 105386496                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7ffff50bc000-7ffff50da000 r--p 00054000 08:01 105386496                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7ffff50da000-7ffff50db000 r--p 00071000 08:01 105386496                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7ffff50db000-7ffff50dc000 rw-p 00072000 08:01 105386496                  /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7ffff50dc000-7ffff50de000 r--p 00000000 08:01 38535181                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7ffff50de000-7ffff50e3000 r-xp 00002000 08:01 38535181                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7ffff50e3000-7ffff50e4000 r--p 00007000 08:01 38535181                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7ffff50e4000-7ffff50e5000 r--p 00007000 08:01 38535181                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7ffff50e5000-7ffff50e6000 rw-p 00008000 08:01 38535181                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7ffff50e6000-7ffff50e8000 r--p 00000000 08:01 38538844                   /usr/lib/x86_64-linux-gnu/libthai.so.0.3.1
7ffff50e8000-7ffff50ec000 r-xp 00002000 08:01 38538844                   /usr/lib/x86_64-linux-gnu/libthai.so.0.3.1
7ffff50ec000-7ffff50ef000 r--p 00006000 08:01 38538844                   /usr/lib/x86_64-linux-gnu/libthai.so.0.3.1
7ffff50ef000-7ffff50f0000 r--p 00008000 08:01 38538844                   /usr/lib/x86_64-linux-gnu/libthai.so.0.3.1
7ffff50f0000-7ffff50f1000 rw-p 00009000 08:01 38538844                   /usr/lib/x86_64-linux-gnu/libthai.so.0.3.1
7ffff50f1000-7ffff50f4000 r--p 00000000 08:01 38548110                   /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.2.1
7ffff50f4000-7ffff5116000 r-xp 00003000 08:01 38548110                   /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.2.1
7ffff5116000-7ffff511b000 r--p 00025000 08:01 38548110                   /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.2.1
7ffff511b000-7ffff511d000 r--p 00029000 08:01 38548110                   /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.2.1
7ffff511d000-7ffff511e000 rw-p 0002b000 08:01 38548110                   /usr/lib/x86_64-linux-gnu/libgraphite2.so.3.2.1
7ffff511e000-7ffff5122000 r--p 00000000 08:01 105392482                  /lib/x86_64-linux-gnu/libresolv-2.30.so
7ffff5122000-7ffff5131000 r-xp 00004000 08:01 105392482                  /lib/x86_64-linux-gnu/libresolv-2.30.so
7ffff5131000-7ffff5134000 r--p 00013000 08:01 105392482                  /lib/x86_64-linux-gnu/libresolv-2.30.so
7ffff5134000-7ffff5135000 ---p 00016000 08:01 105392482                  /lib/x86_64-linux-gnu/libresolv-2.30.so
7ffff5135000-7ffff5136000 r--p 00016000 08:01 105392482                  /lib/x86_64-linux-gnu/libresolv-2.30.so
7ffff5136000-7ffff5137000 rw-p 00017000 08:01 105392482                  /lib/x86_64-linux-gnu/libresolv-2.30.so
7ffff5137000-7ffff5139000 rw-p 00000000 00:00 0
7ffff5139000-7ffff513f000 r--p 00000000 08:01 105392533                  /lib/x86_64-linux-gnu/libselinux.so.1
7ffff513f000-7ffff5158000 r-xp 00006000 08:01 105392533                  /lib/x86_64-linux-gnu/libselinux.so.1
7ffff5158000-7ffff515f000 r--p 0001f000 08:01 105392533                  /lib/x86_64-linux-gnu/libselinux.so.1
7ffff515f000-7ffff5160000 ---p 00026000 08:01 105392533                  /lib/x86_64-linux-gnu/libselinux.so.1
7ffff5160000-7ffff5161000 r--p 00026000 08:01 105392533                  /lib/x86_64-linux-gnu/libselinux.so.1
7ffff5161000-7ffff5162000 rw-p 00027000 08:01 105392533                  /lib/x86_64-linux-gnu/libselinux.so.1
7ffff5162000-7ffff5164000 rw-p 00000000 00:00 0
7ffff5164000-7ffff516f000 r--p 00000000 08:01 105392340                  /lib/x86_64-linux-gnu/libmount.so.1.1.0
7ffff516f000-7ffff51ad000 r-xp 0000b000 08:01 105392340                  /lib/x86_64-linux-gnu/libmount.so.1.1.0
7ffff51ad000-7ffff51c0000 r--p 00049000 08:01 105392340                  /lib/x86_64-linux-gnu/libmount.so.1.1.0
7ffff51c0000-7ffff51c3000 r--p 0005b000 08:01 105392340                  /lib/x86_64-linux-gnu/libmount.so.1.1.0
7ffff51c3000-7ffff51c4000 rw-p 0005e000 08:01 105392340                  /lib/x86_64-linux-gnu/libmount.so.1.1.0
7ffff51c4000-7ffff51d3000 r--p 00000000 08:01 38536771                   /usr/lib/x86_64-linux-gnu/libatspi.so.0.0.1
7ffff51d3000-7ffff51e9000 r-xp 0000f000 08:01 38536771                   /usr/lib/x86_64-linux-gnu/libatspi.so.0.0.1
7ffff51e9000-7ffff51f6000 r--p 00025000 08:01 38536771                   /usr/lib/x86_64-linux-gnu/libatspi.so.0.0.1
7ffff51f6000-7ffff51f7000 ---p 00032000 08:01 38536771                   /usr/lib/x86_64-linux-gnu/libatspi.so.0.0.1
7ffff51f7000-7ffff51fa000 r--p 00032000 08:01 38536771                   /usr/lib/x86_64-linux-gnu/libatspi.so.0.0.1
7ffff51fa000-7ffff51fb000 rw-p 00035000 08:01 38536771                   /usr/lib/x86_64-linux-gnu/libatspi.so.0.0.1
7ffff51fb000-7ffff5206000 r--p 00000000 08:01 105391452                  /lib/x86_64-linux-gnu/libdbus-1.so.3.19.10
7ffff5206000-7ffff5233000 r-xp 0000b000 08:01 105391452                  /lib/x86_64-linux-gnu/libdbus-1.so.3.19.10
7ffff5233000-7ffff5248000 r--p 00038000 08:01 105391452                  /lib/x86_64-linux-gnu/libdbus-1.so.3.19.10
7ffff5248000-7ffff5249000 r--p 0004c000 08:01 105391452                  /lib/x86_64-linux-gnu/libdbus-1.so.3.19.10
7ffff5249000-7ffff524a000 rw-p 0004d000 08:01 105391452                  /lib/x86_64-linux-gnu/libdbus-1.so.3.19.10
7ffff524a000-7ffff524c000 r--p 00000000 08:01 105383614                  /lib/x86_64-linux-gnu/libz.so.1.2.11
7ffff524c000-7ffff525d000 r-xp 00002000 08:01 105383614                  /lib/x86_64-linux-gnu/libz.so.1.2.11
7ffff525d000-7ffff5263000 r--p 00013000 08:01 105383614                  /lib/x86_64-linux-gnu/libz.so.1.2.11
7ffff5263000-7ffff5264000 ---p 00019000 08:01 105383614                  /lib/x86_64-linux-gnu/libz.so.1.2.11
7ffff5264000-7ffff5265000 r--p 00019000 08:01 105383614                  /lib/x86_64-linux-gnu/libz.so.1.2.11
7ffff5265000-7ffff5266000 rw-p 0001a000 08:01 105383614                  /lib/x86_64-linux-gnu/libz.so.1.2.11
7ffff5266000-7ffff526f000 r-xp 00000000 08:01 38542778                   /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0
7ffff526f000-7ffff546e000 ---p 00009000 08:01 38542778                   /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0
7ffff546e000-7ffff546f000 r--p 00008000 08:01 38542778                   /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0
7ffff546f000-7ffff5470000 rw-p 00009000 08:01 38542778                   /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0
7ffff5470000-7ffff5475000 r--p 00000000 08:01 38578032                   /usr/lib/x86_64-linux-gnu/libxcb-render.so.0.0.0
7ffff5475000-7ffff547a000 r-xp 00005000 08:01 38578032                   /usr/lib/x86_64-linux-gnu/libxcb-render.so.0.0.0
7ffff547a000-7ffff547c000 r--p 0000a000 08:01 38578032                   /usr/lib/x86_64-linux-gnu/libxcb-render.so.0.0.0
7ffff547c000-7ffff547d000 ---p 0000c000 08:01 38578032                   /usr/lib/x86_64-linux-gnu/libxcb-render.so.0.0.0
7ffff547d000-7ffff547e000 r--p 0000c000 08:01 38578032                   /usr/lib/x86_64-linux-gnu/libxcb-render.so.0.0.0
7ffff547e000-7ffff547f000 rw-p 0000d000 08:01 38578032                   /usr/lib/x86_64-linux-gnu/libxcb-render.so.0.0.0
7ffff547f000-7ffff5480000 r--p 00000000 08:01 38539684                   /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0.0.0
7ffff5480000-7ffff5481000 r-xp 00001000 08:01 38539684                   /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0.0.0
7ffff5481000-7ffff5482000 r--p 00002000 08:01 38539684                   /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0.0.0
7ffff5482000-7ffff5483000 r--p 00002000 08:01 38539684                   /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0.0.0
7ffff5483000-7ffff5484000 rw-p 00003000 08:01 38539684                   /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0.0.0
7ffff5484000-7ffff5489000 r--p 00000000 08:01 38539482                   /usr/lib/x86_64-linux-gnu/libpng16.so.16.37.0
7ffff5489000-7ffff54ae000 r-xp 00005000 08:01 38539482                   /usr/lib/x86_64-linux-gnu/libpng16.so.16.37.0
7ffff54ae000-7ffff54ba000 r--p 0002a000 08:01 38539482                   /usr/lib/x86_64-linux-gnu/libpng16.so.16.37.0
7ffff54ba000-7ffff54bb000 r--p 00035000 08:01 38539482                   /usr/lib/x86_64-linux-gnu/libpng16.so.16.37.0
7ffff54bb000-7ffff54bc000 rw-p 00036000 08:01 38539482                   /usr/lib/x86_64-linux-gnu/libpng16.so.16.37.0
7ffff54bc000-7ffff54c6000 r--p 00000000 08:01 38543590                   /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.38.4
7ffff54c6000-7ffff554a000 r-xp 0000a000 08:01 38543590                   /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.38.4
7ffff554a000-7ffff555a000 r--p 0008e000 08:01 38543590                   /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.38.4
7ffff555a000-7ffff5562000 r--p 0009d000 08:01 38543590                   /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.38.4
7ffff5562000-7ffff5563000 rw-p 000a5000 08:01 38543590                   /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.38.4
7ffff5563000-7ffff556e000 r--p 00000000 08:01 38543606                   /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
7ffff556e000-7ffff5581000 r-xp 0000b000 08:01 38543606                   /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
7ffff5581000-7ffff558a000 r--p 0001e000 08:01 38543606                   /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
7ffff558a000-7ffff558b000 r--p 00026000 08:01 38543606                   /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
7ffff558b000-7ffff558c000 rw-p 00027000 08:01 38543606                   /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
7ffff558c000-7ffff5590000 r--p 00000000 08:01 38548159                   /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
7ffff5590000-7ffff559b000 r-xp 00004000 08:01 38548159                   /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
7ffff559b000-7ffff559f000 r--p 0000f000 08:01 38548159                   /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
7ffff559f000-7ffff55a0000 r--p 00012000 08:01 38548159                   /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
7ffff55a0000-7ffff55a1000 rw-p 00013000 08:01 38548159                   /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
7ffff55a1000-7ffff55a6000 r--p 00000000 08:01 38544626                   /usr/lib/x86_64-linux-gnu/libwayland-client.so.0.3.0
7ffff55a6000-7ffff55ac000 r-xp 00005000 08:01 38544626                   /usr/lib/x86_64-linux-gnu/libwayland-client.so.0.3.0
7ffff55ac000-7ffff55af000 r--p 0000b000 08:01 38544626                   /usr/lib/x86_64-linux-gnu/libwayland-client.so.0.3.0
7ffff55af000-7ffff55b1000 r--p 0000d000 08:01 38544626                   /usr/lib/x86_64-linux-gnu/libwayland-client.so.0.3.0
7ffff55b1000-7ffff55b2000 rw-p 0000f000 08:01 38544626                   /usr/lib/x86_64-linux-gnu/libwayland-client.so.0.3.0
7ffff55b2000-7ffff55b3000 r--p 00000000 08:01 38543870                   /usr/lib/x86_64-linux-gnu/libwayland-egl.so.1.0.0
7ffff55b3000-7ffff55b4000 r-xp 00001000 08:01 38543870                   /usr/lib/x86_64-linux-gnu/libwayland-egl.so.1.0.0
7ffff55b4000-7ffff55b5000 r--p 00002000 08:01 38543870                   /usr/lib/x86_64-linux-gnu/libwayland-egl.so.1.0.0
7ffff55b5000-7ffff55b6000 r--p 00002000 08:01 38543870                   /usr/lib/x86_64-linux-gnu/libwayland-egl.so.1.0.0
7ffff55b6000-7ffff55b7000 rw-p 00003000 08:01 38543870                   /usr/lib/x86_64-linux-gnu/libwayland-egl.so.1.0.0
7ffff55b7000-7ffff55b8000 r--p 00000000 08:01 38544752                   /usr/lib/x86_64-linux-gnu/libwayland-cursor.so.0.0.0
7ffff55b8000-7ffff55ba000 r-xp 00001000 08:01 38544752                   /usr/lib/x86_64-linux-gnu/libwayland-cursor.so.0.0.0
7ffff55ba000-7ffff55bb000 r--p 00003000 08:01 38544752                   /usr/lib/x86_64-linux-gnu/libwayland-cursor.so.0.0.0
7ffff55bb000-7ffff55bc000 r--p 00003000 08:01 38544752                   /usr/lib/x86_64-linux-gnu/libwayland-cursor.so.0.0.0
7ffff55bc000-7ffff55c0000 rw-p 00004000 08:01 38544752                   /usr/lib/x86_64-linux-gnu/libwayland-cursor.so.0.0.0
7ffff55c0000-7ffff55c4000 r--p 00000000 08:01 38542212                   /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
7ffff55c4000-7ffff55df000 r-xp 00004000 08:01 38542212                   /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
7ffff55df000-7ffff55fd000 r--p 0001f000 08:01 38542212                   /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
7ffff55fd000-7ffff55fe000 ---p 0003d000 08:01 38542212                   /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
7ffff55fe000-7ffff5600000 r--p 0003d000 08:01 38542212                   /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
7ffff5600000-7ffff5601000 rw-p 0003f000 08:01 38542212                   /usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
7ffff5601000-7ffff5602000 r--p 00000000 08:01 38544613                   /usr/lib/x86_64-linux-gnu/libXdamage.so.1.1.0
7ffff5602000-7ffff5603000 r-xp 00001000 08:01 38544613                   /usr/lib/x86_64-linux-gnu/libXdamage.so.1.1.0
7ffff5603000-7ffff5604000 r--p 00002000 08:01 38544613                   /usr/lib/x86_64-linux-gnu/libXdamage.so.1.1.0
7ffff5604000-7ffff5605000 r--p 00002000 08:01 38544613                   /usr/lib/x86_64-linux-gnu/libXdamage.so.1.1.0
7ffff5605000-7ffff5606000 rw-p 00003000 08:01 38544613                   /usr/lib/x86_64-linux-gnu/libXdamage.so.1.1.0
7ffff5606000-7ffff5607000 r--p 00000000 08:01 38544839                   /usr/lib/x86_64-linux-gnu/libXcomposite.so.1.0.0
7ffff5607000-7ffff5608000 r-xp 00001000 08:01 38544839                   /usr/lib/x86_64-linux-gnu/libXcomposite.so.1.0.0
7ffff5608000-7ffff5609000 r--p 00002000 08:01 38544839                   /usr/lib/x86_64-linux-gnu/libXcomposite.so.1.0.0
7ffff5609000-7ffff560a000 r--p 00002000 08:01 38544839                   /usr/lib/x86_64-linux-gnu/libXcomposite.so.1.0.0
7ffff560a000-7ffff560b000 rw-p 00003000 08:01 38544839                   /usr/lib/x86_64-linux-gnu/libXcomposite.so.1.0.0
7ffff560b000-7ffff560e000 r--p 00000000 08:01 38542176                   /usr/lib/x86_64-linux-gnu/libXcursor.so.1.0.2
7ffff560e000-7ffff5614000 r-xp 00003000 08:01 38542176                   /usr/lib/x86_64-linux-gnu/libXcursor.so.1.0.2
7ffff5614000-7ffff5616000 r--p 00009000 08:01 38542176                   /usr/lib/x86_64-linux-gnu/libXcursor.so.1.0.2
7ffff5616000-7ffff5617000 r--p 0000a000 08:01 38542176                   /usr/lib/x86_64-linux-gnu/libXcursor.so.1.0.2
7ffff5617000-7ffff5618000 rw-p 0000b000 08:01 38542176                   /usr/lib/x86_64-linux-gnu/libXcursor.so.1.0.2
7ffff5618000-7ffff561a000 r--p 00000000 08:01 38542224                   /usr/lib/x86_64-linux-gnu/libXrandr.so.2.2.0
7ffff561a000-7ffff5621000 r-xp 00002000 08:01 38542224                   /usr/lib/x86_64-linux-gnu/libXrandr.so.2.2.0
7ffff5621000-7ffff5623000 r--p 00009000 08:01 38542224                   /usr/lib/x86_64-linux-gnu/libXrandr.so.2.2.0
7ffff5623000-7ffff5624000 r--p 0000a000 08:01 38542224                   /usr/lib/x86_64-linux-gnu/libXrandr.so.2.2.0
7ffff5624000-7ffff5625000 rw-p 0000b000 08:01 38542224                   /usr/lib/x86_64-linux-gnu/libXrandr.so.2.2.0
7ffff5625000-7ffff5626000 r--p 00000000 08:01 38536086                   /usr/lib/x86_64-linux-gnu/libXinerama.so.1.0.0
7ffff5626000-7ffff5627000 r-xp 00001000 08:01 38536086                   /usr/lib/x86_64-linux-gnu/libXinerama.so.1.0.0
7ffff5627000-7ffff5628000 r--p 00002000 08:01 38536086                   /usr/lib/x86_64-linux-gnu/libXinerama.so.1.0.0
7ffff5628000-7ffff5629000 r--p 00002000 08:01 38536086                   /usr/lib/x86_64-linux-gnu/libXinerama.so.1.0.0
7ffff5629000-7ffff562a000 rw-p 00003000 08:01 38536086                   /usr/lib/x86_64-linux-gnu/libXinerama.so.1.0.0
7ffff562a000-7ffff5636000 r--p 00000000 08:01 38539772                   /usr/lib/x86_64-linux-gnu/libfreetype.so.6.16.1
7ffff5636000-7ffff56b3000 r-xp 0000c000 08:01 38539772                   /usr/lib/x86_64-linux-gnu/libfreetype.so.6.16.1
7ffff56b3000-7ffff56dd000 r--p 00089000 08:01 38539772                   /usr/lib/x86_64-linux-gnu/libfreetype.so.6.16.1
7ffff56dd000-7ffff56e4000 r--p 000b2000 08:01 38539772                   /usr/lib/x86_64-linux-gnu/libfreetype.so.6.16.1
7ffff56e4000-7ffff56e5000 rw-p 000b9000 08:01 38539772                   /usr/lib/x86_64-linux-gnu/libfreetype.so.6.16.1
7ffff56e5000-7ffff56ec000 r--p 00000000 08:01 38545986                   /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.12.0
7ffff56ec000-7ffff5710000 r-xp 00007000 08:01 38545986                   /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.12.0
7ffff5710000-7ffff5728000 r--p 0002b000 08:01 38545986                   /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.12.0
7ffff5728000-7ffff572a000 r--p 00042000 08:01 38545986                   /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.12.0
7ffff572a000-7ffff572b000 rw-p 00044000 08:01 38545986                   /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.12.0
7ffff572b000-7ffff5747000 r--p 00000000 08:01 38544590                   /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.4
7ffff5747000-7ffff57ca000 r-xp 0001c000 08:01 38544590                   /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.4
7ffff57ca000-7ffff584f000 r--p 0009f000 08:01 38544590                   /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.4
7ffff584f000-7ffff5850000 ---p 00124000 08:01 38544590                   /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.4
7ffff5850000-7ffff5851000 r--p 00124000 08:01 38544590                   /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.4
7ffff5851000-7ffff5852000 rw-p 00125000 08:01 38544590                   /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.4
7ffff5852000-7ffff5853000 rw-p 00000000 00:00 0
7ffff5853000-7ffff5861000 r--p 00000000 08:01 38546453                   /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.4
7ffff5861000-7ffff5894000 r-xp 0000e000 08:01 38546453                   /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.4
7ffff5894000-7ffff58ac000 r--p 00041000 08:01 38546453                   /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.4
7ffff58ac000-7ffff58af000 r--p 00058000 08:01 38546453                   /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.4
7ffff58af000-7ffff58b0000 rw-p 0005b000 08:01 38546453                   /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.4
7ffff58b0000-7ffff58bd000 r--p 00000000 08:01 38542829                   /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0.4200.3
7ffff58bd000-7ffff58e1000 r-xp 0000d000 08:01 38542829                   /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0.4200.3
7ffff58e1000-7ffff58f7000 r--p 00031000 08:01 38542829                   /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0.4200.3
7ffff58f7000-7ffff58f8000 ---p 00047000 08:01 38542829                   /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0.4200.3
7ffff58f8000-7ffff58fb000 r--p 00047000 08:01 38542829                   /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0.4200.3
7ffff58fb000-7ffff58fc000 rw-p 0004a000 08:01 38542829                   /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0.4200.3
7ffff58fc000-7ffff5903000 r--p 00000000 08:01 38546490                   /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0.4200.3
7ffff5903000-7ffff590f000 r-xp 00007000 08:01 38546490                   /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0.4200.3
7ffff590f000-7ffff5913000 r--p 00013000 08:01 38546490                   /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0.4200.3
7ffff5913000-7ffff5914000 ---p 00017000 08:01 38546490                   /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0.4200.3
7ffff5914000-7ffff5915000 r--p 00017000 08:01 38546490                   /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0.4200.3
7ffff5915000-7ffff5916000 rw-p 00018000 08:01 38546490                   /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0.4200.3
7ffff5916000-7ffff591f000 r--p 00000000 08:01 38535890                   /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0.20600.1
7ffff591f000-7ffff59d9000 r-xp 00009000 08:01 38535890                   /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0.20600.1
7ffff59d9000-7ffff5a0a000 r--p 000c3000 08:01 38535890                   /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0.20600.1
7ffff5a0a000-7ffff5a0b000 ---p 000f4000 08:01 38535890                   /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0.20600.1
7ffff5a0b000-7ffff5a0c000 r--p 000f4000 08:01 38535890                   /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0.20600.1
7ffff5a0c000-7ffff5a0d000 rw-p 000f5000 08:01 38535890                   /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0.20600.1
7ffff5a0d000-7ffff5a44000 r--p 00000000 08:01 38536955                   /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.6200.4
7ffff5a44000-7ffff5b54000 r-xp 00037000 08:01 38536955                   /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.6200.4
7ffff5b54000-7ffff5bdc000 r--p 00147000 08:01 38536955                   /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.6200.4
7ffff5bdc000-7ffff5bdd000 ---p 001cf000 08:01 38536955                   /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.6200.4
7ffff5bdd000-7ffff5be5000 r--p 001cf000 08:01 38536955                   /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.6200.4
7ffff5be5000-7ffff5be6000 rw-p 001d7000 08:01 38536955                   /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.6200.4
7ffff5be6000-7ffff5be8000 rw-p 00000000 00:00 0
7ffff5be8000-7ffff5bea000 r--p 00000000 08:01 38542300                   /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0
7ffff5bea000-7ffff5bee000 r-xp 00002000 08:01 38542300                   /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0
7ffff5bee000-7ffff5c03000 r--p 00006000 08:01 38542300                   /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0
7ffff5c03000-7ffff5c04000 r--p 0001a000 08:01 38542300                   /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0
7ffff5c04000-7ffff5c05000 rw-p 0001b000 08:01 38542300                   /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0
7ffff5c05000-7ffff5c63000 r--p 00000000 08:01 38544850                   /usr/lib/x86_64-linux-gnu/libepoxy.so.0.0.0
7ffff5c63000-7ffff5cc7000 r-xp 0005e000 08:01 38544850                   /usr/lib/x86_64-linux-gnu/libepoxy.so.0.0.0
7ffff5cc7000-7ffff5d28000 r--p 000c2000 08:01 38544850                   /usr/lib/x86_64-linux-gnu/libepoxy.so.0.0.0
7ffff5d28000-7ffff5d30000 r--p 00122000 08:01 38544850                   /usr/lib/x86_64-linux-gnu/libepoxy.so.0.0.0
7ffff5d30000-7ffff5d37000 rw-p 0012a000 08:01 38544850                   /usr/lib/x86_64-linux-gnu/libepoxy.so.0.0.0
7ffff5d37000-7ffff5d43000 r--p 00000000 08:01 38537949                   /usr/lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0.0.0
7ffff5d43000-7ffff5d5b000 r-xp 0000c000 08:01 38537949                   /usr/lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0.0.0
7ffff5d5b000-7ffff5d69000 r--p 00024000 08:01 38537949                   /usr/lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0.0.0
7ffff5d69000-7ffff5d6a000 ---p 00032000 08:01 38537949                   /usr/lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0.0.0
7ffff5d6a000-7ffff5d6b000 r--p 00032000 08:01 38537949                   /usr/lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0.0.0
7ffff5d6b000-7ffff5d6c000 rw-p 00033000 08:01 38537949                   /usr/lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0.0.0
7ffff5d6c000-7ffff5d6d000 rw-p 00000000 00:00 0
7ffff5d6d000-7ffff5d78000 r--p 00000000 08:01 38535336                   /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0.23312.1
7ffff5d78000-7ffff5d87000 r-xp 0000b000 08:01 38535336                   /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0.23312.1
7ffff5d87000-7ffff5d92000 r--p 0001a000 08:01 38535336                   /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0.23312.1
7ffff5d92000-7ffff5d93000 ---p 00025000 08:01 38535336                   /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0.23312.1
7ffff5d93000-7ffff5d96000 r--p 00025000 08:01 38535336                   /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0.23312.1
7ffff5d96000-7ffff5d97000 rw-p 00028000 08:01 38535336                   /usr/lib/x86_64-linux-gnu/libatk-1.0.so.0.23312.1
7ffff5d97000-7ffff5d9d000 r--p 00000000 08:01 38536617                   /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0.4000.0
7ffff5d9d000-7ffff5db4000 r-xp 00006000 08:01 38536617                   /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0.4000.0
7ffff5db4000-7ffff5dbd000 r--p 0001d000 08:01 38536617                   /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0.4000.0
7ffff5dbd000-7ffff5dbe000 r--p 00025000 08:01 38536617                   /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0.4000.0
7ffff5dbe000-7ffff5dbf000 rw-p 00026000 08:01 38536617                   /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0.4000.0
7ffff5dbf000-7ffff5dd1000 r--p 00000000 08:01 38546420                   /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0
7ffff5dd1000-7ffff5e9d000 r-xp 00012000 08:01 38546420                   /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0
7ffff5e9d000-7ffff5ed9000 r--p 000de000 08:01 38546420                   /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0
7ffff5ed9000-7ffff5edd000 r--p 00119000 08:01 38546420                   /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0
7ffff5edd000-7ffff5ede000 rw-p 0011d000 08:01 38546420                   /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0
7ffff5ede000-7ffff5edf000 rw-p 00000000 00:00 0
7ffff5edf000-7ffff5ee4000 r-xp 00000000 08:01 38542758                   /usr/lib/x86_64-linux-gnu/libXfixes.so.3.1.0
7ffff5ee4000-7ffff60e3000 ---p 00005000 08:01 38542758                   /usr/lib/x86_64-linux-gnu/libXfixes.so.3.1.0
7ffff60e3000-7ffff60e4000 r--p 00004000 08:01 38542758                   /usr/lib/x86_64-linux-gnu/libXfixes.so.3.1.0
7ffff60e4000-7ffff60e5000 rw-p 00005000 08:01 38542758                   /usr/lib/x86_64-linux-gnu/libXfixes.so.3.1.0
7ffff60e5000-7ffff60e7000 r--p 00000000 08:01 38555508                   /usr/lib/x86_64-linux-gnu/libXi.so.6.1.0
7ffff60e7000-7ffff60f2000 r-xp 00002000 08:01 38555508                   /usr/lib/x86_64-linux-gnu/libXi.so.6.1.0
7ffff60f2000-7ffff60f5000 r--p 0000d000 08:01 38555508                   /usr/lib/x86_64-linux-gnu/libXi.so.6.1.0
7ffff60f5000-7ffff60f6000 r--p 0000f000 08:01 38555508                   /usr/lib/x86_64-linux-gnu/libXi.so.6.1.0
7ffff60f6000-7ffff60f7000 rw-p 00010000 08:01 38555508                   /usr/lib/x86_64-linux-gnu/libXi.so.6.1.0
7ffff60f7000-7ffff6110000 r--p 00000000 08:01 38545329                   /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
7ffff6110000-7ffff619b000 r-xp 00019000 08:01 38545329                   /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
7ffff619b000-7ffff622f000 r--p 000a4000 08:01 38545329                   /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
7ffff622f000-7ffff6230000 ---p 00138000 08:01 38545329                   /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
7ffff6230000-7ffff6231000 r--p 00138000 08:01 38545329                   /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
7ffff6231000-7ffff6235000 rw-p 00139000 08:01 38545329                   /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
7ffff6235000-7ffff625f000 r--p 00000000 08:01 38537075                   /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.8
7ffff625f000-7ffff62df000 r-xp 0002a000 08:01 38537075                   /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.8
7ffff62df000-7ffff632e000 r--p 000aa000 08:01 38537075                   /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.8
7ffff632e000-7ffff6337000 r--p 000f8000 08:01 38537075                   /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.8
7ffff6337000-7ffff6339000 rw-p 00101000 08:01 38537075                   /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.8
7ffff6339000-7ffff63be000 r--p 00000000 08:01 38537300                   /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.8
7ffff63be000-7ffff672c000 r-xp 00085000 08:01 38537300                   /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.8
7ffff672c000-7ffff6a33000 r--p 003f3000 08:01 38537300                   /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.8
7ffff6a33000-7ffff6a34000 ---p 006fa000 08:01 38537300                   /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.8
7ffff6a34000-7ffff6a44000 r--p 006fa000 08:01 38537300                   /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.8
7ffff6a44000-7ffff6a46000 rw-p 0070a000 08:01 38537300                   /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.8
7ffff6a46000-7ffff6a4c000 rw-p 00000000 00:00 0
7ffff6a4c000-7ffff6a4f000 r--p 00000000 08:01 40513282                   /usr/share/locale-langpack/en_CA/LC_MESSAGES/gtk30-properties.mo
7ffff6a4f000-7ffff6a58000 r--p 00000000 08:01 38545428                   /usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2.3.4
7ffff6a58000-7ffff6a6c000 r-xp 00009000 08:01 38545428                   /usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2.3.4
7ffff6a6c000-7ffff6a75000 r--p 0001d000 08:01 38545428                   /usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2.3.4
7ffff6a75000-7ffff6a77000 r--p 00025000 08:01 38545428                   /usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2.3.4
7ffff6a77000-7ffff6a78000 rw-p 00027000 08:01 38545428                   /usr/lib/x86_64-linux-gnu/libdbus-glib-1.so.2.3.4
7ffff6a78000-7ffff6a79000 r--p 00000000 08:01 38535536                   /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1.0.0
7ffff6a79000-7ffff6a7a000 r-xp 00001000 08:01 38535536                   /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1.0.0
7ffff6a7a000-7ffff6a7b000 r--p 00002000 08:01 38535536                   /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1.0.0
7ffff6a7b000-7ffff6a7c000 r--p 00002000 08:01 38535536                   /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1.0.0
7ffff6a7c000-7ffff6a7d000 rw-p 00003000 08:01 38535536                   /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1.0.0
7ffff6a7d000-7ffff6a7f000 r--p 00000000 08:01 92102535                   /home/mstange/code/obj-m-debuig/widget/gtk/mozwayland/libmozwayland.so
7ffff6a7f000-7ffff6a80000 r-xp 00002000 08:01 92102535                   /home/mstange/code/obj-m-debuig/widget/gtk/mozwayland/libmozwayland.so
7ffff6a80000-7ffff6a81000 r--p 00003000 08:01 92102535                   /home/mstange/code/obj-m-debuig/widget/gtk/mozwayland/libmozwayland.so
7ffff6a81000-7ffff6a82000 rw-p 00004000 08:01 92102535                   /home/mstange/code/obj-m-debuig/widget/gtk/mozwayland/libmozwayland.so
7ffff6a82000-7ffff6aad000 r--p 00000000 08:01 91831726                   /home/mstange/code/obj-m-debuig/security/nss/lib/ssl/ssl_ssl3/libssl3.so
7ffff6aad000-7ffff6b2b000 r-xp 0002b000 08:01 91831726                   /home/mstange/code/obj-m-debuig/security/nss/lib/ssl/ssl_ssl3/libssl3.so
7ffff6b2b000-7ffff6b2f000 r--p 000a9000 08:01 91831726                   /home/mstange/code/obj-m-debuig/security/nss/lib/ssl/ssl_ssl3/libssl3.so
7ffff6b2f000-7ffff6b30000 rw-p 000ad000 08:01 91831726                   /home/mstange/code/obj-m-debuig/security/nss/lib/ssl/ssl_ssl3/libssl3.so
7ffff6b30000-7ffff6b31000 rw-p 00000000 00:00 0
7ffff6b31000-7ffff6b98000 r--p 00000000 08:01 91831634                   /home/mstange/code/obj-m-debuig/config/external/sqlite/libmozsqlite3.so
7ffff6b98000-7ffff6cf1000 r-xp 00067000 08:01 91831634                   /home/mstange/code/obj-m-debuig/config/external/sqlite/libmozsqlite3.so
7ffff6cf1000-7ffff6cf5000 r--p 001c0000 08:01 91831634                   /home/mstange/code/obj-m-debuig/config/external/sqlite/libmozsqlite3.so
7ffff6cf5000-7ffff6cf8000 rw-p 001c4000 08:01 91831634                   /home/mstange/code/obj-m-debuig/config/external/sqlite/libmozsqlite3.so
7ffff6cf8000-7ffff6cf9000 rw-p 00000000 00:00 0
7ffff6cf9000-7ffff6d09000 r--p 00000000 08:01 91831719                   /home/mstange/code/obj-m-debuig/security/nss/lib/smime/smime_smime3/libsmime3.so
7ffff6d09000-7ffff6d37000 r-xp 00010000 08:01 91831719                   /home/mstange/code/obj-m-debuig/security/nss/lib/smime/smime_smime3/libsmime3.so
7ffff6d37000-7ffff6d3a000 r--p 0003e000 08:01 91831719                   /home/mstange/code/obj-m-debuig/security/nss/lib/smime/smime_smime3/libsmime3.so
7ffff6d3a000-7ffff6d3b000 rw-p 00041000 08:01 91831719                   /home/mstange/code/obj-m-debuig/security/nss/lib/smime/smime_smime3/libsmime3.so
7ffff6d3b000-7ffff6d74000 r--p 00000000 08:01 91831711                   /home/mstange/code/obj-m-debuig/security/nss/lib/nss/nss_nss3/libnss3.so
7ffff6d74000-7ffff6e22000 r-xp 00039000 08:01 91831711                   /home/mstange/code/obj-m-debuig/security/nss/lib/nss/nss_nss3/libnss3.so
7ffff6e22000-7ffff6e27000 r--p 000e7000 08:01 91831711                   /home/mstange/code/obj-m-debuig/security/nss/lib/nss/nss_nss3/libnss3.so
7ffff6e27000-7ffff6e29000 rw-p 000ec000 08:01 91831711                   /home/mstange/code/obj-m-debuig/security/nss/lib/nss/nss_nss3/libnss3.so
7ffff6e29000-7ffff6e42000 r--p 00000000 08:01 91831688                   /home/mstange/code/obj-m-debuig/security/nss/lib/util/util_nssutil3/libnssutil3.so
7ffff6e42000-7ffff6e62000 r-xp 00019000 08:01 91831688                   /home/mstange/code/obj-m-debuig/security/nss/lib/util/util_nssutil3/libnssutil3.so
7ffff6e62000-7ffff6e68000 r--p 00039000 08:01 91831688                   /home/mstange/code/obj-m-debuig/security/nss/lib/util/util_nssutil3/libnssutil3.so
7ffff6e68000-7ffff6e69000 rw-p 0003f000 08:01 91831688                   /home/mstange/code/obj-m-debuig/security/nss/lib/util/util_nssutil3/libnssutil3.so
7ffff6e69000-7ffff6eb2000 r--p 00000000 08:01 91831683                   /home/mstange/code/obj-m-debuig/security/sandbox/linux/libmozsandbox.so
7ffff6eb2000-7ffff6f22000 r-xp 00049000 08:01 91831683                   /home/mstange/code/obj-m-debuig/security/sandbox/linux/libmozsandbox.so
7ffff6f22000-7ffff6f23000 r--p 000b9000 08:01 91831683                   /home/mstange/code/obj-m-debuig/security/sandbox/linux/libmozsandbox.so
7ffff6f23000-7ffff6f24000 rw-p 000ba000 08:01 91831683                   /home/mstange/code/obj-m-debuig/security/sandbox/linux/libmozsandbox.so
7ffff6f24000-7ffff6f26000 rw-p 00000000 00:00 0
7ffff6f26000-7ffff6f3d000 r--p 00000000 08:01 91831672                   /home/mstange/code/obj-m-debuig/config/external/nspr/pr/libnspr4.so
7ffff6f3d000-7ffff6f79000 r-xp 00017000 08:01 91831672                   /home/mstange/code/obj-m-debuig/config/external/nspr/pr/libnspr4.so
7ffff6f79000-7ffff6f7a000 r--p 00053000 08:01 91831672                   /home/mstange/code/obj-m-debuig/config/external/nspr/pr/libnspr4.so
7ffff6f7a000-7ffff6f7c000 rw-p 00054000 08:01 91831672                   /home/mstange/code/obj-m-debuig/config/external/nspr/pr/libnspr4.so
7ffff6f7c000-7ffff6f7e000 rw-p 00000000 00:00 0
7ffff6f7e000-7ffff6f7f000 ---p 00000000 00:00 0
7ffff6f7f000-7ffff777f000 rw-p 00000000 00:00 0
7ffff777f000-7ffff7800000 ---p 00000000 00:00 0
7ffff7800000-7ffff78ff000 rw-p 00000000 00:00 0
7ffff78ff000-7ffff7900000 ---p 00000000 00:00 0
7ffff7900000-7ffff7903000 rw-p 00000000 00:00 0
7ffff7903000-7ffff7a00000 ---p 00000000 00:00 0
7ffff7a00000-7ffff7a01000 r--p 00000000 08:01 40509511                   /usr/share/locale-langpack/en_CA/LC_MESSAGES/atk10.mo
7ffff7a01000-7ffff7a05000 r--p 00000000 08:01 38547205                   /usr/lib/x86_64-linux-gnu/libcairo-gobject.so.2.11600.0
7ffff7a05000-7ffff7a07000 r-xp 00004000 08:01 38547205                   /usr/lib/x86_64-linux-gnu/libcairo-gobject.so.2.11600.0
7ffff7a07000-7ffff7a09000 r--p 00006000 08:01 38547205                   /usr/lib/x86_64-linux-gnu/libcairo-gobject.so.2.11600.0
7ffff7a09000-7ffff7a0a000 ---p 00008000 08:01 38547205                   /usr/lib/x86_64-linux-gnu/libcairo-gobject.so.2.11600.0
7ffff7a0a000-7ffff7a0c000 r--p 00008000 08:01 38547205                   /usr/lib/x86_64-linux-gnu/libcairo-gobject.so.2.11600.0
7ffff7a0c000-7ffff7a0d000 rw-p 0000a000 08:01 38547205                   /usr/lib/x86_64-linux-gnu/libcairo-gobject.so.2.11600.0
7ffff7a0d000-7ffff7a12000 r--p 00000000 08:01 38546986                   /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0.4200.3
7ffff7a12000-7ffff7a19000 r-xp 00005000 08:01 38546986                   /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0.4200.3
7ffff7a19000-7ffff7a1c000 r--p 0000c000 08:01 38546986                   /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0.4200.3
7ffff7a1c000-7ffff7a1d000 r--p 0000e000 08:01 38546986                   /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0.4200.3
7ffff7a1d000-7ffff7a1e000 rw-p 0000f000 08:01 38546986                   /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0.4200.3
7ffff7a1e000-7ffff7a23000 rw-p 00000000 00:00 0
7ffff7a23000-7ffff7a48000 r--p 00000000 08:01 105386469                  /lib/x86_64-linux-gnu/libc-2.30.so
7ffff7a48000-7ffff7bc0000 r-xp 00025000 08:01 105386469                  /lib/x86_64-linux-gnu/libc-2.30.so
7ffff7bc0000-7ffff7c0a000 r--p 0019d000 08:01 105386469                  /lib/x86_64-linux-gnu/libc-2.30.so
7ffff7c0a000-7ffff7c0d000 r--p 001e6000 08:01 105386469                  /lib/x86_64-linux-gnu/libc-2.30.so
7ffff7c0d000-7ffff7c10000 rw-p 001e9000 08:01 105386469                  /lib/x86_64-linux-gnu/libc-2.30.so
7ffff7c10000-7ffff7c14000 rw-p 00000000 00:00 0
7ffff7c14000-7ffff7c17000 r--p 00000000 08:01 105392299                  /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7c17000-7ffff7c28000 r-xp 00003000 08:01 105392299                  /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7c28000-7ffff7c2c000 r--p 00014000 08:01 105392299                  /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7c2c000-7ffff7c2d000 r--p 00017000 08:01 105392299                  /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7c2d000-7ffff7c2e000 rw-p 00018000 08:01 105392299                  /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7c2e000-7ffff7c3d000 r--p 00000000 08:01 105392460                  /lib/x86_64-linux-gnu/libm-2.30.so
7ffff7c3d000-7ffff7ce4000 r-xp 0000f000 08:01 105392460                  /lib/x86_64-linux-gnu/libm-2.30.so
7ffff7ce4000-7ffff7d7b000 r--p 000b6000 08:01 105392460                  /lib/x86_64-linux-gnu/libm-2.30.so
7ffff7d7b000-7ffff7d7c000 r--p 0014c000 08:01 105392460                  /lib/x86_64-linux-gnu/libm-2.30.so
7ffff7d7c000-7ffff7d7d000 rw-p 0014d000 08:01 105392460                  /lib/x86_64-linux-gnu/libm-2.30.so
7ffff7d7d000-7ffff7e13000 r--p 00000000 08:01 38537196                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7ffff7e13000-7ffff7f10000 r-xp 00096000 08:01 38537196                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7ffff7f10000-7ffff7f5a000 r--p 00193000 08:01 38537196                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7ffff7f5a000-7ffff7f65000 r--p 001dc000 08:01 38537196                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7ffff7f65000-7ffff7f68000 rw-p 001e7000 08:01 38537196                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7ffff7f68000-7ffff7f6b000 rw-p 00000000 00:00 0
7ffff7f6b000-7ffff7f6c000 r--p 00000000 08:01 105390471                  /lib/x86_64-linux-gnu/libdl-2.30.so
7ffff7f6c000-7ffff7f6e000 r-xp 00001000 08:01 105390471                  /lib/x86_64-linux-gnu/libdl-2.30.so
7ffff7f6e000-7ffff7f6f000 r--p 00003000 08:01 105390471                  /lib/x86_64-linux-gnu/libdl-2.30.so
7ffff7f6f000-7ffff7f70000 r--p 00003000 08:01 105390471                  /lib/x86_64-linux-gnu/libdl-2.30.so
7ffff7f70000-7ffff7f71000 rw-p 00004000 08:01 105390471                  /lib/x86_64-linux-gnu/libdl-2.30.so
7ffff7f71000-7ffff7f78000 r--p 00000000 08:01 105392481                  /lib/x86_64-linux-gnu/libpthread-2.30.so
7ffff7f78000-7ffff7f89000 r-xp 00007000 08:01 105392481                  /lib/x86_64-linux-gnu/libpthread-2.30.so
7ffff7f89000-7ffff7f8e000 r--p 00018000 08:01 105392481                  /lib/x86_64-linux-gnu/libpthread-2.30.so
7ffff7f8e000-7ffff7f8f000 r--p 0001c000 08:01 105392481                  /lib/x86_64-linux-gnu/libpthread-2.30.so
7ffff7f8f000-7ffff7f90000 rw-p 0001d000 08:01 105392481                  /lib/x86_64-linux-gnu/libpthread-2.30.so
7ffff7f90000-7ffff7f96000 rw-p 00000000 00:00 0
7ffff7f96000-7ffff7f97000 r--p 00000000 08:01 40502830                   /usr/share/locale-langpack/en/LC_MESSAGES/gtk30.mo
7ffff7f97000-7ffff7f98000 r--p 00000000 08:01 38546443                   /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.4
7ffff7f98000-7ffff7f9a000 r-xp 00001000 08:01 38546443                   /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.4
7ffff7f9a000-7ffff7f9b000 r--p 00003000 08:01 38546443                   /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.4
7ffff7f9b000-7ffff7f9c000 r--p 00003000 08:01 38546443                   /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.4
7ffff7f9c000-7ffff7f9d000 rw-p 00004000 08:01 38546443                   /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.4
7ffff7f9d000-7ffff7f9e000 r--p 00000000 08:01 92102524                   /home/mstange/code/obj-m-debuig/widget/gtk/mozgtk/gtk3/libmozgtk.so
7ffff7f9e000-7ffff7f9f000 r-xp 00001000 08:01 92102524                   /home/mstange/code/obj-m-debuig/widget/gtk/mozgtk/gtk3/libmozgtk.so
7ffff7f9f000-7ffff7fa0000 r--p 00002000 08:01 92102524                   /home/mstange/code/obj-m-debuig/widget/gtk/mozgtk/gtk3/libmozgtk.so
7ffff7fa0000-7ffff7fa1000 rw-p 00003000 08:01 92102524                   /home/mstange/code/obj-m-debuig/widget/gtk/mozgtk/gtk3/libmozgtk.so
7ffff7fa1000-7ffff7fa8000 r--p 00000000 08:01 91831632                   /home/mstange/code/obj-m-debuig/config/external/lgpllibs/liblgpllibs.so
7ffff7fa8000-7ffff7fb1000 r-xp 00007000 08:01 91831632                   /home/mstange/code/obj-m-debuig/config/external/lgpllibs/liblgpllibs.so
7ffff7fb1000-7ffff7fb2000 r--p 00010000 08:01 91831632                   /home/mstange/code/obj-m-debuig/config/external/lgpllibs/liblgpllibs.so
7ffff7fb2000-7ffff7fb3000 rw-p 00011000 08:01 91831632                   /home/mstange/code/obj-m-debuig/config/external/lgpllibs/liblgpllibs.so
7ffff7fb3000-7ffff7fb6000 r--p 00000000 08:01 105392483                  /lib/x86_64-linux-gnu/librt-2.30.so
7ffff7fb6000-7ffff7fba000 r-xp 00003000 08:01 105392483                  /lib/x86_64-linux-gnu/librt-2.30.so
7ffff7fba000-7ffff7fbc000 r--p 00007000 08:01 105392483                  /lib/x86_64-linux-gnu/librt-2.30.so
7ffff7fbc000-7ffff7fbd000 r--p 00008000 08:01 105392483                  /lib/x86_64-linux-gnu/librt-2.30.so
7ffff7fbd000-7ffff7fbe000 rw-p 00009000 08:01 105392483                  /lib/x86_64-linux-gnu/librt-2.30.so
7ffff7fbe000-7ffff7fc0000 r--p 00000000 08:01 92102576                   /home/mstange/code/obj-m-debuig/config/external/nspr/ds/libplds4.so
7ffff7fc0000-7ffff7fc2000 r-xp 00002000 08:01 92102576                   /home/mstange/code/obj-m-debuig/config/external/nspr/ds/libplds4.so
7ffff7fc2000-7ffff7fc3000 r--p 00004000 08:01 92102576                   /home/mstange/code/obj-m-debuig/config/external/nspr/ds/libplds4.so
7ffff7fc3000-7ffff7fc4000 rw-p 00005000 08:01 92102576                   /home/mstange/code/obj-m-debuig/config/external/nspr/ds/libplds4.so
7ffff7fc4000-7ffff7fc6000 r--p 00000000 08:01 92102578                   /home/mstange/code/obj-m-debuig/config/external/nspr/libc/libplc4.so
7ffff7fc6000-7ffff7fca000 r-xp 00002000 08:01 92102578                   /home/mstange/code/obj-m-debuig/config/external/nspr/libc/libplc4.so
7ffff7fca000-7ffff7fcb000 r--p 00006000 08:01 92102578                   /home/mstange/code/obj-m-debuig/config/external/nspr/libc/libplc4.so
7ffff7fcb000-7ffff7fcc000 rw-p 00007000 08:01 92102578                   /home/mstange/code/obj-m-debuig/config/external/nspr/libc/libplc4.so
7ffff7fcc000-7ffff7fcf000 r--p 00000000 00:00 0                          [vvar]
7ffff7fcf000-7ffff7fd0000 r-xp 00000000 00:00 0                          [vdso]
7ffff7fd0000-7ffff7fd1000 r--p 00000000 08:01 105381906                  /lib/x86_64-linux-gnu/ld-2.30.so
7ffff7fd1000-7ffff7ff3000 r-xp 00001000 08:01 105381906                  /lib/x86_64-linux-gnu/ld-2.30.so
7ffff7ff3000-7ffff7ffb000 r--p 00023000 08:01 105381906                  /lib/x86_64-linux-gnu/ld-2.30.so
7ffff7ffb000-7ffff7ffc000 r--p 00000000 08:01 40513285                   /usr/share/locale-langpack/en_CA/LC_MESSAGES/gtk30.mo
7ffff7ffc000-7ffff7ffd000 r--p 0002b000 08:01 105381906                  /lib/x86_64-linux-gnu/ld-2.30.so
7ffff7ffd000-7ffff7ffe000 rw-p 0002c000 08:01 105381906                  /lib/x86_64-linux-gnu/ld-2.30.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
mstange@hawkman:~$
mstange commented 4 years ago

(The prelude to this was https://gist.github.com/mstange/9d14bea3f34452d62d63458a588e0fb4)

mstange commented 4 years ago

another thing to do would be to repeat what you did in #2417 (comment)

Just to confirm: This was for the call to open, right?

and instead of printing the file name, get a stack trace. Then, re-run under gdb (no rr) setting a breakpoint somewhere so we can actually observe that file open outside of rr. Then stat /proc/fd/<opened-fd> and cat /proc/<pid>/maps. And also stat <actual file name>.

I got those three things right after the open call. Let me know if you want them at a different point.

rocallahan commented 4 years ago

Just to confirm: This was for the call to open, right?

The second call to open (actually openat) that opens the data.mdb file, yes.

From the absence of is_mapped_shared is shared in your rr log, I deduce is_mapped_shared is returning false. I cannot explain that; the device and inode numbers for the data.mdb file, via the proc-fd symlink, match what's in /proc/.../maps.

So Markus, this is going to suck a little bit, but you could make further progress by running rr record -n under gdb, Set a breakpoint in handle_opened_file after we get pathname. Cobble together a condition that checks for data.mdb at the end of the path. In the second such call, check /proc/.../maps and /proc/.../fd/... to verify that the opened fd appears to be pointing to that file already mapped, and that the device/inode numbers match what's in st in handle_opened_file, and then step through is_mapped_shared to see why it's returning false, i.e. failing to notice that map entry.

rocallahan commented 4 years ago

It's possible, albeit unlikely, that as we iterate over the maps we miss the map of that particular file, because our cache is wrong somehow.

rocallahan commented 4 years ago

Of course you could just add tons of logging to is_mmapped_shared and debug the voluminous log output.

mstange commented 4 years ago

Thanks! I'm planning to do that but I'm not sure when yet.