non-debug build still refer to .debug files, while there is no symlink for it in target dir "build/$arch$/bin/"

[tor-m6]

in the commit https://github.com/genodelabs/genode/commit/b41df1fb7a09faa68f261726977b642f9306ead1 it was introduced the reference to separate debug file in the build. Inside build/$arch$/debug directory it creates 2 separate links: to the stripped file and to the debug info file:

build/x86_64/debug/ld-linux.lib.so -> ... build/x86_64/var/libcache/ld-linux/ld-linux.lib.so.stripped
build/x86_64/debug/ld-linux.lib.so.debug -> ... build/x86_64/var/libcache/ld-linux/ld-linux.lib.so.debug

but during the "release" build it place only first link into "bin" dir, like

build/x86_64/bin/ld-linux.lib.so -> ... build/x86_64/var/libcache/ld-linux/ld-linux.lib.so.stripped

For applications and code which use stack unwinding (and should access to related binaries), like golang, process fail during runtime in attempt to enumerate every binary. This is because there are still link to debug file in .stripped one, while it is not available for release version.

I see 3 options to fix a problem

1. link .debug file into "bin" directory as well
2. generate 2 stripped files - without and with debug link (and link it accordingly to "bin" and "debug" dirs).
3. link non-stripped file $(TARGET) to "bin" (contradict whole idea of patch above)

[nfeske]

@tor-m6 linking .debug files into bin would not work because all content of bin/ ends up being integrated in the binary archives. We want to keep those archive free from the overhead of debug infos.

For shipping the debug information, @cproc has enabled the creation of dbg archives via the tool/depot/create and publish tools (specify dbg instead of bin), and also equipped the on-target depot_download subsystem to handle the installation of such archives. So there are already mechanisms in place to create/publish/download debug infos via the depot tools. The question seems to be how to best apply them.

Should the tools not work for you, we'd need to add a hook to the build system that allows for optionally skipping the stripping for certain targets. But I'd prefer to avoid that.

@cproc may you chime in about how to best apply the tools in the case of @tor-m6 ?

[tor-m6]

@nfeske IMHO we can have simply 2 stripped binaries - same size, one with reference inside to .debug file (as now), and second one completely stripped without reference to .debug

Just omit from call to OBJCOPY reference to flag --add-gnu-debuglink

[chelmuth]

@tor-m6 Do I get you right that golang works without debug information in your cases but not if gnu-debuglink is set but the referenced file is not available? Wouldn't it make more sense to fallback to "no debug info" in this case?

[tor-m6]

@tor-m6 Do I get you right that golang works without debug information in your cases but not if gnu-debuglink is set but the referenced file is not available? Wouldn't it make more sense to fallback to "no debug info" in this case?

@chelmuth golang work without debug info, this is correct. But, libbacktrace from gcc do try to enumerate all binaries referenced (just take their names from the file, obviously for such files as .so this is not anyhow related to debug info), and, if it found in a list of associated binaries another one (in our case .debug file) it try to open it. And fail (double fail, because it try to drop C++ exception/panic which again try to call libbacktrace, internally call loader and recursively call for mutex to protect this enumeration).

So, if we have this flag --add-gnu-debuglink during creation of stripped binary - then libbacktrace (not golang!) found it and request it (try to open by name, try to resolve symlink in Genode open() and try to throw exception - "not found"). Not anyhow related to debug info.

I checked that if I just omit this flag for stripped binaries - everything works correctly.

Problem that this is normal behaviour of gcc libbacktrace in combination with Genode loader. This is very complex frame-processing engine and I prefer not to modify it. I cant force it to fallback and work only with provided debug info/etc - this is more generic mechanism.

In theory, one can generate empty debug file for every such a case and link it to "bin" and to overall genode run process, while this is not seems elegant solution.

[cproc]

Commit a59f73f replaces the Symlink_resolve_error exception by a Genode::Attempt return type, which avoids the recursive dl_iterate_phdr() call and deadlock in go_app without rtc.

@tor-m6: do you think we can keep the debug link with this solution?

[cproc]

For reference, this is the deadlock backtrace I got on base-linux:

#0  pseudo_end () at /.../repos/base-linux/src/lib/syscall/spec/x86_64/lx_syscall.S:29
#1  0x0000000050049ffe in lx_futex (val=0, op=0, uaddr=<optimized out>) at /.../repos/base-linux/src/lib/syscall/linux_syscalls.h:432
#2  thread_stop_myself (myself=<optimized out>) at /.../repos/base-linux/src/include/base/internal/lock_helper.h:62
#3  Genode::Lock::lock (this=this@entry=0x501b2510 <Linker::Object::_object_list()::_list+16>, myself=...) at /.../repos/base/src/lib/base/lock.cc:128
#4  0x000000005004a4c6 in Genode::Mutex::acquire (this=this@entry=0x501b2510 <Linker::Object::_object_list()::_list+16>) at /.../repos/base/src/lib/base/mutex.cc:25
#5  0x00000000500a8437 in Genode::Mutex::Guard::Guard (mutex=..., this=<synthetic pointer>) at /.../repos/base/include/base/mutex.h:62
#6  Linker::Object::Object_list::for_each<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Linker::Object&)> > (func=..., this=0x501b2500 <Linker::Object::_object_list()::_list>) at /.../repos/base/src/lib/ldso/include/linker.h:203
#7  operator() (list=..., __closure=<optimized out>) at /.../repos/base/src/lib/ldso/include/linker.h:274
#8  Linker::Object::with_object_list<Linker::for_each_object<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Object&)> >(const dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Object&)>&)::<lambda(Linker::Object::Object_list&)> > (func=...) at /.../repos/base/src/lib/ldso/include/linker.h:211
#9  Linker::for_each_object<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Linker::Object&)> > (func=...) at /.../repos/base/src/lib/ldso/include/linker.h:273
#10 dl_iterate_phdr (callback=callback@entry=0x500963f0 <_Unwind_IteratePhdrCallback>, data=data@entry=0x407fbc70) at /.../repos/base/src/lib/ldso/exception.cc:42
#11 0x0000000050097c3e in _Unwind_Find_FDE (pc=0x10f25348 <Libc::resolve_symlinks(char const*, Genode::Path<1024u>&)+2664>, bases=0x407fbed8) at /.../contrib/gcc-093a037675b45e9b2aaddc75281f560f3b9dfe22/src/noux-pkg/gcc/libgcc/unwind-dw2-fde-dip.c:531
#12 0x0000000050094ef3 in uw_frame_state_for (context=context@entry=0x407fbe30, fs=fs@entry=0x407fbf20) at /.../contrib/gcc-093a037675b45e9b2aaddc75281f560f3b9dfe22/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1263
#13 0x0000000050095ba4 in _Unwind_RaiseException (exc=0x7ffff7ffad28) at /.../contrib/gcc-093a037675b45e9b2aaddc75281f560f3b9dfe22/src/noux-pkg/gcc/libgcc/unwind.inc:104
#14 0x000000005009d46a in __cxa_throw () from ld.lib.so
#15 0x0000000010f25349 in Libc::resolve_symlinks (path=<optimized out>, resolved_path=...) at /.../repos/libports/src/lib/libc/file_operations.cc:155
#16 0x0000000010f26837 in __sys_open (pathname=pathname@entry=0x7fff95c32198 "ld-linux.lib.so.debug", flags=flags@entry=1048576) at /.../repos/os/include/os/path.h:264
#17 0x00000000014566df in backtrace_open (filename=filename@entry=0x7fff95c32198 "ld-linux.lib.so.debug", error_callback=error_callback@entry=0x1002750 <error_callback>, data=data@entry=0x407fe6c0, does_not_exist=does_not_exist@entry=0x407fd80c) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/posix.c:67
#18 0x0000000001450470 in elf_try_debugfile (state=state@entry=0x7fff95c47000, prefix=prefix@entry=0x1471298 "", prefix_len=prefix_len@entry=0, prefix2=prefix2@entry=0x1471298 "", prefix2_len=prefix2_len@entry=0, debuglink_name=debuglink_name@entry=0x7fff95c4cbc4 "ld-linux.lib.so.debug", error_callback=0x1002750 <error_callback>, data=0x407fe6c0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/elf.c:939
#19 0x0000000001450683 in elf_find_debugfile_by_debuglink (state=state@entry=0x7fff95c47000, filename=<optimized out>, filename@entry=0x501b2000 <Linker::Ld::linker()::_linker+32> "ld.lib.so", debuglink_name=debuglink_name@entry=0x7fff95c4cbc4 "ld-linux.lib.so.debug", error_callback=error_callback@entry=0x1002750 <error_callback>, data=data@entry=0x407fe6c0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/elf.c:1028
#20 0x0000000001454172 in elf_open_debugfile_by_debuglink (data=0x407fe6c0, error_callback=0x1002750 <error_callback>, debuglink_crc=322094962, debuglink_name=0x7fff95c4cbc4 "ld-linux.lib.so.debug", filename=0x501b2000 <Linker::Ld::linker()::_linker+32> "ld.lib.so", state=0x7fff95c47000) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/elf.c:1075
#21 elf_add (state=<optimized out>, filename=filename@entry=0x501b2000 <Linker::Ld::linker()::_linker+32> "ld.lib.so", descriptor=3, memory=memory@entry=0x0, memory_size=memory_size@entry=0, base_address=0, error_callback=0x1002750 <error_callback>, data=0x407fe6c0, fileline_fn=0x407fe158, found_sym=0x407fe230, found_dwarf=0x407fe154, fileline_entry=0x0, exe=0, debuginfo=0, with_buildid_data=0x0, with_buildid_size=0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/elf.c:4461
#22 0x000000000145587a in phdr_callback (info=0x407fe1a0, size=<optimized out>, pdata=0x407fe240) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/elf.c:4848
#23 0x00000000500a84d9 in operator() (obj=..., __closure=<optimized out>) at /.../repos/base/src/lib/ldso/exception.cc:54
#24 Genode::Fifo<Linker::Object>::for_each<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Linker::Object&)> > (this=0x501b2500 <Linker::Object::_object_list()::_list>, func=...) at /.../repos/base/include/util/fifo.h:142
#25 Linker::Object::Object_list::for_each<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Linker::Object&)> > (func=..., this=0x501b2500 <Linker::Object::_object_list()::_list>) at /.../repos/base/src/lib/ldso/include/linker.h:204
#26 operator() (list=..., __closure=<optimized out>) at /.../repos/base/src/lib/ldso/include/linker.h:274
#27 Linker::Object::with_object_list<Linker::for_each_object<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Object&)> >(const dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Object&)>&)::<lambda(Linker::Object::Object_list&)> > (func=...) at /.../repos/base/src/lib/ldso/include/linker.h:211
#28 Linker::for_each_object<dl_iterate_phdr(int (*)(Phdr_info*, Genode::size_t, void*), void*)::<lambda(Linker::Object&)> > (func=...) at /.../repos/base/src/lib/ldso/include/linker.h:273
#29 dl_iterate_phdr (callback=callback@entry=0x14557f0 <phdr_callback>, data=data@entry=0x407fe240) at /.../repos/base/src/lib/ldso/exception.cc:42
#30 0x0000000001455a13 in backtrace_initialize (state=state@entry=0x7fff95c47000, filename=filename@entry=0x1987a63 "/proc/self/exe", descriptor=3, error_callback=error_callback@entry=0x1002750 <error_callback>, data=data@entry=0x407fe6c0, fileline_fn=fileline_fn@entry=0x407fe2d8) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/elf.c:4892
#31 0x0000000001455e4a in fileline_initialize (state=state@entry=0x7fff95c47000, error_callback=error_callback@entry=0x1002750 <error_callback>, data=data@entry=0x407fe6c0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/fileline.c:261
#32 0x0000000001455f00 in backtrace_pcinfo (state=0x7fff95c47000, pc=16788949, callback=0x1002a40 <callback>, error_callback=0x1002750 <error_callback>, data=0x407fe6c0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/fileline.c:295
#33 0x000000000144883d in unwind (context=<optimized out>, vdata=0x407fe670) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/backtrace.c:91
#34 0x0000000001467d84 in _Unwind_Backtrace (trace=trace@entry=0x1448790 <unwind>, trace_argument=trace_argument@entry=0x407fe670) at /.../contrib/gcc-093a037675b45e9b2aaddc75281f560f3b9dfe22/src/noux-pkg/gcc/libgcc/unwind.inc:309
#35 0x00000000014488e5 in backtrace_full (state=state@entry=0x7fff95c47000, skip=skip@entry=0, callback=callback@entry=0x1002a40 <callback>, error_callback=error_callback@entry=0x1002750 <error_callback>, data=data@entry=0x407fe6c0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libbacktrace/backtrace.c:127
#36 0x0000000001002dd6 in runtime_callers (skip=skip@entry=1, locbuf=locbuf@entry=0x407fe880, m=m@entry=100, keep_thunks=keep_thunks@entry=false) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/runtime/go-callers.c:255
#37 0x000000000135a151 in runtime.traceback (skip=skip@entry=0) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/traceback_gccgo.go:80
#38 0x000000000136111a in runtime.dopanic__m (gp=gp@entry=0xc000003b00, pc=pc@entry=20310267, sp=sp@entry=1082129424) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/panic.go:1207
#39 0x000000000135bb4f in runtime.fatalthrow () at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/panic.go:1073
#40 runtime.throw (s=...) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/panic.go:1044
#41 0x000000000135e8fb in runtime.semasleep (ns=98352000) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/os_genode.go:69
#42 0x0000000001371ce9 in runtime.notetsleep_internal (n=n@entry=0x1a7bed0 <runtime[sched]+240>, ns=ns@entry=98352000, deadline=110271000, gp=0xc000003b00) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/lock_sema.go:243
#43 0x0000000001371efe in runtime.notetsleep (n=n@entry=0x1a7bed0 <runtime[sched]+240>, ns=ns@entry=98352000) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/lock_sema.go:294
#44 0x000000000138e922 in runtime.sysmon () at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/proc.go:4838
#45 0x00000000013919fb in runtime.mstart1 () at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/go/runtime/proc.go:1340
#46 0x00000000010060b9 in runtime_mstart (arg=<optimized out>) at /.../contrib/libgo-a4bb84ead9a6a33857f9184b311405912b4f3680/src/lib/gcc/libgo/runtime/proc.c:617
#47 0x0000000010f5aa9f in Libc::Pthread::Thread_object::entry (this=0x7ffff5ce0078) at /.../repos/libports/src/lib/libc/pthread.cc:91
#48 0x0000000050088360 in Genode::Thread::_thread_start () at /.../repos/base-linux/src/lib/base/thread_linux.cc:94
#49 0x000000005009f71c in thread_start () at /.../repos/base-linux/src/lib/syscall/spec/x86_64/lx_clone.S:62

[chelmuth]

Commit a59f73f replaces the Symlink_resolve_error exception by a Genode::Attempt return type, which avoids the recursive dl_iterate_phdr() call and deadlock in go_app without rtc.

Thanks @cproc that's a very welcome contribution and supports our ambition to reduce the use of exceptions. Merged to staging.