Can't read debug info for dlopened libraries

[honggyukim]

uftrace currently doesn't store .dbg info for dlopened libraries. The example is as follows.

$ gcc -o libabc_test_lib.so -shared -fPIC -pg -g tests/s-lib.c
$ gcc -o t-dlopen -pg -g tests/s-dlopen.c -ldl

$ uftrace record -a ./t-dlopen

$ uftrace replay -f +module
# DURATION     TID        MODULE NAME   FUNCTION
            [ 13633]         t-dlopen | main(1, 0x7ffe751b2878) {
 336.727 us [ 13633]         t-dlopen |   dlopen("./libabc_test_lib.so", RTLD_LAZY) = 0xff08f0;
   1.617 us [ 13633]         t-dlopen |   dlsym(0xff08f0, "lib_a") = &lib_a;
            [ 13633]        [unknown] |   lib_a() {
            [ 13633]        [unknown] |     lib_b() {
   0.632 us [ 13633]        [unknown] |       lib_c();
   1.163 us [ 13633]        [unknown] |     } /* lib_b */
   1.503 us [ 13633]        [unknown] |   } /* lib_a */
  12.232 us [ 13633]         t-dlopen |   dlclose(0xff08f0) = 0;
  99.946 us [ 13633]         t-dlopen |   dlopen("./libfoo.so", RTLD_LAZY) = 0;
 458.720 us [ 13633]         t-dlopen | } = -1; /* main */

It shows the module name as [unknown] and doesn't set the debug info of dlopen-ed library libabc_test_lib.so.

$ cat uftrace.data/libabc_test_lib.so.sym
# symbols: 5
# path name: ./libabc_test_lib.so
# build-id: 61a70fe3583bd66e771224257d8eec50167d4a52
0000000000000570 P getpid
0000000000000580 ? __dynsym_end
0000000000000645 T lib_a
0000000000000668 t lib_b
000000000000068b t lib_c
00000000000006a9 ? __func_end
0000000000201028 d completed.8023
0000000000201029 ? __sym_end

$ ls uftrace.data/libabc_test_lib.so.dbg
ls: cannot access 'uftrace.data/libabc_test_lib.so.dbg': No such file or directory

[honggyukim]

The .sym file of dlopened library is currently created at https://github.com/namhyung/uftrace/blob/v0.10/libmcount/dynamic.c#L644.

The .dbg file can also be generated at this point. For example,

diff --git a/libmcount/dynamic.c b/libmcount/dynamic.c
index 7d95a61e..43ed6be2 100644
--- a/libmcount/dynamic.c
+++ b/libmcount/dynamic.c
@@ -657,6 +657,8 @@ void mcount_dynamic_dlopen(struct symtabs *symtabs, struct dl_phdr_info *info,
        mdi->map = map;

        map->mod = load_module_symtab(symtabs, map->libname, map->build_id);
+       prepare_debug_info(symtabs, PATT_REGEX, NULL, NULL, false, true);
+       save_debug_info(symtabs, symtabs->dirname);
        mcount_arch_find_module(mdi, &map->mod->symtab);

        if (mcount_setup_trampoline(mdi) < 0) {

But filters including arguments can't be applied at this point. We need to find anther way to apply them.

[namhyung]

I think it's a duplicate of #842.

[honggyukim]

Ah, you’re right. But this also shows the module name is missing.

[honggyukim]

https://github.com/namhyung/uftrace/issues/842#issuecomment-757488907

It turns out to be a non trivial change since it needs to modify the filter/trigger in libmcount at runtime. Currently it initializes them at load time and accesses them in a lock-less way assuming it's not changing. But with dlopen, we need to add or delete some.

Can we inform the program with a special option that the target library will be used so apply filters even before the dlopen call?

[honggyukim]

We might be able to provide --dlopen option for this. For example,

$ uftrace record -a --dlopen libabc_test_lib.so ./t-dlopen

[honggyukim]

Otherwise, we can also load the dwarf info if we have a library name as a postfix after -A or -R as follows.

$ uftrace -A.@libabc_test_lib.so -R.@libabc_test_lib.so t-dlopen

After parsing the command, we can generate and load dwarf info for the library.

[honggyukim]

When a dlopened library is loaded, we may have to apply filters after acquiring a lock as @gpollo did at https://github.com/namhyung/uftrace/blob/71154aed62c365692ec1f11c4682b64d9691d7b5/libmcount/mcount.c#L902-L908.

[namhyung]

Could be. But I'd like to separate the filters/triggers tree for dlopen from the main filters/triggers if possible.