search

Pernosco / pernosco

General-purpose public wiki and issue tracking
13 stars 0 forks source link

Unable to view nix sources in a Pernosco trace #47

Closed georgewsinger closed 3 years ago

georgewsinger commented 3 years ago

Problem: Our Pernosco logs are unable to show sources from a nix lib we're using called libwlroots.so.0.0.0. Here's a photo from the session where we inspect calls of the symbol surface_commit:

image

Weirdly enough, Pernsoco knows (in the call stack pane) to look in wlr_surface.c:442, but the actual "sources" pane is still empty.

Source of libwlroots.so.0.0.0. The primary binary we are inspecting with Pernosco is godot.x11.tools.64, which is compiled locally in a nix-shell which provides libwlroots.so.0.0.0 as a background dependency (we made sure that nix passes -g to gcc and doesn't strip the library of debug symbols). A general issue is that Nix libs confuse rr/Pernosco because they are compiled in some sort of nix chroot tmp directory and are then copied into /nix/store, so that rr/Pernosco doesn't know where to find sources.

Our attempt to provide rr/Pernosco with the sources. Inspecting the output of

RR_LOG=all rr sources

shows that

[INFO build_symlink_map()] File /build/wlroots/build/../types/wlr_surface.c not found, skipping

So to get around this, we pass --substitute=libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend to pernosco-submit so rr/Pernosco can actually pick up the file:

rr sources --substitute=libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend rr/latest-trace | grep types/wlr_surface.c

"types/wlr_surface.c",

Despite this, Pernosco still isn't showing us the sources?

rocallahan commented 3 years ago

One issue is that the binary file in the trace is actually mmap_pack_131_libwlroots.so.0.0.0 so you need to use that with --substitute. We could fix things so to accept an "original binary file name" there or part of one, I suppose. Try resubmitting with that. After this is sorted out we'll refund your relevant credits.

There may be other issues...

Is there any way for pernosco-submit to automatically discover these Nix paths?

georgewsinger commented 3 years ago

@rocallahan

  1. The mmap_pack_*_* prefix seems to change from trace to trace, so coming up with a way to substitute across arbitrary traces would be useful.
  2. Unfortunately, we're not aware of any way for nix to to automatically discover/provide these paths. For our use case, we're primarily interested in this libwlroots*.so library though, and if we become interested in other libraries we can track them down and --substitute them (though figuring out a way to get nix to provide them all automatically would be the holy grail solution).
  3. No worries about the credits.
georgewsinger commented 3 years ago

After passing --substitute=mmap_pack_131_libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend:

image

Note that the path it provides is correct (/home/george/Simula/submodules/wlroots/types/wlr_surface.c). Maybe the source file just isn't inside the latest-trace that we are pushing?

georgewsinger commented 3 years ago

Oh -- we didn't provide ./submodules/wlroots at the end of pernosco-submit. Trying that now.

georgewsinger commented 3 years ago

Didn't work. Also tried passing $PWD/submodules/wlroots and $(find $PWD/submodules/wlroots -type d).

rocallahan commented 3 years ago

I think the problem is that pernosco-submit doesn't know about git submodules.

I guess /home/george/Simula/submodules/wlroots/types/wlr_surface.c is clean so git status doesn't list it as changed, so pernosco-submit doesn't package it into files.user/sources.zip and assumes it can be found at https://raw.githubusercontent.com/SimulaVR/Simula/0f79879f2845cea9ebd9756e0e26e2894da5c85f/submodules/wlroots/types/wlr_surface.c:, but it can't because submodules/wlroots is actually a git submodule.

georgewsinger commented 3 years ago

Interesting.

For future reference (once this stuff is sorted out), is the proper thing to append at the end of pernosco-submit to include the wlroots folder (i) $PWD/submodules/wlroots or (ii) $(find $PWD/submodules/wlroots -type d)?

rocallahan commented 3 years ago

$PWD/submodules/wlroots should work, whitelisting a directory allows everything under that directory.

rocallahan commented 3 years ago

Ok, try pulling pernosco-submit to get https://github.com/Pernosco/pernosco-submit/commit/c8c2888f61ef2be2bb169b79a14cafa7b67ed4d8

rocallahan commented 3 years ago

Your latest submission didn't pick up the wlroots submodule, I'm not sure why:

    "files": [
      {
        "url": "https://raw.githubusercontent.com/SimulaVR/Simula/5fd73d0f4ed48e39d8570b79ebbb2bfefa047173/",
        "at": "/home/george/Simula"
      },
      {
        "url": "https://raw.githubusercontent.com/SimulaVR/godot/8c97a8586bc2e200d1f758d860a5ba0c04a5ba40/",
        "at": "/home/george/Simula/submodules/godot"
      },
      {
        "url": "https://raw.githubusercontent.com/SimulaVR/gdwlroots/dd83d6c4e7ec8101eeee44bfbc807ebb34b53bf7/",
        "at": "/home/george/Simula/submodules/godot/modules/gdwlroots"
      },
      {
        "archive": "files.user/sources.zip",
        "at": "/"
      },
...
rocallahan commented 3 years ago

Does /home/george/Simula/submodules/wlroots/.git exist?

You could probably debug the pernosco-submit script to figure this out (using --dry-run). The generated sources.user should get an entry that looks like

      {
        "url": "https://raw.githubusercontent.com/SimulaVR/wlroots/<revision>/",
        "at": "/home/george/Simula/submodules/godot/modules/wlroots"
      },
georgewsinger commented 3 years ago

@rocallahan

The binary we're running is /home/george/Simula/submodules/godot/bin/godot.x11.tools.64. Wlroots is located at /home/george/Simula/submodules/wlroots/.git (and I just confirmed it exists).

We also have /home/george/Simula/submodules/godot/modules/gdwlroots/.git, which is a separate module from wlroots, and which Pernosco is already showing sources correctly.

rocallahan commented 3 years ago

If you run pernosco-submit with -x and then rerun the rr sources command and dump its output somewhere I can see it, that would be helpful.

georgewsinger commented 3 years ago

@rocallahan

I re-ran pernosco-submit with -x (it's still uploading now).

Here is the output of RR_LOG=all rr sources > wlroots.txt 2>&1.

EDIT: This log file looks really messed up/minimal. I'm going to re-run the trace and re-upload in case state somehow got changed.

EDIT2: Here is the updated log file. I am re-uploading a fresh trace with -x passed (should be finished uploading in a few minutes).

georgewsinger commented 3 years ago

To clarify: should I be running

--substitute=mmap_pack_132_libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend

or 

--substitute=libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend

?

rocallahan commented 3 years ago 
--substitute=mmap_pack_132_libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend
georgewsinger commented 3 years ago

Here are updated logs with that substitution in place.

rocallahan commented 3 years ago

Actually I'm wrong, it needs to be --substitute=/nix/store/8mr6imw5ddch6qqsqsja73vny0h4gwnf-wlroots/lib/libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend

rocallahan commented 3 years ago

that's what you get from

rr filename mmap_pack_132_libwlroots.so.0.0.0
rocallahan commented 3 years ago

Maybe this feature is just broken and rr sources expects the original file name and pernosco-submit expects the trace file name.

rocallahan commented 3 years ago

(It was contributed and hardly anyone has used it)

rocallahan commented 3 years ago

So in your latest submissions, mmap_pack_132_libwlroots.so.0.0.0 has basically no debuginfo.

rocallahan commented 3 years ago

Maybe this feature is just broken and rr sources expects the original file name and pernosco-submit expects the trace file name.

This seems to be the case. I'm working on it.

rocallahan commented 3 years ago

Alright, you'll need to update to rr master to get https://github.com/rr-debugger/rr/commit/856f5d2517a174ca1f08a8c42e1c6355ce6380e8, and pernosco-submit to get https://github.com/Pernosco/pernosco-submit/commit/d02ccf99dddc6e496f310e3b9f0bdad25a36fcbb, and then submit with --substitute=libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend ... then I hope things will work! :-)

georgewsinger commented 3 years ago

@rocallahan Does not appear to work. Here are logs. I'm uploading with -x now also.

rocallahan commented 3 years ago

That's because of what I mentioned here:

So in your latest submissions, mmap_pack_132_libwlroots.so.0.0.0 has basically no debuginfo.

In https://pernos.co/debug/3SM3WQREox8lztRuCbPNtQ/index.html#f{m[AVVh,B++5_,t[AQ,GHVA_,f{e[AVVh,B++5___ mmap_pack_132_libwlroots.so.0.0.0 has just this:

Contents of the .debug_info section:

  Compilation Unit @ offset 0x0:
   Length:        0x1e (32-bit)
   Version:       2
   Abbrev Offset: 0x0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_stmt_list   : 0x0
    <10>   DW_AT_ranges      : 0x0
    <14>   DW_AT_name        : (indirect string, offset: 0x0): ../sysdeps/x86_64/crti.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x19): /build/glibc-2.32/csu
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2f): GNU AS 2.35.1
    <20>   DW_AT_language    : 32769    (MIPS assembler)
  Compilation Unit @ offset 0x22:
   Length:        0x1e (32-bit)
   Version:       2
   Abbrev Offset: 0x12
   Pointer Size:  8
 <0><2d>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <2e>   DW_AT_stmt_list   : 0x65
    <32>   DW_AT_ranges      : 0x40
    <36>   DW_AT_name        : (indirect string, offset: 0x3d): ../sysdeps/x86_64/crtn.S
    <3a>   DW_AT_comp_dir    : (indirect string, offset: 0x19): /build/glibc-2.32/csu
    <3e>   DW_AT_producer    : (indirect string, offset: 0x2f): GNU AS 2.35.1
    <42>   DW_AT_language    : 32769    (MIPS assembler)

and there is no .gnu_debuglink. It looks like it was built without -g.

georgewsinger commented 3 years ago

@rocallahan

I had mistakenly believed that libwlroots was compiled with -g. Just fixed and now it works!

image

Thanks for helping me through this. Pernosco continues to escalate in usefulness for our project.

rocallahan commented 3 years ago

I refunded you 14 subscription credits. Probably doesn't matter in practice but seemed like the right thing to do :-).

georgewsinger commented 3 years ago

Facing this issue again in https://pernos.co/debug/XvtsNY4WxynP8l3JBCcPvg/index.html

rocallahan commented 3 years ago

Have you got the full pernosco-submit command-line and transcript you used?

khuey commented 3 years ago

The libwlroots.so in this trace appears to have been built without debug info.

georgewsinger commented 3 years ago

I'm probably doing something dumb, but am still having issue with this.

  1. Pernosco trace. I uploaded a fresh Pernosco trace here (<- EDIT: now with -x passed to pernosco-submit). We pass --substitute=libwlroots.so.0.0.0=$(pwd)/submodules/wlroots/backend as discussed above.

  2. rr sources. The result of

    RR_LOG=all ./result/bin/rr sources ./rr/latest-trace > sources.txt

    is here, which picks up mmap_hardlink_92_libwlroots.so.0.0.0.

  3. Debug info. When compiling libwlroots.so (via nix), we pass -0g to gcc and dontStrip = true to nix.

    But just to make sure: if we inspect our root binary, it is reading from /nix/store/0lj3f0z6iwgw0pc9wfqajwdg01da1csm-wlroots/lib/libwlroots.so.0:

    ldd ~/Simula/submodules/godot/bin/godot.x11.tools.64 | grep wlroots
    libwlroots.so.0 => /nix/store/0lj3f0z6iwgw0pc9wfqajwdg01da1csm-wlroots/lib/libwlroots.so.0 (0x00007fc62a754000)

    and if we inspect that for debug info:

    objdump --syms  /nix/store/0lj3f0z6iwgw0pc9wfqajwdg01da1csm-wlroots/lib/libwlroots.so.0 | grep debug
    0000000000000000 l    d  .debug_aranges  0000000000000000              .debug_aranges
0000000000000000 l    d  .debug_info 0000000000000000              .debug_info
0000000000000000 l    d  .debug_abbrev   0000000000000000              .debug_abbrev
0000000000000000 l    d  .debug_line 0000000000000000              .debug_line
0000000000000000 l    d  .debug_str  0000000000000000              .debug_str
0000000000000000 l    d  .debug_ranges   0000000000000000              .debug_ranges
0000000000072220 l     O .rodata 0000000000000048              debug_attribs.6

    Then it seems to indicate debug info should be working?

rocallahan commented 3 years ago

Those symbols aren't really relevant. You want something like readelf -a. Kyle's results for libwlroots.so submitted in XvtsNY4WxynP8l3JBCcPvg were

  [25] .debug_aranges    PROGBITS         0000000000000000  0009cf20
       0000000000000080  0000000000000000           0     0     16
  [26] .debug_info       PROGBITS         0000000000000000  0009cfa0
       0000000000000044  0000000000000000           0     0     1
  [27] .debug_abbrev     PROGBITS         0000000000000000  0009cfe4
       0000000000000024  0000000000000000           0     0     1
  [28] .debug_line       PROGBITS         0000000000000000  0009d008
       00000000000000c6  0000000000000000           0     0     1
  [29] .debug_str        PROGBITS         0000000000000000  0009d0ce
       0000000000000056  0000000000000001  MS       0     0     1
  [30] .debug_ranges     PROGBITS         0000000000000000  0009d130
       0000000000000080  0000000000000000           0     0     16

There's data there, but it's tiny. The output there means the size of the .debug_info section is 0x44 bytes. It's just debuginfo for ../sysdeps/x86_64/crti.S and ../sysdeps/x86_64/crtn.S.

georgewsinger commented 3 years ago

The result of readelf -a is here. I'm assuming it shows lack of debug info.

georgewsinger commented 3 years ago

:heavy_check_mark: Traces now show sources properly when we use LD_PRELOAD to force load a version of libwlroots with debug symbols.

This was an issue on our end.

rocallahan commented 3 years ago

No problem. We need better diagnostics for this, though it's not clear what the "minimum amount" of debuginfo should be that would trigger a pernosco-submit warning...

georgewsinger commented 3 years ago

On this trace again we

  1. Seem to be showing all libwlroots.so sources, which is good.

  2. Seem to be showing only some sources from our primary godot.x11.tools.64 binary. For example we can see sources for WlrXWayland::WlrXWayland() function:

    image

    However, we don't see any sources for WaylandDisplay::WaylandDisplay(), which is located in a source in the same folder:

    image

Do you know what could be causing this?

rocallahan commented 3 years ago

Can you run pernosco-submit with --dry-run and -x and then rerun the rr sources command and dump the output of that?

sources.zip contains

Archive:  files.user/sources.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
    4609  Defl:N     2376  48% 2021-05-03 12:35 55256646  /home/george/Simula/submodules/godot/core/authors.gen.h
    8976  Defl:N     4198  53% 2021-05-03 12:35 8edd4919  /home/george/Simula/submodules/godot/core/donors.gen.h
   75743  Defl:N    21390  72% 2021-05-03 12:35 30527a54  /home/george/Simula/submodules/godot/core/license.gen.h
  137574  Defl:N     4218  97% 2021-05-03 12:35 744881e7  /home/george/Simula/submodules/godot/core/method_bind.gen.inc
     319  Defl:N      161  50% 2021-05-03 12:35 888c8af9  /home/george/Simula/submodules/godot/core/version_generated.gen.h
      63  Defl:N       65  -3% 2021-05-03 12:35 7a67a6ad  /home/george/Simula/submodules/godot/core/version_hash.gen.h
   17121  Defl:N     2457  86% 2021-05-03 12:35 14696e99  /home/george/Simula/submodules/godot/drivers/gles3/shaders/blend_shape.glsl.gen.h
   66855  Defl:N     7895  88% 2021-05-03 12:35 3ec8c72e  /home/george/Simula/submodules/godot/drivers/gles3/shaders/canvas.glsl.gen.h
    8456  Defl:N     1815  79% 2021-05-03 12:35 a107bb88  /home/george/Simula/submodules/godot/drivers/gles3/shaders/canvas_shadow.glsl.gen.h
   28493  Defl:N     4331  85% 2021-05-03 12:35 ecb0f1a2  /home/george/Simula/submodules/godot/drivers/gles3/shaders/copy.glsl.gen.h
   11451  Defl:N     2135  81% 2021-05-03 12:35 0bc85bc6  /home/george/Simula/submodules/godot/drivers/gles3/shaders/cube_to_dp.glsl.gen.h
   38397  Defl:N     5343  86% 2021-05-03 12:35 d22ae4be  /home/george/Simula/submodules/godot/drivers/gles3/shaders/cubemap_filter.glsl.gen.h
   37644  Defl:N     4614  88% 2021-05-03 12:35 1635bb3b  /home/george/Simula/submodules/godot/drivers/gles3/shaders/effect_blur.glsl.gen.h
   13833  Defl:N     2338  83% 2021-05-03 12:35 d126c4fb  /home/george/Simula/submodules/godot/drivers/gles3/shaders/exposure.glsl.gen.h
    9526  Defl:N     2009  79% 2021-05-03 12:35 915349b4  /home/george/Simula/submodules/godot/drivers/gles3/shaders/lens_distorted.glsl.gen.h
   23959  Defl:N     3833  84% 2021-05-03 12:35 4a5bc1b3  /home/george/Simula/submodules/godot/drivers/gles3/shaders/particles.glsl.gen.h
    7709  Defl:N     1734  78% 2021-05-03 12:35 f9e626e9  /home/george/Simula/submodules/godot/drivers/gles3/shaders/resolve.glsl.gen.h
  234927  Defl:N    25947  89% 2021-05-03 12:35 6201638e  /home/george/Simula/submodules/godot/drivers/gles3/shaders/scene.glsl.gen.h
   34724  Defl:N     5609  84% 2021-05-03 12:35 978335eb  /home/george/Simula/submodules/godot/drivers/gles3/shaders/screen_space_reflection.glsl.gen.h
   36124  Defl:N     6074  83% 2021-05-03 12:35 f5de05b5  /home/george/Simula/submodules/godot/drivers/gles3/shaders/ssao.glsl.gen.h
   17549  Defl:N     3453  80% 2021-05-03 12:35 fdfb9418  /home/george/Simula/submodules/godot/drivers/gles3/shaders/ssao_blur.glsl.gen.h
   10128  Defl:N     2179  79% 2021-05-03 12:35 20518d81  /home/george/Simula/submodules/godot/drivers/gles3/shaders/ssao_minify.glsl.gen.h
   21875  Defl:N     3606  84% 2021-05-03 12:35 2c5b67d5  /home/george/Simula/submodules/godot/drivers/gles3/shaders/subsurf_scattering.glsl.gen.h
   37869  Defl:N     5341  86% 2021-05-03 12:35 65015eb0  /home/george/Simula/submodules/godot/drivers/gles3/shaders/tonemap.glsl.gen.h
    5763  Defl:N      899  84% 2021-05-03 12:35 392f60f3  /home/george/Simula/submodules/godot/editor/doc/doc_data_class_path.gen.h
   17224  Defl:N     5776  67% 2021-05-03 12:35 bac34254  /home/george/Simula/submodules/godot/main/app_icon.gen.h
   65021  Defl:N    19265  70% 2021-05-03 12:35 67e49ee5  /home/george/Simula/submodules/godot/main/splash.gen.h
  125785  Defl:N    38154  70% 2021-05-03 12:35 841bd139  /home/george/Simula/submodules/godot/main/splash_editor.gen.h
   21663  Defl:N     3686  83% 2021-05-03 12:35 972659e6  /home/george/Simula/submodules/godot/modules/gdwlroots/wlr_xwayland.cpp
   54748  Defl:N    12065  78% 2021-05-03 12:35 06b1af78  /home/george/Simula/submodules/godot/modules/gdwlroots/xdg-shell-protocol.h

so wayland_display.cpp was not collected for some reason, but it's not clear why.

georgewsinger commented 3 years ago

The issue was again internal: we renamed our root folder, and then submitted a trace before cleaning our compile cache (which had references to our old absolute folder names).

Sorry for bringing two false alarms to the table. :cold_sweat:

rocallahan commented 3 years ago

No worries. We definitely need better diagnostics here so users can self-diagnose more easily.

georgewsinger commented 3 years ago

Today we attempted to bring several more nix sources into rr/Pernosco traces. The libs in question are:

  1. libwayland-client.so.0.3.0
  2. libwayland-egl.so.1.0.0
  3. libwayland-server.so.0.1.0
  4. libxcb.so.1.1.0
  5. Xwayland (a binary launched during runtime)

We hard-copied the source folders to ./srcs_local/* (just in case rr/Pernosco gets confused by symlinks to sources in /nix/store/* locations), recorded a fresh trace, and then pushed to Pernosco via:

pernosco-submit -x upload --title $2 \
     --substitute=libwlroots.so.0.0.0=$PWD/submodules/wlroots/backend \
     --substitute=libwayland-client.so.0.3.0=$PWD/srcs_local/wayland \
     --substitute=libwayland-egl.so.1.0.0=$PWD/srcs_local/wayland \
     --substitute=libwayland-server.so.0.1.0=$PWD/srcs_local/wayland \
     --substitute=libxcb.so.1.1.0=$PWD/srcs_local/libxcb \
     --substitute=Xwayland=$PWD/srcs_local/xwayland \
     $1 ./. \
     $PWD/submodules/wlroots \
     $PWD/srcs_local/wayland \
     $PWD/srcs_local/wayland-protocols \
     $PWD/srcs_local/xwayland \
     $PWD/srcs_local/libxcb

Result: sources not showing. Using xcb_connect as a reference symbol (from libxcb), it seems that we have source lines but not actual sources being visible in Pernosco:

image

We're propagating -g -ggdb -Og for these libs to gcc via nix.

Any idea on how to get sources visible?

rocallahan commented 3 years ago

One thing that could be a problem is that you have two different libxcb.so.1.1.0s in XCpbjb4KQX7ke3eKQJCWpw:

$ ls -l mmap_pack_32_libxcb.so.1.1.0
-rwx------ 1 999 999 223808 May  6 01:00 mmap_pack_32_libxcb.so.1.1.0
$ ls -l mmap_pack_264_libxcb.so.1.1.0
-rwx------ 1 999 999 166488 May  6 01:00 mmap_pack_264_libxcb.so.1.1.0

The former one doesn't have a build ID.

Also is there a mixup with these logs? because the rr sources log says Examining /home/george/Simula/rr/godot.x11.tools.64-2/mmap_pack_266_libxcb.so.1.1.0, but there is no such file in XCpbjb4KQX7ke3eKQJCWpw.

georgewsinger commented 3 years ago

Here's an updated rr sources in case the other one was corrupted: https://www.wolframcloud.com/obj/69f6c0c7-e407-4c26-9153-41143c20ff96

rocallahan commented 3 years ago

That's still weird because that log says

[SourcesCommand] Looking for comp_dir substitutions for libxcb.so.1.1.0
[SourcesCommand]    None found
[ElfReader] Missing section .debug_info
[ElfReader] Missing section .debug_abbrev
[ElfReader] Missing section .debug_str
[ElfReader] Missing section .debug_str_offsets
[ElfReader] Missing section .debug_line
[ElfReader] Missing section .debug_line_str
[WARN find_auxiliary_file() errno: ENOENT] Can't find external debuginfo file /usr/lib/x86_64-linux-gnu/2ac5e19fd601fa6da4fa2e5630bcf8e3148d05.debug
[WARN find_auxiliary_file() errno: ENOENT] Can't find external debuginfo file /usr/lib/x86_64-linux-gnu/.debug/2ac5e19fd601fa6da4fa2e5630bcf8e3148d05.debug
[WARN find_auxiliary_file() errno: ENOENT] Can't find external debuginfo file /usr/lib/debug/2ac5e19fd601fa6da4fa2e5630bcf8e3148d05.debug
[WARN find_auxiliary_file() errno: ENOENT] Can't find external debuginfo file /usr/lib/debug/usr/lib/x86_64-linux-gnu/2ac5e19fd601fa6da4fa2e5630bcf8e3148d05.debug
[WARN find_auxiliary_file() errno: ENOENT] Exhausted auxilliary debuginfo search locations for 2ac5e19fd601fa6da4fa2e5630bcf8e3148d05.debug
[INFO sources()] No debuginfo found

but I get

$ sudo readelf -n mmap_pack_264_libxcb.so.1.1.0 

Displaying notes found in: .note.gnu.property
  Owner                 Data size   Description
  GNU                  0x00000010   NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature: IBT, SHSTK

Displaying notes found in: .note.gnu.build-id
  Owner                 Data size   Description
  GNU                  0x00000014   NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: b12ac5e19fd601fa6da4fa2e5630bcf8e3148d05

i.e. the build IDs are different

rocallahan commented 3 years ago

Anyway the fact that rr sources does not output xcb_util.cc means the problem is caused by something before or during rr sources.

georgewsinger commented 3 years ago

We just tried propagating LD_PRELOAD=${libxcb-dev}/lib/libxcb.so.1 to avoid the double lib loading issue. Also maybe now we can get our logs straight :)

rocallahan commented 3 years ago 
[INFO sources()] Examining /home/george/Simula/rr/godot.x11.tools.64-4/mmap_pack_32_libxcb.so.1.1.0
[ElfReader] Missing section .gnu_debugaltlink
[SourcesCommand] Looking for comp_dir substitutions for libxcb.so.1.1.0
[SourcesCommand]    None found
[ElfReader] Missing section .debug_info
[ElfReader] Missing section .debug_abbrev
[ElfReader] Missing section .debug_str
[ElfReader] Missing section .debug_str_offsets
[ElfReader] Missing section .debug_line
[ElfReader] Missing section .debug_line_str
[ElfReader] Missing section .gnu_debuglink
[INFO sources()] No debuginfo found

so it looks like you eliminated the libxcb.so.1.1.0 that had debuginfo...

georgewsinger commented 3 years ago

😬 Ok. I'll have a look at this tomorrow morning to see if I can eliminate the proper libxcb

rocallahan commented 3 years ago

Because rr sources isn't listing xbc_util.cc, this is something you should be able to figure out locally.

georgewsinger commented 3 years ago

We have fixed the remainder of these issues internally, and now have sources working for a bunch more nix libs. :+1: Thanks for your help @rocallahan