"bad fde" warning when compiled for x86_64-unknown-linux-musl target

[ueno]

When wasmtime is compiled with cargo build --target x86_64-unknown-linux-musl, it produces a runtime warning as below:

$ ./target/x86_64-unknown-linux-musl/debug/wasmtime tests/wasm/hello_wasi_snapshot1.wat           
libunwind: __unw_add_dynamic_fde: bad fde: FDE is really a CIE
libunwind: __unw_add_dynamic_fde: bad fde: FDE is really a CIE
Hello, world!

• What are the steps to reproduce the issue? See above.

• What do you expect to happen? What does actually happen? Does it panic, and if so, with which assertion? I am actually not sure about the impact, but it looks like the FDE record is corrupted in the generated code.

• Which Wasmtime version / commit hash / branch are you using? git master (f4e3e4efde28ac90be9fd68dcbda2943b3851155)

• If relevant, can you include some extra information about your environment? (Rust version, operating system, architecture...)

  • rustc 1.43.0 (4fb7144ed 2020-04-20)
  • Fedora 32 (x86_64)

[peterhuene]

I'm able to reproduce this and I'll investigate.

[peterhuene]

There are two warnings because two frame tables are being registered, one for the the wasm functions and one for the trampolines.

Here's one of the decoded frame tables being passed to __unw_add_dynamic_fde:

[
    20,
    0,
    0,
    0,      // CIE length: 20 bytes

    0,
    0,
    0,
    0,      // CIE ID: 0

    1,      // Version: 1

    0,      // Agumentation (none)

    1,      // Code alignment factor

    120,    // Data alignment factor (-8)

    16,     // Return register: RAX

    12,     // Start of initial instructions
    7,
    8,
    144,
    1,
    0,
    0,
    0,
    0,
    0,
    0,      // End of initial instructions

    36,
    0,
    0,
    0,      // FDE length: 36 bytes

    28,
    0,
    0,
    0,      // CIE offset: 4 (FDE length) + 24 (CIE record + length) = 28

    0,
    0,
    221,
    247,
    255,
    127,
    0,
    0,      // Function address: 0x00007ffff7dd0000

    28,
    0,
    0,
    0,      
    0,
    0,
    0,
    0,      // Function length: 28 bytes

    66,     // Start of call frame instructions
    14,
    16,
    134,
    2,
    67,
    13,
    6,
    86,
    12,
    7,
    8,
    0,
    0,
    0,
    0,      // End of call frame instructions

    36,
    0,
    0,
    0,      // FDE length: 36 bytes 

    68,
    0,
    0,
    0,      // CIE offset: 4 (this FDE length) + 40 (first FDE record + length) + 24 (CIE record + length) = 68

    28,
    0,
    221,
    247,
    255,
    127,
    0,
    0,      // Function address: 0x00007FFFF7DD001C

    11,
    0,
    0,
    0,
    0,
    0,
    0,
    0,      // Function length: 11 bytes

    66,     // Start of call frame instructions
    14,
    16,
    134,
    2,
    67,
    13,
    6,
    69,
    12,
    7,
    8,
    0,
    0,
    0,
    0,      // End of call frame instructions

    0,
    0,
    0,
    0,      // FDE with length 0 (end of records marker)
]

This table appears correct.

There appears to be a libunwind issue unrelated to wasmtime that hasn't been investigated, although I'm not sure if that one is musl-related or not.

I'll see if I can debug into libunwind and see what's tripping it up.

[peterhuene]

This bug is due to Wasmtime expecting __register_frame (aliased to __unw_add_dynamic_fde in libunwind) to take a whole table like it does with libgcc, but musl, like the macOS implementation, expects a single FDE.

We need to update the UnwindRegistry to iterate the FDEs like we do for macOS.

[peterhuene]

So #1914 fixes these warnings and Wasmtime now correctly registers the JIT unwind information for musl builds of Wasmtime.

Unfortunately, we discovered that musl-targeted LLVM's libunwind is unable to walk stacks properly from signal handlers (see discussion in #1914), which is what Wasmtime needs to capture trap backtraces. As a result, musl-targeted Wasmtime won't show any backtraces for traps.

We have a related issue to this where foreign frames that don't have unwind information (#1845) also trip this up. We may stop using the system-provided unwinder for walking the stack in the future and this will likely fix this issue for musl builds as well.