Symbols missing for parts of the stack

[MrMage]

First of all, thank you for creating this! Having stack traces on Linux would be tremendously useful.

Unfortunately, only parts of the stack trace are being currently symbolicated:

          /Run(+0x497bbf) [0x556ef41d6bbf]
          /Run(+0x497439) [0x556ef41d6439]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f1b9e1c2330]
          /Run(+0x7ed4b1) [0x556ef452c4b1]
          /Run(+0x7fd62e) [0x556ef453c62e]
          /Run(+0x7fd343) [0x556ef453c343]
          /Run($S8NIOHTTP211HTTP2ParserC11channelRead3ctx4datay3NIO21ChannelHandlerContextC_AG6NIOAnyVtF+0x4e1) [0x556ef4523171]
          /Run(+0x7e7af9) [0x556ef4526af9]
          /Run(+0x71b95b) [0x556ef445a95b]
          /Run(+0x78b878) [0x556ef44ca878]
          /Run(+0x7439b7) [0x556ef44829b7]
          /Run(+0x73bb96) [0x556ef447ab96]
          /Run(+0x74782c) [0x556ef448682c]
          /Run(+0x782c03) [0x556ef44c1c03]
          /Run(+0x73b760) [0x556ef447a760]
          /Run(+0x73d555) [0x556ef447c555]
          /Run(+0x7477f9) [0x556ef44867f9]
          /Run(+0x747814) [0x556ef4486814]
          /Run(+0x79635f) [0x556ef44d535f]
          /Run(+0x7963a9) [0x556ef44d53a9]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f1b9e1ba184]
          /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1b9c72a03d]

My swift build command looks like this:

RUN swift build -c $config --product Run -Xcc -g -Xswiftc -g -Xlinker --export-dynamic

(The last option was mentioned at https://github.com/nodes-vapor/stacked#exporting-symbols-for-the-stracktracesl, and I added the cc option in case the symbols were actually in a C-based library.)

I could try using https://github.com/norio-nomura/SwiftBacktrace or https://github.com/nodes-vapor/stacked instead, but first wanted to ask for ideas of what could be the culprit.

[MrMage]

Update: I have switched to https://github.com/norio-nomura/SwiftBacktrace for now, which seems to be able to recognize these symbols. However, that library does not seem able to retrieve file names and line numbers.

[t089]

You should be able to re-symbolicate the stack trace by running the following command on the linux server next to the binary assuming the trace is saved in a file /tmp/stacktrace

cat /tmp/stacktrace | tr '()' '  ' | while read bin addr junk; do addr2line -e "$bin" -a "$addr" -ipf; done | swift demangle

See this post from @weissi.

[MrMage]

Thanks! I guess that would work; the only problem is that my stack traces occur inside Docker containers and it would be awesome to have these print the symbolicated stack trace right after the crash instead of having to do that manually later on. Not a high priority for me, though, as it looks like I fixed my last remaining crash in production by now ;-)

[t089]

Yes, totally agree. It's a bit of a pain. Definitely something the SSWG should tackle probably.

[ianpartridge]

Thanks for opening this issue, and apologies for the delay in replying - I've been on paternity leave! 👶

Are you able to supply a small testcase that shows stackframes that aren't symbolicated? I can take a look at why this might be.

Did you try the nodes-vapor package? It would be interesting to know if that works better because although it also uses backtrace(3) for unwinding, it uses dladdr() for symbolicating instead of backtrace_symbols().

I've opened a thread on the forums about the wider issues, please contribute: https://forums.swift.org/t/crash-backtraces/25021

[ianpartridge]

Hi, I've been looking into this some more - see my post at https://forums.swift.org/t/crash-backtraces/25021/19

TDLR: the backtrace_symbols() API this package currently uses only reads ELF symbols, not DWARF symbols, which is why the stacktraces aren't properly symbolicated. I've added a bash one-liner to the README.md that will allow post-mortem symbolication.

So for now, this is "working as designed" however I do plan to try and get better symbolication through https://github.com/ianlancetaylor/libbacktrace - watch this space.

[MrMage]

Thank you for the elaboration. For now I need symbolicated backtraces right at the point of logging to have these parsed by my cloud provider, so I'll need to stick with https://github.com/norio-nomura/SwiftBacktrace. I can imagine that the bash one-liner will be sufficient for many use cases, however.

[ianpartridge]

Hi @MrMage I've opened https://github.com/ianpartridge/swift-backtrace/pull/8 which switches this library to use libbacktrace. I am finding much improved backtraces.

Would you be willing to give it a try and let me know your experience? How does it compare to Norio's library (which uses libunwind)?

[MrMage]

Hi @MrMage I've opened #8 which switches this library to use libbacktrace. I am finding much improved backtraces.

Would you be willing to give it a try and let me know your experience? How does it compare to Norio's library (which uses libunwind)?

@ianpartridge thanks for the changes! Unfortunately, my codebase is still on Swift 4.2 and I need to customize the backtrace's format for compatibility with Google Cloud, both of which the other library provides. This means that I don't have a good way of testing these changes right now.

[ianpartridge]

OK no problem - thanks for replying.

Could you tell me what customisation and format you need? I can make sure to support it when redesigning the API.

P.S. This library does support Swift 4.2.

[MrMage]

The format I need is the Stackdriver logs format. Each log line is JSON-formatted; see https://cloud.google.com/logging/docs/structured-logging for the format and https://gist.github.com/MrMage/dea8a766eaa614bb0c6402379c0085de for my implementation of it.

The way I print these stack traces with the other library is the following:

let stackdriverStyleFormat = SymbolFormatter { symbol -> String in
    let (module, name, _, _) = symbol
    let sanitizedName = name
        // All these need to be replaced to make Stackdriver happy. Characters we can keep: $.<>
        .replacingOccurrences(of: "(", with: "[")
        .replacingOccurrences(of: ")", with: "]")
        .replacingOccurrences(of: " ", with: "_")
        .replacingOccurrences(of: "?", with: "0")
        .replacingOccurrences(of: "->", with: "_returns_")
        .replacingOccurrences(of: "-", with: "_")
        .replacingOccurrences(of: "@", with: "")
        .replacingOccurrences(of: ":", with: "_")
        .replacingOccurrences(of: "#", with: "")
    return "   at \(sanitizedName) (\(module):1:1)"
}

addSignalHandler {
    JSONLogger.sharedInstance.fatal("UnknownCrash\n"
        + backtrace(500, formatter: .init(stackdriverStyleFormat.compose(.default)))
            .joined(separator: "\n"))
}

The formatted stack trace then is in a format that Stackdriver can understand (I think I've used the Javascript, Python or Ruby stack trace format as a template, as that is supported by Stackdriver).

[ianpartridge]

Thanks @MrMage! I've release 1.0.3 which switches the backend to libbacktrace so I think the missing symbols should be resolved.

I will open another issue to track the redesign of the API so the Stackdriver format can be supported.

[emarashliev]

Hello there, I have the same issue, this is the output I got when execute swift run

Current stack trace:
0    libswiftCore.so                    0x00007fe8012657a0 _swift_stdlib_reportFatalErrorInFile + 115
1    libswiftCore.so                    0x00007fe8011a09cc <unavailable> + 3463628
2    libswiftCore.so                    0x00007fe8011a0abe <unavailable> + 3463870
3    libswiftCore.so                    0x00007fe800f9c31a <unavailable> + 1348378
4    libswiftCore.so                    0x00007fe801174ab2 <unavailable> + 3283634
5    libswiftCore.so                    0x00007fe800ff1d6e <unavailable> + 1699182
6    Run                                0x00005600b8130e32 <unavailable> + 5533234
7    libc.so.6                          0x00007fe7fe66cab0 __libc_start_main + 231
8    Run                                0x00005600b7c8595a <unavailable> + 637274
0x5600b7c9d6a9, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install() -> () at /app/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:53
0x5600b7c9d6c8, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install() -> () at /app/<compiler-generated>:0
0x7fe800c4688f
0x7fe801174aba
0x7fe800ff1d6d
0x5600b8130e31, main at /app/Sources/Run/main.swift:5
0x7fe7fe66cb96
0x5600b7c85959
0xffffffffffffffff
Illegal instruction

I'm running it in a Docker container with this image nodesvapor/vapor-ci:swift-5.0 there is my main.swift file

import App
import Backtrace

Backtrace.install()
try app(.detect()).run()

and my Package.swift

// swift-tools-version:5.0
import PackageDescription

let package = Package(
    name: "database-tests",
    products: [
        .library(name: "database-tests", targets: ["App"]),
    ],
    dependencies: [
        // 💧 A server-side Swift web framework.
        .package(url: "https://github.com/vapor/vapor.git", from: "3.0.0"),

        // 🔵 Swift ORM (queries, models, relations, etc) built on SQLite 3.
        .package(url: "https://github.com/vapor/fluent-mysql.git", from: "3.0.0"),

        .package(url: "https://github.com/ianpartridge/swift-backtrace.git", from: "1.1.0")
    ],
    targets: [
        .target(name: "App", dependencies: ["FluentMySQL", "Vapor"]),
        .target(name: "Run", dependencies: ["App", "Backtrace"]),
        .testTarget(name: "AppTests", dependencies: ["App"])
    ]
)

I'll appreciate any suggestions, what I'm going wrong? 🙂

[ianpartridge]

I think this indicates that the try on line 5 is throwing.

[emarashliev]

I thought should be more detailed or I'm wrong.