search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.31k stars 1.59k forks source link

File paths in DWARF are not absolute #44325

Closed jan-auer closed 1 year ago

jan-auer commented 4 years ago

I'm reporting this here since I've seen symbolication being discussed in this issue tracker.

DWARF debugging information generated for Android builds contains logical file paths, either prefixed by dart: or package:, followed by the package and finally the source path. For example, this is extracted using dwarfdump from an example app:

0x00000650:   DW_TAG_subprogram
                DW_AT_name      ("_IntListMixin.contains")
                DW_AT_decl_file ("dart:typed_data-patch/typed_data_patch.dart")

0x0000066b:   DW_TAG_subprogram
                DW_AT_name      ("_RawMaterialButtonState.didUpdateWidget")
                DW_AT_decl_file ("package:flutter/src/material/button.dart")

This way of representing files makes sense to a developer, as it clearly states where the file belongs to and how it can be located. However, it is hard for a general-purpose debugger to locate these files, as it usually has no concept of "packages".

Usually, paths like the one referred to by DW_AT_decl_file are relative to a location called the "compilation directory". This is the location in which the compiler was invoked, and is often also considered the "project root". It is defined on the compilation unit, which in our case looks like this:

0x0000000b: DW_TAG_compile_unit
              DW_AT_name        ("kad")
              DW_AT_producer    ("Dart VM")
              DW_AT_comp_dir    ("")
              DW_AT_low_pc      (0x0000000000010000)
              DW_AT_high_pc     (0x0000000000259188)
              DW_AT_stmt_list   (0x00000000)

It would be great to follow DWARF conventions and allow debuggers to locate source files by emitting absolute file paths. This would allow to step through code while debugging, and crash reporting tools like Sentry could attach source code to their crash reports. A possible approach to this would be:

It is still possible to retain custom paths by adding an extension to DWARF line programs after DW_LNCT_MD5. Please let me know if I should elaborate on this point.

mkustermann commented 3 years ago

/cc @sstrickl

sstrickl commented 3 years ago

I've added a --resolve-dwarf-paths flag in CL 196491, submitted as 6730b12edc, that does most of this. I'm not sure how feasible it is to set DW_AT_comp_dir appropriately, because I'm not sure if enough information is provided via the embedder interface to do so.

marandaneto commented 2 years ago

@sstrickl thanks, where do I pass the --resolve-dwarf-paths parameter? flutter build apk --split-debug-info=symbols --obfuscate --resolve-dwarf-paths does not work. Thanks for doing this btw (Flutter 3.0.1).

vaind commented 2 years ago

@sstrickl I've tried adding the --resolve-dwarf-paths to the flutter tool and am getting the following errors while compiling a flutter app (for android first):

../../third_party/dart/runtime/vm/dwarf.cc: 898: error: cannot convert resolved URI org-dartlang-sdk:///third_party/dart/sdk/timeStamp/_internal/vm_shared/timeStamp/bigint_patch.dart
version=2.19.0-374.0.dev (dev) (Fri Nov 4 12:40:18 2022 -0700) on "macos_arm"
pid=42698, thread=-1, isolate_group=isolate(0x7fa964809a00), isolate=isolate(0x7fa96581ec00)
os=macos, arch=arm, comp=no, sim=no
isolate_instructions=0, vm_instructions=0
  pc 0x00000001027bea25 fp 0x00000003098472c0 dart::Profiler::DumpStackTrace(void*)+0x85
  pc 0x00000001026423d4 fp 0x00000003098473a0 dart::Assert::Fail(char const*, ...) const+0x84
  pc 0x00000001026ebb8a fp 0x0000000309847400 std::_LIBCPP_ABI_NAMESPACE::__function::__func<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1::operator()() const::'lambda'(), std::_LIBCPP_ABI_NAMESPACE::allocator<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1::operator()() const::'lambda'()>, void ()>::operator()()+0x37a
  pc 0x00000001026f42a8 fp 0x0000000309847440 dart::DwarfElfStream::WritePrefixedLength(char const*, std::_LIBCPP_ABI_NAMESPACE::function<void ()>)+0x58
  pc 0x00000001026eb439 fp 0x00000003098474f0 std::_LIBCPP_ABI_NAMESPACE::__function::__func<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1, std::_LIBCPP_ABI_NAMESPACE::allocator<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1>, void ()>::operator()()+0x69
  pc 0x00000001026f42a8 fp 0x0000000309847530 dart::DwarfElfStream::WritePrefixedLength(char const*, std::_LIBCPP_ABI_NAMESPACE::function<void ()>)+0x58
  pc 0x00000001026e9ce0 fp 0x0000000309847590 dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)+0x50
  pc 0x00000001026edc33 fp 0x0000000309847650 dart::Elf::FinalizeDwarfSections()+0x453
  pc 0x00000001026eed7f fp 0x00000003098476e0 dart::Elf::Finalize()+0x3f
  pc 0x0000000102ad529a fp 0x0000000309847a60 dart::CreateAppAOTSnapshot(void (*)(void*, unsigned char const*, long), void*, bool, bool, void*, dart::GrowableArray<dart::LoadingUnitSerializationData*>*, dart::LoadingUnitSerializationData*, unsigned int)+0x4ba
  pc 0x0000000102ad5eec fp 0x0000000309847b40 Dart_CreateAppAOTSnapshotAsElf+0x13c
  pc 0x000000010263a6ce fp 0x0000000309847cd0 dart::bin::main(int, char**)+0x10ee
  pc 0x000000020315e52e fp 0x0000000309847de0 Unknown symbol
-- End of DumpStackTrace
Dart snapshot generator failed with exit code -6
../../third_party/dart/runtime/vm/dwarf.cc: 898: error: cannot convert resolved URI org-dartlang-sdk:///third_party/dart/sdk/timeStamp/internal/cast.dart
version=2.19.0-374.0.dev (dev) (Fri Nov 4 12:40:18 2022 -0700) on "macos_simarm64"
pid=42699, thread=-1, isolate_group=isolate(0x7f8c67808200), isolate=isolate(0x7f8c6780d800)
os=macos, arch=arm64, comp=yes, sim=yes
isolate_instructions=0, vm_instructions=0
  pc 0x000000010477b8f5 fp 0x000000030d5d62e0 dart::Profiler::DumpStackTrace(void*)+0x85
  pc 0x00000001045f03d4 fp 0x000000030d5d63c0 dart::Assert::Fail(char const*, ...) const+0x84
  pc 0x000000010469e37a fp 0x000000030d5d6420 std::_LIBCPP_ABI_NAMESPACE::__function::__func<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1::operator()() const::'lambda'(), std::_LIBCPP_ABI_NAMESPACE::allocator<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1::operator()() const::'lambda'()>, void ()>::operator()()+0x39a
  pc 0x00000001046a6a78 fp 0x000000030d5d6460 dart::DwarfElfStream::WritePrefixedLength(char const*, std::_LIBCPP_ABI_NAMESPACE::function<void ()>)+0x58
  pc 0x000000010469dbdd fp 0x000000030d5d6520 std::_LIBCPP_ABI_NAMESPACE::__function::__func<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1, std::_LIBCPP_ABI_NAMESPACE::allocator<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1>, void ()>::operator()()+0x6d
  pc 0x00000001046a6a78 fp 0x000000030d5d6560 dart::DwarfElfStream::WritePrefixedLength(char const*, std::_LIBCPP_ABI_NAMESPACE::function<void ()>)+0x58
  pc 0x000000010469c430 fp 0x000000030d5d65c0 dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)+0x50
  pc 0x00000001046a0423 fp 0x000000030d5d6680 dart::Elf::FinalizeDwarfSections()+0x453
  pc 0x00000001046a156f fp 0x000000030d5d6710 dart::Elf::Finalize()+0x3f
  pc 0x0000000104aa5baa fp 0x000000030d5d6a90 dart::CreateAppAOTSnapshot(void (*)(void*, unsigned char const*, long), void*, bool, bool, void*, dart::GrowableArray<dart::LoadingUnitSerializationData*>*, dart::LoadingUnitSerializationData*, unsigned int)+0x4ba
  pc 0x0000000104aa67ac fp 0x000000030d5d6b70 Dart_CreateAppAOTSnapshotAsElf+0x13c
  pc 0x00000001045e86ce fp 0x000000030d5d6d00 dart::bin::main(int, char**)+0x10ee
  pc 0x0000000204f3f52e fp 0x000000030d5d6e10 Unknown symbol
-- End of DumpStackTrace
Dart snapshot generator failed with exit code -6
../../third_party/dart/runtime/vm/dwarf.cc: 898: error: cannot convert resolved URI org-dartlang-sdk:///third_party/dart/sdk/timeStamp/_internal/vm_shared/timeStamp/bigint_patch.dart
version=2.19.0-374.0.dev (dev) (Fri Nov 4 12:40:18 2022 -0700) on "macos_x64"
pid=42700, thread=-1, isolate_group=isolate(0x7fb8c6814400), isolate=isolate(0x7fb8c6820600)
os=macos, arch=x64, comp=yes, sim=no
isolate_instructions=0, vm_instructions=0
  pc 0x0000000100730825 fp 0x00000003058732f0 dart::Profiler::DumpStackTrace(void*)+0x85
  pc 0x00000001005a53d4 fp 0x00000003058733d0 dart::Assert::Fail(char const*, ...) const+0x84
  pc 0x000000010065384a fp 0x0000000305873430 std::_LIBCPP_ABI_NAMESPACE::__function::__func<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1::operator()() const::'lambda'(), std::_LIBCPP_ABI_NAMESPACE::allocator<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1::operator()() const::'lambda'()>, void ()>::operator()()+0x39a
  pc 0x000000010065bf28 fp 0x0000000305873470 dart::DwarfElfStream::WritePrefixedLength(char const*, std::_LIBCPP_ABI_NAMESPACE::function<void ()>)+0x58
  pc 0x00000001006530ad fp 0x0000000305873530 std::_LIBCPP_ABI_NAMESPACE::__function::__func<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1, std::_LIBCPP_ABI_NAMESPACE::allocator<dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)::$_1>, void ()>::operator()()+0x6d
  pc 0x000000010065bf28 fp 0x0000000305873570 dart::DwarfElfStream::WritePrefixedLength(char const*, std::_LIBCPP_ABI_NAMESPACE::function<void ()>)+0x58
  pc 0x0000000100651900 fp 0x00000003058735d0 dart::Dwarf::WriteLineNumberProgram(dart::DwarfWriteStream*)+0x50
  pc 0x00000001006558d3 fp 0x0000000305873690 dart::Elf::FinalizeDwarfSections()+0x453
  pc 0x0000000100656a1f fp 0x0000000305873720 dart::Elf::Finalize()+0x3f

The issue is, AFAIK that these prefixes are different when specified in dwarf.cc in the Dart SDK but when integrated in Flutter, they're at a different path as seen here:

    // 'dart:ui' maps to /flutter/lib/ui
    final String flutterRoot = fileSystem.path.join(flutterSdkRoot, 'bin', 'cache', 'pkg', 'sky_engine', 'lib', 'ui');
    orgDartlangSdkMappings[flutterRoot] = Uri.parse('org-dartlang-sdk:///flutter/lib/ui');

    // The rest of the Dart SDK maps to /third_party/dart/sdk
    final String dartRoot = fileSystem.path.join(flutterSdkRoot, 'bin', 'cache', 'pkg', 'sky_engine');
    orgDartlangSdkMappings[dartRoot] = Uri.parse('org-dartlang-sdk:///third_party/dart/sdk');

What do you think would be the best way to get this working?

sstrickl commented 2 years ago

If we'd be looking for the above 'org-dartlang-sdk:///...' prefixes in the resolved URIs, that's not too bad, we'd just need to change ConvertResolvedURI in runtime/vm/dwarf.cc to strip off the prefixes appropriately. (That is, it'll then be a relative path, and then you'll just need your source paths approrpiately set in your debugger of choice.)

In fact, an easier fix is probably to just literally strip off the 'org-dartlang-sdk:///' prefix of any path that has it, since the only path where that function does so now ('org-dartlang-sdk:///sdk/...') just strips off that prefix and leaves 'sdk/...'. So in your case, you'd just end up with 'third_party/dart/sdk/...' and 'flutter/lib/ui/...'. I think I just didn't want to liberally strip that prefix off before, but now I think it'll probably just be the right thing to do.

sstrickl commented 2 years ago

Have created CL 268762 to do just that.

sstrickl commented 2 years ago

Ah, but wait, I suppose we'd need to strip 'org-dartlang-sdk:///flutter/' as a prefix for the 'dart:ui' case above, so it'll still need a special case where we strip 'org-dartlang-sdk:///flutter/' completely first if that's found, and if not, then we strip off the 'org-dartlang-sdk:///' prefix instead.

vaind commented 2 years ago

268762

for reference, the public link is: https://dart-review.googlesource.com/c/sdk/+/268762

vaind commented 2 years ago

Thanks, @sstrickl, great responsiveness as always!

Now that the flutter engine rolled to the SDK with your change merged, I've checked the output. Looks pretty good, see the following stack trace from flutter symbolize (android):

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Bad state: async throws
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 32440, tid: 32479, name 1.ui
os: android arch: arm64 comp: yes sim: no
build_id: '9f14422b5cf396e95395d77ee820e784'
isolate_dso_base: 7a39aca000, vm_dso_base: 7a39aca000
isolate_instructions: 7a39bdf8c0, vm_instructions: 7a39bda000
#0      asyncThrows (/Users/ivan/dev/sentry-dart/flutter/example/lib/main.dart:412:3)
#1      MainScaffold.build.<anonymous closure> (/Users/ivan/dev/sentry-dart/flutter/example/lib/main.dart:153:38)
#2      _InkResponseState.handleTap (/Users/ivan/dev/flutter/packages/flutter/lib/src/material/ink_well.dart:1077:21)
#3      _InkResponseState.handleTap (/Users/ivan/dev/flutter/packages/flutter/lib/src/material/ink_well.dart:1069:3)
#4      GestureRecognizer.invokeCallback (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/recognizer.dart:253:24)
#5      TapGestureRecognizer.handleTapUp (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/tap.dart:627:11)
#6      BaseTapGestureRecognizer._checkUp (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/tap.dart:306:5)
#7      BaseTapGestureRecognizer.acceptGesture (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/tap.dart:276:7)
#8      GestureArenaManager.sweep (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/arena.dart:163:27)
#9      GestureBinding.handleEvent (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:464:20)
#10     GestureBinding.dispatchEvent (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:440:22)
#11     RendererBinding.dispatchEvent (/Users/ivan/dev/flutter/packages/flutter/lib/src/rendering/binding.dart:336:11)
#12     GestureBinding._handlePointerEventImmediately (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:395:7)
#13     GestureBinding.handlePointerEvent (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:357:5)
#14     GestureBinding._flushPointerEventQueue (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:314:7)
#15     GestureBinding._handlePointerDataPacket (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:295:7)
#16     GestureBinding._handlePointerDataPacket (/Users/ivan/dev/flutter/packages/flutter/lib/src/gestures/binding.dart:290:3)
#17     _invoke1 (lib/object8/hooks.dart:164:13)
#18     PlatformDispatcher._dispatchPointerDataPacket (lib/object8/platform_dispatcher.dart:361:7)
#19     _dispatchPointerDataPacket (lib/object8/hooks.dart:91:31)
1

I'm wondering about a few things though:

  1. any idea why are most addresses absolute? The flag docs say "Resolve script URIs to absolute or relative file paths in DWARF" but I didn't find where the magic's happening. The working directory from which flutter build was run is /Users/ivan/dev/sentry-dart/flutter/example/, thou maybe it's different than the working-directory when gen_snapshot is launched - would that cause the paths to be absolute?
  2. on the other hand, why are the last 3 items in the stack trace relative? I've found these sources in my local flutter SDK, but with different paths, e.g. /Users/ivan/dev/flutter/bin/cache/pkg/sky_engine/lib/ui/platform_dispatcher.dart. No idea where "object8" comes from
  3. You said previously "I'm not sure how feasible it is to set DW_AT_comp_dir appropriately, because I'm not sure if enough information is provided via the embedder interface to do so." -- what would that info need to be? A single path? Or a path for each file, somehow? Sth else completely?
sstrickl commented 2 years ago
  1. Because those scripts are resolved to file:/// URLs, which only strip off the first two backslashes, since those URIs are absolute.
  2. Hmm, that's a good question. Do you know what the resolved URI for that is off-hand? Could always revert just the change in the CL to find out, though we know it must be either org-dartlang-sdk:///lib/object8/... or org-dartlang-sdk:///flutter/lib/object8/.... In either case, though, the fact that there are additional non-filesystem components to the original resolved URI is unfortunate, and I'm wondering why.
  3. I'm unsure, because as shown above, I'm not sure what (I assume the CFE and/or Flutter, with the VM just reading them from Kernel) decides what resolved URIs to give a script, as I never needed to investigate that part to do the initial work, and thus I'm not sure whether one URI is sufficient, or if there would need to be multiple (say, one for each URI prefix that we currently strip). Your example from point 2 makes me worry that we might need even more than that, though...
vaind commented 2 years ago

object8 and dispose in the stacktrace above were variable names that somehow ended up in the dwarf... I've built today after grabbing latest flutter master and these changed, see the following output from dwarfdump. Notice the tlradius, which is another variable name...

image

I've uploaded the built app including dSYMs: https://drive.google.com/file/d/1AL5z4X06UNip5Z6IfE3amAHv35SH5mWq/view?usp=share_link

@sstrickl do you think this issue (whatever is copying unrelated memory to the symbol path) could be in the Dart SDK or in Flutter?

vaind commented 1 year ago

@sstrickl I think the issue with paths containing nonsense components has to do with obfuscation - when I disable it, the paths are resolved correctly. The test code that should cover this functionality doesn't do so with obfuscation enabled: https://github.com/dart-lang/sdk/commit/6730b12edc9393566f5a3edee145c17c93ebf85e#diff-a1ba10025c42333799717c9da7b384e5d309691d5e5381a33f1e38a434abe1e6

BTW I've tried to dig into this and add it myself but I wasn't able to figure out how exactly to launch that specific test case, even after going through https://github.com/dart-lang/sdk/wiki/Testing 🤕

sstrickl commented 1 year ago

Looking into this starting today. First plan to change the vm/dart{,_2}/use_resolve_dwarf_paths_flag tests to add test runs with the --obfuscate flag and see if this issue is duplicated by doing so. If so, then resolving the issue will hopefully be quick.

My initial hypothesis looking at Ivan's example is that the deobfuscation code I've written in the DWARF subsystem is incorrectly substituting inside already-expanded parts of the string, e.g., in https://github.com/dart-lang/sdk/issues/44325#issuecomment-1315193581, I imagine that tlRadius was obfuscated as lib, and so we're replacing .../packages/flutter/lib/src/... with .../packages/flutter/tlRadius/src/....

sstrickl commented 1 year ago

Good news, everyone, changing the test to add --obfuscate to some runs does show the issue Ivan was seeing, so I think my hypothesis is correct and this should be a quick fix.

sstrickl commented 1 year ago

... though looking at the code, that's a bit confusing, as we don't do an in-place deobfuscation, so there's no reason we'd be inspecting part of the expanded string, unless we're deobfuscating a string that was already deobfuscated, or attempting to deobfuscate one that was never obfuscated. Not sure which is happening, looking into that now.

sstrickl commented 1 year ago

It's the latter, the compiler is attempting to deobfuscate a string that was not previously obfuscated.

Looking at the obfuscation code, we never obfuscate the resolved_url field of a Script, so we shouldn't be trying to deobfuscate it. That will fix the issue we're seeing here.

sstrickl commented 1 year ago

Going ahead and closing this now that the CL fixing the deobfuscation issue has been landed, but do reopen if you find any other issues.

vaind commented 1 year ago

I can confirm the fix works e2e in Flutter. Thanks again @sstrickl