dwarf v4 debug types support

[driftregion]

This builds on #520.

I've added a couple of .elf files produced by TI Code Composer studio C2000 compiler to the corpus.

My use-case is currently unblocked by these changes. pyelftools can parse the TI CCS files without crashing. However I haven't yet achieved llvm-dwarfdump consistency. Two issues have been solved but I'm at a loss on the third:

The relevant function is _desc_expression(expr, die): in dwarfdump.py

https://github.com/eliben/pyelftools/blob/master/scripts/dwarfdump.py#L277

And the mismatch is shown below:

....for file test/testfiles_for_dwarfdump/dwarf_v3_ticcs.elf
....for option "--debug-info"
....Output #1 is llvm-dwarfdump, Output #2 is pyelftools
@@ Mismatch on line #53:
>>                dw_at_location [dw_form_block1]       (dw_op_reg0)<<
>>              dw_at_location [dw_form_block1] (dw_op_reg0 r0)<<

Unlike the previous inconsistencies, when I attempt to modify this code I quickly break the other dwarfdump tests. I'm not yet sure if or how this should be special-cased.

[eliben]

@sevaa

[sevaa]

@driftregion

First off, check the version of llvm-dwarfdump that you are testing against. Output format is not a contract, tool vendors change it all the time. We don't update llvm-dwarfdump in the test scripts folder every time they drop. This particular change looks like a bug fix on the LLVM side; DW_OP_regX doesn't take any arguments.

[driftregion]

Hi seva, thanks for your response.

check the version of llvm-dwarfdump that you are testing against.

I'm running the tests on linux with the command

python test/run_dwarfdump_tests.py

I've confirmed that running this command uses test/external_tools/llvm-dwarfdump.

furthermore, I've tried using the dwarfdump installed on my machine by modifying test/run_dwarfdump_tests.py:38 to point to llvm-dwarfdump-15. it produces the same output as above https://github.com/eliben/pyelftools/pull/530#issue-2060868106

llvm-dwarfdump-15  --version
Ubuntu LLVM version 15.0.7
  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: goldmont

test/external_tools/llvm-dwarfdump --version
LLVM (http://llvm.org/):
  LLVM version 15.0.0git
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: goldmont

This particular change looks like a bug fix on the LLVM side; DW_OP_regX doesn't take any arguments.

Got it. Is there something other than the DWARF spec that specifies this? I'm reading DWARF Debugging Information Format, Version 4 and from this alone it is ambiguous to me whether DW_OP_reg0 takes arguments or not. Is there a more authoritative resource?

I had a quick look at the llvm bugtracker and found only this https://bugs.llvm.org/show_bug.cgi?id=38714 which seems to indicate that the arguments to DW_OP_reg are used.

[sevaa]

DWARFv5 has been out for several years now. However, the DW_OP_regX operations have been stable since DWARFv2. The register number is a part of the opcode; one may interpret it as an implicit argument, but a low level parser, such as pyelftools, would be advised not to.

This is not about the spec, this is about matching the output of llvm-dwarfdump which is not governed by any spec. Have you tried looking at the sources of llvm-dwarfdump, see if there are any conditionals around that spot? You can build it for debudding and debug it, too.

[driftregion]

Hey Seva, thanks for taking me on this adventure. This is wild. I learned a bunch of new gdb tricks and found that building llvm takes 114G of disk.

I found the conditional:

llvm-project/llvm/lib/DebugInfo/DWARF/DWARFExpression.cpp:275

  auto RegName = DumpOpts.GetNameForDWARFReg(DwarfRegNum, DumpOpts.IsEH);
  if (!RegName.empty()) {

GetNameForDWARFReg points to a lambda which returns a StringRef which is empty in the case of the TI binary test/testfiles_for_dwarfdump/dwarf_v3_ticcs.elf. This is because createRegInfo returns NULL MCRegInfo for the TI binary due to target lookup failure.

test/testfiles_for_dwarfdump/dwarf_gnuops4.so.elf has non-NULL MCRegInfo and therefore prints the register names.

[sevaa]

So looks like the logic is, for the handful of CPUs that llvm-dwarfdump is aware of, there is the logic of translating the register number to a friendly register name. Your TI binaries target a rather exotic (in the grand scheme of things, not in your own life) CPU. We have a similar logic in describe_reg_name() under elftools/dwarf/descriptions.py. It would return None for your TI binaries. The invokation of that function is in _desc_operation() in dwarfdump.py. If describe_reg_name() returns None, it should not print the friendly register name, but it does (?). Try debugging that piece on the Python side of things.

[driftregion]

Great. python test/run_dwarfdump_tests.py passes now.

Note: TI's DWARF extensions (https://www.ti.com/lit/an/spraab5/spraab5.pdf?ts=1705994928599) are in conflict with and are incorrectly reported as HP dwarf attributes by llvm-dwarfdump (llvm/include/llvm/BinaryFormat/Dwarf.def:446)

[sevaa]

Get rid of the custom dwarfdump please, or it will get in the way of the CI autotest. If you'd like an option to point at a dwarfdump of your choice, use an environment variable.

[driftregion]

done in a612ffe

[nickvazz]

@eliben Wondering if this will get pulled in / released anytime soon. Thank you!

[eliben]

@sevaa has been reviewing this, so I'll let him finish the review. @driftregion have all of @sevaa 's comments been addressed at this point?

[driftregion]

@eliben , yes afaik

[sevaa]

Yes.