Callframe information parsing issues

Maxicu5 commented 2 months ago

For the ELF files (generated by IAR) where the referenced CIE may go after the FDE. CIE is loaded in the scope of FDE and placed into a cache, then when the queue comes to that CIE it takes it from the cache and doesn't move a stream offset, so this function stuck:

    def _parse_entries(self):
        entries = []
        offset = 0
        while offset < self.size:
            entries.append(self._parse_entry_at(offset))
            offset = self.stream.tell()
        return entries

locally (temporary) fixed like this (it works):

  def _parse_entries(self):
    entries = []
    offset = 0
    while offset < self.size:
      e = self._parse_entry_at(offset)
      entries.append(e)
      offset = e.offset + e.header.length +e.structs.initial_length_field_size()
    return entries

Also, CIE of version 4 is parsed by the structs of version 2, but there are two additional fields, so after the header everything is shifted, I had to add the fix in two places, seems it works:

        if self.for_eh_frame:
            is_CIE = CIE_id == 0
        else:
            is_CIE = (
                (dwarf_format == 32 and CIE_id == 0xFFFFFFFF) or
                CIE_id == 0xFFFFFFFFFFFFFFFF)

        # Parse the header, which goes up to and excluding the sequence of
        # instructions.
        if is_CIE:
>           version = struct_parse(entry_structs.Dwarf_uint8(''), self.stream)
>           entry_structs = DWARFStructs(
>               little_endian=self.base_structs.little_endian,
>               dwarf_format=dwarf_format,
>               address_size=self.base_structs.address_size,
>                dwarf_version=version)
            header_struct = (entry_structs.EH_CIE_header
                             if self.for_eh_frame else
                             entry_structs.Dwarf_CIE_header)
            header = struct_parse(
                header_struct, self.stream, offset)
        else:
            header = self._parse_fde_header(entry_structs, offset)

        # If this is DWARF version 4 or later, we can have a more precise
        # address size, read from the CIE header.
        if not self.for_eh_frame and entry_structs.dwarf_version >= 4:
            entry_structs = DWARFStructs(
                little_endian=entry_structs.little_endian,
                dwarf_format=entry_structs.dwarf_format,
                address_size=header.address_size,
>                dwarf_version=entry_structs.dwarf_version)

P.S. Very good library. Looking forward removing my workarounds and use fixed version soon.

sevaa commented 2 months ago

Thanks for letting us know. Lack of CIEv4 header support is clearly an oversight. In our readelf autotest corpus, there are no files with V4 CIEs - I've checked, only 1 and 3 (despite DWARF proper being anywhere from 2 to 5). So it never came up. That I'll fix in a PR. Can you share a binary for our autotest, please? Doesn't have to be a real one, a "Hello world" would suffice - as long as it has V4 CIEs.

The caching issue is slightly more subtle. The only user facing function in CallFrameInfo is get_entries() - and by its parse-and-store nature, it doesn't suffer from the cache/stream offset issue. The underscore prefixed class methods are considered, by Pythonic convention, an implementation detail - sort of private (in the OOP sense), if you will. To the best of my knowledge, the frames section doesn't allow for random or indexed access - only sequential. What is your usage scenario exactly so that calling _parse_entries() while some entries are already cached came up?

Maxicu5 commented 2 months ago

Related to the test ELF file, will see what can I do, I'm very busy for now, but maybe it will help you: as I remember it was not the part of my code. My code has generated CIEv2 or 3, but that was IAR built-in linked library for ARM double precision float point processing, probably like this one 'm7M_tls.a'. I can't attach it as it is a part of licensed product, but it should be easy to find it, I think. Related to the caching, probably I was bad in explanation: When CIE goes before all the references to it, there will be no issues, but in my case, the ELF file was generated like this: FDE->CIE->FDE->FDE->FDE All this FDEs are pointing to the same CIE, and the first FDE is going ahead, so in the scope of loading first FDE, you request the CIE which is not loaded yet, and it is cached after that. Then, after the first FDE, you request the CIE which is already in the cache and the stream offset is not updated. It is probably compiler specific, I haven't try to find the requirements to the sequence in the specs, but the fact it is possible to get such sequence on practice.

sevaa commented 2 months ago

So the v4 CIEs sit in the debug info of a run-time library that ships with a proprietary compiler. It would probably be a copyright violation to share the RTL file by itself, but it definitely won't be if you build a binary of your own, linked against said RTL, and share that. If you do so, put some calls to make sure the library is not optimized away.

Understood re: the FDE ahead of its CIE issue. The logic in _parse_cie_for_fde() will throw off the parse/cache loop, if it looks ahead instead of back. Will fix.

@Maxicu5: EDIT: I have a patch in the works, but I really need a test binary. At least for the first issue.

Maxicu5 commented 2 months ago

I've rechecked the library with CIE v4 and I was wrong, it is not the one from proprietary compiler, it is this one 'stm32wb_zigbee_wb_lib.a', it can be simply found on the github. I've tried to build something with it, but any single function has a lot of dependencies on some interfaces, and the examples have a lot of build errors out of the box, maybe you will be able to find something prebuilt with it or to use the objects itself from inside of this lib archive as the "debug_frame" looks the same for ELF and for the obj

sevaa commented 2 months ago

Can you please track one of the v4 CIEs to a specific function in said library? The debug_frame section in object files within the library looks weird, I suspect the linker is doing a lot of magic while building the executable.

sevaa commented 2 months ago

@eliben: it's crazier than I thought. In an object file, such as those found in this static library, there can be multiple sections with the same name - debug ones included. Specifically, there are two sections called .debug_frames in an object file I'm staring at right now. GNU readelf handles that fine, pyelftools doesn't, and it will necessitate an API change. How do you want to approach that? Superficially, object files are ELF and pyelftools opens them - if you disable relocation.

Linking-type relocation for an object file means something completely different than loading time relocation.

I can sort of see how this might work specifically for debug_frames, since it has no cross-section references, but I don't think it can be made to work with other DWARF sections (except maybe ones that link out but not in - e. g. aranges).

eliben commented 2 months ago

How does readelf handle it? We can borrow the approach

sevaa commented 1 month ago

Moved the .o stuff into #564. I'll take a closer look at this once I'm done poking around readelf's bugs.