Closed sevaa closed 4 months ago
@eliben
On the topic of performance, there is a disconnect between ELF proper parsing and DWARF parsing. ELF parsing, at least in the typical use case, happens against a file stream. DWARF parsing is all in memory (barring exotic scenarios where users monkeypatch or subclass ELFFile
), and yet we still treat DWARF sections as streams instead of byte arrays in memory that they really are. Rather than read()
ing (and constructing bytes
objects) all the time, the parser could have worked with bytes
elements and slices. The workhorse method of parsing primitive datatypes - struct..unpack()
- can take a buffer and an offset.
But that would pretty much mean abandoning construct
for the DWARF part of it. As far as I understand, construct
was meant to work like protobuf
, working with wire and file streams.
Yes, I'd rather keep this capability. A more interesting direction would be to enable more incremental DWARF parsing without slurping whole sections into memory
Reliance on construct is not a capability per se, it's more of an implementation detail. The DWARF parser mostly spits out Python native structures - lists, namedtuple
s and such. Off the top of my head, the only Construct
objects that we get in the public API are CU headers. And the interface of Construct
, one there x["a"]
and x.a
are equivalent, is hardly magical.
OBTW, construct technically has a parse-from-buffer method. Only it works by constructing a BytesIO
around it, then parses the stream :) Overhead wise, not a net win.
Anyway, were I to implement buffer style parsing, I won't be getting rid of construct altogether - compound datatypes can stay. I'd implement a buffer+position object that walks like a stream but doesn't quite quack like a stream, and I'd teach the primitive type parsers (there is just a handful) to recognize those. The compound parsers - Struct
, Array
- can stay as they are.
I gave some thought to incremental parsing of DWARF. One big obstacle to that would be the transforms that DWARF sections undergo - two decompression hooks, and relocations (also the phantom bytes thing, but that's an edge case by far). Compressed streams are not seekable. Also, I'm assuming we are talking about support for extra large binaries here; were were to implement a no-slurp mode, I'm afraid it'd have to be accompanied by a no-cache mode. At least no CU/DIE cache - lack of an abbrev cache, I'm afraid, would be to costly.
But let's assume there are no transforms. I had three possible designs in mind:
dup
won't help, dupped handles share their current position)ByteIO
objects around sections and trust the OS to do the right thingWhich one do you think sounds the least crazy?
The mmap idea sounds intriguing
Basically already done this in #481 :)
I made the following enhancements, in the rough order of effect:
struct_parse
calls, and replaced all points of usagestream_preserve
for streams in auxiliary sections