Granary / granary2

Dynamic binary translation framework for instrumenting x86-64 user space Linux programs
MIT License
39 stars 5 forks source link

Investigate duplicate targets in indirect targets #37

Closed pgoodman closed 9 years ago

pgoodman commented 10 years ago

While debugging a kernel bug, I noticed that it seems like indirect jumps are likely to have duplicated targets. It might mean that the best solution to this problem is to expand the granularity of the lock in the IndirectEdge data structure and to put in a TinySet to record the branch targets and to ensure that targets are not repeated. Having a set of targets in place, as well as a more coarse-grained lock will eventually help with flushing parts of the code cache. One thing blocking that from moving forward is how to handle and re-link the indirect jumps.

Another thing to investigate is to see if a good way to enable re-linking of indirect jump tests is to have the code linked list go through an indirect jump to find the target to get to the next comparator. This seems like it would make these structures much friendlier to re-linking.

pgoodman commented 10 years ago

This seems to be a big issue when trying to instrument VLC and libvlccore. One thing that exacerbates this issue is that meta-data associated with non-comparable CompensationBasicBlocks is put into the code cache index. Perhaps the "right thing to do" is to simply not associated meta-data with these blocks, as otherwise it means either:

I think a real solution might eventually need to associate meta-data with only the entry block or DirectBasicBlock / CachedBasicBlocks. Then, a separation of indexable/unifiable and mutable meta-data needs to exist, such that mutable is maybe completely separate and managed by each tool. This could make a lot more sense I think.

Anyway, this needs to be looked into.

pgoodman commented 10 years ago

There seems to be at lease one very long function in VLC that does many different indirect calls through the same register, and so is potentially the source of this problem. Nonetheless, I hit an OOM with vlc, which is clearly a problem.

pgoodman commented 10 years ago

This is coming up as an issue where vlc_entry__main calls vlc_plugin_setter a ridiculous amount of times, presumably because of the things to add command-line options.

I think the indirect edge entrypoints might have gone screwy since some of my recent changes to instruction save/restore for registers, as well as for conditional stack switching (more likely this). In any event, I need to look into this!

pgoodman commented 10 years ago

I think I have resolved the VLC-specific issue, which was actually a few different kinds of memory leaks.

pgoodman commented 9 years ago

This was resolved some time ago.