DynamoRIO / dynamorio

Dynamic Instrumentation Tool Platform
Other
2.67k stars 562 forks source link

Add support for -use_physical with drcachesim -offline #4014

Open derekbruening opened 4 years ago

derekbruening commented 4 years ago

-use_physical is only supported for online drcachesim today, but that is not made clear in the docs nor in actual usage. The tool lets you run -offline -use_physical and ends up with a post-processed trace with physical data addresses but virtual PC fetches, due to an accident. instr_offline_t::get_entry_addr blindly treats the entry as a data ref, so it tries to translate bogus addresses composed of module id + offset encodings.
And indeed we get warnings at -verbose 1+:

virtual2physical translation failure for <10,  0, 0x0004000c00000f30>
virtual 140727803080432 => physical 3824
virtual2physical translation failure for <10,  0, 0x002c000c00001d30>
virtual2physical translation failure for <10,  0, 0x0016000c00001d85>
virtual2physical translation failure for <10,  0, 0x0004000c00001de3>
virtual2physical translation failure for <10,  0, 0x000a000c00001dd3>
virtual 140346642070176 => physical 2720
virtual 140346642067080 => physical 3720
virtual2physical translation failure for <10,  0, 0x0004000c00001de3>
...

Those are the instruction entries:

$ od -t x8 -A x `ls -1td drmemtrace.threadsig.*.dir | head -1`/raw/*.raw | head
000000 c000000000000003 4000000000030385
000010 6000000000030385 802efaa01299a30c
000020 c203000000000003 2004000c00000f30
000030 0000000000000ef0 202c000c00001d30
000040 0000000000000ee8 0000000000000ee0
000050 0000000000000ed8 0000000000000ed0
000060 0000000000000ec8 0000000000000ec0
000070 2016000c00001d85 2004000c00001de3
000080 200a000c00001dd3 0000000000000aa0
000090 0000000000000e88 2004000c00001de3

So we have two action items. First, we should have the front-end refuse to combine -use_physical and -offline for now, and update the docs to reflect this.

Next, we need to decide whether to try to support this combination. Is it possible that future Linux distros will all shut down access to pagemap? Although for research purposes running as sudo or something may still be feasible.

How would we support this? We'd either have to store extra info for each data ref plus an entry for each instr like for DGC (#2062), or have a custom solution just for offline where each block PC has 2 entries: one virtual and one physical. The former will not work well w/ tools that want operands (like opcode_mix or micro-arch simulators): but that would be the same for DGC, so we might need an option to store the full instr bytes.

Another limitation today (which perhaps should have its own issue) with -use_physical relates to static linking which we often use for offline tracing:

        /* Unfortunately the use of std::unordered_map in physaddr_t calls malloc
         * and thus we cannot support it for static linking, so we override the
         * DR_DISALLOW_UNSAFE_STATIC declaration.
         */
        dr_allow_unsafe_static_behavior();
#ifdef DRMEMTRACE_STATIC
        NOTIFY(0, "-use_physical is unsafe with statically linked clients\n");
#endif

Xref #2912 but that issue has many confusing entries so it seemed better to start clean here.

derekbruening commented 2 years ago

Previously, physaddr_t was just used racily. I made it per-thread all at once. This does result in a per-thread file descriptor being opened, which may not scale well: it will exhaust DR's private file-descriptor space and could possibly hit rlimits. Improving scaling is a still-open action item.