Open tandasat opened 8 years ago
Looked into details of timer. A fix I thought, which was disabling timer at the entry point of VM-exit did not seem to be straightfoward at all. I will find some time to investigate more, but I am going to treat this issue as lower priority.
Here is my findings on the timer interrupt. An interrupt handler for the timer is 0xd1 on my tested system. This was identified by looking at IP when this issue was seen.
>!idt
...
d1: fffff801085c1bf8 hal!HalpTimerClockInterrupt (KINTERRUPT fffff801084574b0)
I initially thought it was triggered by Local APIC, but below output shows interrupt vector for the timer is 0xff and the Initial Count and Current Count Registers are both 0 indicating the time was disabled accodring to the "APIC Timer" section in the Intel SDM.
kd> !apic
Apic @ ffd0d000 ID:0 (50015) LogDesc:01000000 DestFmt:ffffffff TPR F0
TimeCnt: 00000000clk SpurVec:df FaultVec:e2 error:0
Ipi Cmd: 01000000`0004002f Vec:2F FixedDel Dest=Self edg high
Timer..: 00000000`000300ff Vec:FF FixedDel Dest=Self edg high m
Linti0.: 00000000`000100d8 Vec:D8 FixedDel Dest=Self edg high m
Linti1.: 00000000`00000400 Vec:00 NMI Dest=Self edg high
TMR: 77, 87, 97, B0
IRR: 66, 76, D1
ISR: D1
kd> !dd 0xfee00380
#fee00380 00000000 00000000 00000000 00000000
#fee00390 00000000 00000000 00000000 00000000
Later, I found that the interrupt 0xd1 was registered by IO APIC. I guess this is because timer is managed by HPET, which is a dedicated device located outside a processor.
kd> !ioapic
Controller at 0xffffffffffd004c0 I/O APIC at VA 0xffffffffffd0e000
IoApic @ FEC00000 ID:1 (11) Arb:1000000
...
Inti02.: 01000000`000008d1 Vec:D1 FixedDel Lg:01000000 edg high
...
It may be still possible to disable the HPET timer, but I would need to understand HPET well to do it. Moreover, time keeping is not only done by HPET. ACPI Power Management Timer (PM Clock), Local APIC timer or any other timers may be used, and I need to investigate all those posibilities to decide what to cover in this project.
Those are resources could be useful for further investigation.
I was trying aswell to bypass timing attacks made by apllications running on the Guest OS. My first thought was changing rdtsc
in an vmexit handler, but it doesn't seem to be called and the guest os isn't influenced by any changes made in this exit handler.
Did you made some progress on bypassing timing checks in the mean time?
This is off topic for this thread, but it should be relatively straightforward to modify results of RDTSC. Just change a guest's registers used by RDTSC in VMM. Also, The "Time-Stamp Counter Offset and Multiplier" section might be interesting for your purpose.
hi @tandasat, I posted about this issue here https://github.com/tandasat/DdiMon/issues/38 and the more I research the more I get confused as to why there's so many ways to change the TSC. For Intel there's
IA32_TSC_ADJUST IA32_TSC_OFFSET IA32_TIME_STAMP_COUNTER TSC scaling
And amd has an extra one for them TscRateMsr
Modifying IA32_TIME_STAMP_COUNTER directly accomplishes my end goal which is hiding cpu cycles from rdtsc but the issue is after awhile process windows start blacking out, or refusing to start and the computer is unusable. This happens even if I filter by process
_Use_decl_annotations_ static void VmmpHandleCpuid( GuestContext *guest_context) {
unsigned int cpu_info[4] = {};
const auto function_id = static_cast<int>(guest_context->gp_regs->ax);
const auto sub_function_id = static_cast<int>(guest_context->gp_regs->cx);
UCHAR Mode = VmmpGetGuestCpl();
LARGE_INTEGER StampCounter = {};
int DoChange = 0;
if (Mode == 3) //usermode
{
if (PsGetCurrentProcessId()==2222) {
StampCounter.QuadPart = UtilReadMsr64(Msr::IA32_TIME_STAMP_COUNTER);
DoChange = 1;
}
}
_cpuidex(reinterpret_cast<int *>(cpu_info), function_id, sub_function_id);
guest_context->gp_regs->ax = cpu_info[0];
guest_context->gp_regs->bx = cpu_info[1];
guest_context->gp_regs->cx = cpu_info[2];
guest_context->gp_regs->dx = cpu_info[3];
if (DoChange)
UtilWriteMsr64(Msr::IA32_TIME_STAMP_COUNTER, StampCounter.QuadPart - 600);
VmmpAdjustGuestInstructionPointer(guest_context);
}
Or if I filter by driver. Either way eventually(within 5 minutes) the system starts going haywire and isn't usable anymore. Do you have any idea how I can fix this?
Found this description
17.13.3 Time-Stamp Counter Adjustment ... Software can modify the value of the time-stamp counter (TSC) of a logical processor by using the WRMSR instruction to write to the IA32_TIME_STAMP_COUNTER MSR (address 10H). Because such a write applies only to that logical processor, software seeking to synchronize the TSC values of multiple logical processors must perform these writes on each logical processor. It may be difficult for software to do this in a way than ensures that all logical processors will have the same value for the TSC at a given point in time. The synchronization of TSC adjustment can be simplified by using the 64-bit IA32_TSC_ADJUST MSR ( address 3BH ). Like the IA32_TIME_STAMP_COUNTER MSR, the IA32_TSC_ADJUST MSR is maintained separately for each logical processor.
and reada comment on stackoverflow that a number of things in your system expect the IA32_TIME_STAMP_COUNTER to be constantly increasing so haven't found a way to keep it stable by decreasing yet.
The VMM does not virtualize the time stamp counter (TSC) or any other timers. This can lead timer interrupt (IDT d1: hal!HalpTimerClockInterrupt, on my test system) immediately after VM-enter when the VMM runs an VM-exit handler overly long time. While this is not an issue by itself, it could cause an infinite between a VM-exit and a timer interrupt handler under certain scenarios. A situation I have seen is as followings:
A quick fix would be streamlining the long run VM-exit handler, but the VMM should not limit what a developer can do on VM-exit in that manner. A more correct way to address this issue is hiding an elapsed time of VM-exit handler from a guest and protect guest context from triggering timer interrupt.