As far as I could track from the code, DetourUpdateThread enlists threads to be processed, and committing the transaction adjust the threads' instruction pointer register (let's go with x64, and call it RIP) depending on if we're adding/attaching, or removing/detaching.
I'm wondering about risk of these operations on 2 main points:
Is the integrity of the thread's state preserved? Any chance of an unbalanced stack push/pop for example?
In the case of removal, especially if they happen to be triggered from DllMain, detaching from process: Since the DLL code is to be removed, are there any guarantees from Detours that the removed code will not be read/(tried to be) executed? RIP shifting would take care if the RIP happened to be in our trampoline, but, what would happen if our code was deeper in the call stack (i.e. a return address from our module's address space exists somewhere in the stack)?
My first instinct is that, this guarantee cannot be provided, and the API user/developer should take care of it himself/herself probably by separating detours-detaching, and dll-unloading, and through some mechanism like reference-counting to ensure that no code of the Dll is still "active" in any of the threads.
(Please also take a look at issue #151 which is related to this API)
As far as I could track from the code,
DetourUpdateThreadenlists threads to be processed, and committing the transaction adjust the threads' instruction pointer register (let's go with x64, and call it RIP) depending on if we're adding/attaching, or removing/detaching.I'm wondering about risk of these operations on 2 main points:
My first instinct is that, this guarantee cannot be provided, and the API user/developer should take care of it himself/herself probably by separating detours-detaching, and dll-unloading, and through some mechanism like reference-counting to ensure that no code of the Dll is still "active" in any of the threads.
(Please also take a look at issue #151 which is related to this API)