Develop: Registry virtualization engine

[GoogleCodeExporter]

The current registry virtualization is completely broken and unreliable.

The log would start with entries like this, and continue like this for the 
next few 100 lines:
...
2010-03-04 15:40:32.817000 [Debug] [Guest 10008] [8] QueryValue(HKey=896 
ValueName=CurrentConfig) => InvalidHandle
2010-03-04 15:40:32.855000 [Debug] [Guest 10008] [8] CloseKey(HKey=924)
2010-03-04 15:40:33.182000 [Debug] [Guest 10008] [8] OpenKey(-2147483646\
\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer) => 931
2010-03-04 15:40:33.196000 [Debug] [Guest 10008] [8] OpenKey(-2147483646\
\software\microsoft\windows\currentversion\policies\explorer) => 931
...
Eventually (if lucky) the targeted process starts, making it look like the 
virtualization engine "works".
But in reality the virtualization engine provides keys starting at handle 
21, obviously handles like 931 aren't provided by AppStract if the engine 
didn't provide the previous 920 handles.
OpenKey is able to return those handles because they are buffered an 
written to the databases by the virtual registry.

This issue is submitted to be able to keep track of enhancements and fixes 
to the virtualization layer.

Original issue reported on code.google.com by simon_al...@hotmail.com on 9 Mar 2010 at 7:30

[GoogleCodeExporter]

Revision 224:
Major rework of registry virtualization implementation, includes the following:
- Reworked RegistryHelper and moved some of its functionality to the new 
HiveHelper 
class.
- Moved methods from RegistryBase to RegistryHelper.
- Updated the IRegistryProvider interface to use NativeResultCode as return 
values.
- Merged VirtualRegistry with RegistryProvider.
- Renamed VirtualRegistryData to VirtualRegistry.
- Reworked RegistryBase so all its class variables are kept private.
- Clean up and minor bug fixes for all classes related to registry 
virtualization.

Results:
- Virtualization engine provides its own handles now, starting at 21
- Logs still contain duplicate lines, like:
2010-03-10 11:03:35.179919 [Debug] [Guest 5656] [9] 
OpenKey(-2147483646\\Software
\Microsoft\CTF\TIP\{0000897b-83df-4b96-be07-0fb58b01c4a4}\LanguageProfile
\0x00000000\{0001bea3-ed56-483d-a2e2-aeae25577436}) => 23
2010-03-10 11:03:35.179919 [Debug] [Guest 5656] [9] QueryValue
2010-03-10 11:03:35.180919 [Debug] [Guest 5656] [9] 
OpenKey(-2147483646\\software
\microsoft\ctf\tip\{0000897b-83df-4b96-be07-0fb58b01c4a4}\languageprofile
\0x00000000\{0001bea3-ed56-483d-a2e2-aeae25577436}) => 23
2010-03-10 11:03:35.346919 [Debug] [Guest 5656] [9] QueryValue(HKey=23 
ValueName=Enable) => FileNotFound
2010-03-10 11:03:35.347919 [Debug] [Guest 5656] [9] CloseKey(HKey=23)
- Target processes start but might (rarely) run into an access violation 
exception, 
which isn't caught the visual studio debugger

Original comment by simon_al...@hotmail.com on 10 Mar 2010 at 9:22

• Changed state: Started

[GoogleCodeExporter]

Correction on comment 1:
Target processes don't run into an access violation exception,
the exceptions were caused by pending changes of this revision (see last item 
of the 
results summary)

Revision 228:
Introduced a thread management system for the API hook handlers.

Results:
- Hook handlers are not interrupted anymore by other API hooks.
- No more duplicate lines in the logs.
- The management system uses native thread id's (acquired with 
AppDomain.GetCurrentThreadId()), which is not 100% reliable for managed threads
- The target process crashes whenever the handler for RegQueryValueEx is able 
to 
retrieve a value

RegQueryValueEx, extracted log lines:
2010-03-11 16:14:59.010919 [Debug] [Guest 17092] [8] 
OpenKey(2147483648\\Directory) 
=> 30
... irrelevant lines ...
2010-03-11 16:14:59.463919 [Debug] [Guest 17092] [8] QueryValue(HKey=30 
ValueName=AlwaysShowExt) => Succes
#End of log, process crashes without being able to report an exception to the 
attached debugger

After extensive debugging the cause of the issue is tracked to
MarshallingHelpers.ToByteArray(object)

Original comment by simon_al...@hotmail.com on 11 Mar 2010 at 2:42

[GoogleCodeExporter]

Revision 231:
Added Object.ToByteArray() and IntPtr.Write() extension methods.
Added support for enum types to all extension methods.
Revision 233:
Registry virtualization engine now makes use of the IntPtrExtensions-class in
stead of the MarshallingHelpers-class.

Results:
- All output data is passed reliably to the hooked functions
- All input data is serialized correctly to byte arrays
- The virtualization engine seems to work now

Considerations:
- Support for managed targets is broken, the main method can't be invoked while 
the 
API hooks are active
- Installers are in some cases able to read the host's registry (depending on 
AccessMechanism), thereby they might detect they are "already installed" (eg 
the VLC 
media player installer)

Original comment by simon_al...@hotmail.com on 12 Mar 2010 at 9:04

[GoogleCodeExporter]

The registry virtualization engine is now considered to be functional and 
stable.
The last two considerations are moved to Issue 14

Original comment by simon_al...@hotmail.com on 1 Apr 2010 at 3:22

• Changed state: Fixed