The IMsRdpClientAdvancedSettings::LoadBalanceInfo property is exposed as a string to .NET and marshalled as a BSTR internally for COM Interop. The problem is that BSTRs normally use UTF-16 encoding internally, just like .NET strings do - but the LoadBalanceInfo property should be encoded as a UTF-8 string over the wire.
An old code snippet shared here in 2013 shows how to create a .NET string from UTF-8 bytes, and checks to ensure the string length is a multiple of two by appending a space character " " to the string.
The problem with this hack is that the extra space character is kept in the final encoding, making the LoadBalanceInfo string value fail to match the string returned by the RDP server, breaking the connection. I suspect the reason for the hack was to force a even character length - converting to the .NET managed string type, even if initialized with a byte array, shaves off the odd byte at the end (49 becomes 48, etc).
After investigating how mstsc.exe sets the string, it looks like this:
Passing the unmanaged BSTR created using SysAllocStringByteLen is not directly possible with the generated COM Interop DLLs for the RDP ActiveX, since it goes through a mandatory string type conversion. The only workaround is to figure out a way to call the same function directly using IntPtr instead of string, and using p/invoke to create an unmanaged BSTR.
The least-problematic strategy has been to define an alternate interface called IMsRdpClientAdvancedSettingsLB with a different function prototype for the LoadBalanceInfo getter/setter prototypes. I wish there was a way to explicitly set a vtable offset, but the best I could come up with was to define empty function definitions in order to get the order and offsets right.
The IMsRdpClientAdvancedSettings::LoadBalanceInfo property is exposed as a string to .NET and marshalled as a BSTR internally for COM Interop. The problem is that BSTRs normally use UTF-16 encoding internally, just like .NET strings do - but the LoadBalanceInfo property should be encoded as a UTF-8 string over the wire.
An old code snippet shared here in 2013 shows how to create a .NET string from UTF-8 bytes, and checks to ensure the string length is a multiple of two by appending a space character " " to the string.
The problem with this hack is that the extra space character is kept in the final encoding, making the LoadBalanceInfo string value fail to match the string returned by the RDP server, breaking the connection. I suspect the reason for the hack was to force a even character length - converting to the .NET managed string type, even if initialized with a byte array, shaves off the odd byte at the end (49 becomes 48, etc).
After investigating how mstsc.exe sets the string, it looks like this:
Passing the unmanaged BSTR created using SysAllocStringByteLen is not directly possible with the generated COM Interop DLLs for the RDP ActiveX, since it goes through a mandatory string type conversion. The only workaround is to figure out a way to call the same function directly using IntPtr instead of string, and using p/invoke to create an unmanaged BSTR.
The least-problematic strategy has been to define an alternate interface called IMsRdpClientAdvancedSettingsLB with a different function prototype for the LoadBalanceInfo getter/setter prototypes. I wish there was a way to explicitly set a vtable offset, but the best I could come up with was to define empty function definitions in order to get the order and offsets right.