'System.ArgumentException: Value does not fall within the expected range' when using Windows Forms user control within VB6 form via COM interop

.NET version

.NET 8.0 preview 5

Did it work in .NET Framework?

Yes

Did it work in any of the earlier releases of .NET Core or .NET 5+?

.NET 7

Issue description

Using a Windows Forms user control embedded into a VB6 form via COM interop fails, with an exception:

System.ArgumentException: Value does not fall within the expected range. at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode, IntPtr errorInfo) at Windows.Win32.System.Com.IGlobalInterfaceTable.RegisterInterfaceInGlobal(IUnknown pUnk, Guid riid, UInt32 pdwCookie) at Windows.Win32.Foundation.GlobalInterfaceTable.RegisterInterface[TInterface](TInterface interface) at Windows.Win32.Foundation.AgileComPointer`1..ctor(TInterface* interface, Boolean takeOwnership) at System.Windows.Forms.Control.ActiveXImpl.InPlaceActivate(OLEIVERB verb) at System.Windows.Forms.Control.ActiveXImpl.OnFocus(Boolean focus) at System.Windows.Forms.Control.ActiveXOnFocus(Boolean focus) at System.Windows.Forms.Control.ChildGotFocus(Control child) at System.Windows.Forms.Control.OnGotFocus(EventArgs e) at System.Windows.Forms.TextBox.OnGotFocus(EventArgs e) at System.Windows.Forms.Control.WmSetFocus(Message& m) at System.Windows.Forms.Control.WndProc(Message& m) at System.Windows.Forms.TextBoxBase.WndProc(Message& m) at System.Windows.Forms.TextBox.WndProc(Message& m) at System.Windows.Forms.NativeWindow.Callback(HWND hWnd, WM msg, WPARAM wparam, LPARAM lparam)

Steps to reproduce

I built a minimal repro application based on the code found in: https://github.com/dotnet/samples/tree/main/core/extensions/COMServerDemo

ComInteropTest.zip

To reproduce the issue, unzip the sample application to some folder and build the COM interop DLL using "dotnet build /p:UseNet8=True". After the build is done, register the created COM interop DLL according to the printed instructions. For example (this must be done in elevated command prompt): regsvr32.exe "C:\Repro\ComInteropTest\COMServer\bin\x86\Debug\COMServer.comhost.dll"

After registering the built COM interop DLL, you can reproduce the issue by running the included VB6 sample app in the folder "VB6TestClient" (source code is included). The folder includes a pre-built binary "Project1.exe", which when run opens a form, with a single command button. Clicking on the button opens another form with the embedded user control on it, which fails with the "System.ArgumentException: Value does not fall within the expected range" exception.

If you unregister the built DLL and re-build the project using "dotnet build", it will be built against .NET 7. If you then re-register the DLL and run the VB6 sample app, the form will open with no exceptions.

The code to embed the User Control within VB6 form is based on code from "Interop Toolkit 2.1" published by Microsoft a long time ago and it works on full .NET framework and still used to work on .NET 7, so I would imagine this to be a regression in some part of the stack. At least I didn't find any breaking changes notes that seem to be related to this issue.

Debugging this issue further a bit, it seems like the code fails in "Control.ActiveXImpl.cs" in the "InPlaceActivate" method when it tries to create "AgileComPointer" in "_inPlaceUiWindow = new(pWindow, takeOwnership: true);".

Looking at the call in debugger it seems like the "pFrame" is set to a value and the "pWindow" is set to null pointer, after the call to "GetWindowContext" is done. Looking at the public documentation for the call: https://learn.microsoft.com/en-us/windows/win32/api/oleidl/nf-oleidl-ioleinplacesite-getwindowcontext

It seems like this is expected behaviour sometimes (like it seems to be in my case): " [out] ppDoc

A pointer to an IOleInPlaceUIWindow pointer variable that receives the interface pointer to the document window. If the document window is the same as the frame window, ppDoc is set to NULL. In this case, the object can only use ppFrame or border negotiation. If an error is returned, the implementation must set ppDoc to NULL. "

The behaviour in .Net 7 codebase differs, since there is no "AgileComPointer", which fails if a null pointer is passed to it.

Hope this helps in helping to locate the issue and fix it, before .NET 8 is released.

@oryhanen Thank you so much for this! We did major refactoring here to utilize the Windows provided metadata (through CSWin32) and to move our code towards a place where we can enable trimming/aot scenarios (where COM interop doesn't work- hopefully in .NET 9).

This particular break was a mistake, but in general we didn't make deliberate breaking changes.

Here is what I know we've changed (we'll have this in the breaking changes docs soon):

In the COM projections of WinForms controls there are likely some very, very subtle differences around trying to grab public Control properties through IDispatchEx via DISPID where one isn't explicitly added via [DispID]. I'm manually providing the IDispatchEx implementation that historically came through IReflect, and I'm expecting that there are probably differences. I also dropped the Move method that was exposed in the proxy as it never actually did anything. See ExtenderProxy.

I would also expect that there would be slight differences in the ITypeInfo given back as we're letting Windows generate this with CreateStdDispatch.

Finally, we don't populate EXCEPINFO when there is a caught COR_E_TARGETINVOCATION. That's a pretty meaty feature and we don't expect that will break anyone.

Please feel free to tag me directly with any issues you have in this space. I'll try to validate the fix as soon as I can.

@oryhanen I'm having issues debugging this, curious how you're checking things? I'll continue to work on trying to repro with private bits, but I've put up the fix regardless as it's pretty obvious and that will get it into an official daily SDK build. See #9489.

@JeremyKuhne Hi, thanks for the fast response to this issue!

Yeah, debugging COM issues has always been "hard" and this was even harder than usual. To get the debugger to break in the correct place, I first had to register the COM interop class using regsrv32 as descibed in the initial report. Also obviously I had to change the debugging options to remove the check mark from the "Enable Just My Code" to allow the debugger to step into the code, which causes the issue (Debug => Options... => Enable Just My Code").

Then I start the VB6 Project1.exe and click on the "Command1" button, which opens the form, which causes the exception. At this point the COM interop DLL has been loaded into the vb6 process, which allows me to attach a visual studio debugger to the session via "Debug => Attach to Process..." and selecting "Project1.exe" from the list. Then I insert a breakpoint inside the "InteropTestUserControl" constructor at the call to "InitializeComponent();" method. In order to get the debugger to break in the breakpoint, I then have to close the "Form1", which causes the issue. To close it, I use the taskbar, which allows me to close the failing form without closing the application (which would kill the debugging session).

After doing this, I can get the debugger to break in the user control constructor by re-opening Form1 by clicking on the "Command" button again on the initial form, which allows me to dig deeper with the debugger (not an easy task even after this).

To get to the real issue, I step over the call the "InitializeComponent();", with F10, and the continue stepping into the code using F11, which moves the debugger to "ComActivator -> CreateInstance" method. From there I just click F10 until the issue comes up and the exception dialog pops up. I then have to click on the "Continue" button on the exception dialogue three times in order to get rid of the dialogue. Then I left mouse click with on the "Form1" surface, which causes the debugger finally to jump to "Control => InPlaceActivate" method. The debugger continues code execution in the finally block on the "comScope.Dispose()" line. This finally allows me to set a breakpoint inside the "InPlaceActivate" method, which allows me to debug the real issue with the "GetWindowContext" method. After doing this, I can get the debugger to break inside the method on every click on the form.

Like I said, complicated. Even after setting the breakpoint in the correct place, closing the VB6 application and then re-opening forces me to dance the same steps again to get the debugger to break in the correct position. I have no idea, why it works this way, I just assume it has to do with the fact I'm debugging against the release version of the 'preview' framework and somekind of interplay with the COM interop magic, which is happening in the background.

dotnet / winforms