Events get ignored by COM in VBA (Excel) when invoked inside a Task

[Spikxzy]

Description

COM events from a COM component created in .NET 6 are not received by a COM client written in VBA (Excel). However, these events are ignored only when they are invoked inside a 'System.Runtime.Action' class run by 'System.Threading.Tasks.Task' class.

Task.Run(() => SomeEvent?.Invoke)

The issue is that the same COM component in .NET Framework 4.7.2 would result in the events working as intended. I would expect that the events should run in .NET 6 just like they used to in .NET Framework 4.7.2.

The code needed to reproduce this issue can be found in this Github repository. It contains the .NET 6 version and the .NET Framework 4.7.2 version of the COM component as well as the COM client as an Excel workbook with macros.

To reproduce the issue just use the COM components as described in the ReadMe file of the repository.

Configuration

• The COM component is created with .NET 6
• OS: Microsoft Windows 10 Enterprise
• The architecture is x64, however the COM component is built in x86 since the local Excel's bit version is x86 and it requires the same bit version

Regression?

As mentioned above this was working in .NET Framework 4.7.2. However, it may be important to mention that .NET 6 imports the 'Action' and the 'Task' class from the assembly 'System.Threading.Tasks', while .NET Framework 4.7.2 imports those classes from the assembly mscorlib.

Other information

The events don't work, when Tasks are used to invoke them. Invoking the event without using a Task works for the .NET 6 Version of the COM component as well.

[chrfin]

@jeffschwMSFT Any update on this? I'm facing the same problem 🙁...

[AaronRobinsonMSFT]

TL;DR The issue is the implicit cross apartment operation that is occurring - operating on an STA object (VBA) from an MTA context (Threadpool thread). This is by-design. A workaround is to register a marshaller for the .NET event interface being used.

The crux of the problem is .NET Remoting is not supported in .NET Core and was being used to make this scenario work in .NET Framework. Speaking specifically about the repo, a QueryInterface() is taking place for the CalculatorEvents interface and is failing. It is failing because the runtime is creating a proxy to the VBA object instance (the callback) since it is being used from an MTA (that is, Threadpool thread via Task.Run()). The CalculatorEvents interface is unknown to COM and hence the QueryInterface() fails with E_NOINTERFACE.

See the call to SafeQueryInterface() below.

https://github.com/dotnet/runtime/blob/3c0c801056c285965097692bfb13a4a53afa2462/src/coreclr/vm/runtimecallablewrapper.cpp#L2050-L2094

[chrfin]

@AaronRobinsonMSFT Thanks for the explanation! Do you have an example of such a marshaller?

[AaronRobinsonMSFT]

@AaronRobinsonMSFT Thanks for the explanation! Do you have an example of such a marshaller?

COM will generate one for you if the type is known. Adding a definition of the interface in the IDL file is the easiest way to achieve that.

[Spikxzy]

@AaronRobinsonMSFT Thanks for the reply. Could you maybe explain what you mean by adding an interface definition? Because, isn't there already a definition of the interface given inside my example project? Do I need to add anything else to my IDL file? See: GitHub Repository with .NET6 Calculator

[AaronRobinsonMSFT]

@Spikxzy and @chrfin Apologies for missing what side of the COM boundary this was occurring on. I'd also forgotten how COM events are done and that makes this even more complicated. As mentioned above the issue is around marshalling and the built-in system is going to do that for COM Events using all the build in mechanics. We have some narrow tests for this, see here and here, but none deal with cross-apartment logic and that makes things more complicated. The simplest way to get around this is to perform the marshal directly and then call Invoke() on the underlying IDispatch.

public void TriggerAdditionEvent()
{
    Task.Run(() =>
    {
        foreach (Delegate d in OnAdditionDone.GetInvocationList())
        {
            // This is only needed for callbacks defined in COM.
            Debug.Assert(d.Target != null && Marshal.IsComObject(d.Target));

            // Performing the QI for IDispatch will marshal the interface into
            // this apartment.
            var inst = Marshal.GetIDispatchForObject(d.Target!);

            // Arguments to the event should be defined
            Guid empty = Guid.Empty;
            DISPPARAMS args = new();

            unsafe
            {
                // Get the IDispatch::Invoke() slot.
                var dispatchInvoke = (delegate* unmanaged[Stdcall]<IntPtr, int, Guid*, short, short, void*, void*, void*, void*, int>)(*(*(void***)inst + 6));

                // Call the Invoke().
                int hr = dispatchInvoke(inst, 1, &empty, 0, 1, &args, null, null, null);
                Debug.Assert(hr == 0);
            }

            // Release the marshalled instance.
            Marshal.Release(inst);
        }
    });
}

[AaronRobinsonMSFT]

@Spikxzy and @chrfin I see at least one thing that is missing from the fully integrated support. See ComEventInterfaceAttribute and example in our test suite:

https://github.com/dotnet/runtime/blob/c6fd07b93c5f68e4989ff607cd061f3af72b0b03/src/tests/Interop/COM/ServerContracts/Server.Events.cs#L54-L58

[Spikxzy]

@AaronRobinsonMSFT thanks for the help. The solution you posted with unmanaged code works for Excel. However, I started testing this solution with Matlab and Python too. For those the solution it doesn't work unfortunately. Therefore, I am asking if, you know what could be done to solve the issue for Matlab and Python as well.

[AaronRobinsonMSFT]

However, I started testing this solution with Matlab and Python too. For those the solution it doesn't work unfortunately.

@Spikxzy How is this scenario being used in MATLAB or Python? Are you trying to translate the above code into M or Python or do you mean when the above C# is running inside of the MATLAB process, the code doesn't work?

[Spikxzy]

However, I started testing this solution with Matlab and Python too. For those the solution it doesn't work unfortunately.

@Spikxzy How is this scenario being used in MATLAB or Python? Are you trying to translate the above code into M or Python or do you mean when the above C# is running inside of the MATLAB process, the code doesn't work?

@AaronRobinsonMSFT So basically I am testing this C# calculator over COM with several different COM clients (in Excel, Python and Matlab). All of them can create COM clients that can receive events. However, just like Excel used to, Matlab and Python do not receive the 'AdditionDone' event that is thrown by the C# COM object inside a task Therefore, it is the second case in which the C# code is running inside a Matlab process and Matlab for example will not receive the event thrown inside a C# task.

Here are the Python and Matlab codes, that try to access the COM object:

Python:

import` win32com.client
import pythoncom
import pywintypes
from win32com.client import Dispatch

unkownObject = pythoncom.CoCreateInstance(pywintypes.IID('ComCalculatorTest'), None, pythoncom.CLSCTX_ALL, pythoncom.IID_IUnknown)
dispatchEvents = Dispatch(unkownObject.QueryInterface(pythoncom.IID_IDispatch))

class CalculatorEvents:
    def OnAdditionDone(self):
        print('Addition done.')

EventListener = win32com.client.WithEvents(dispatchEvents, CalculatorEvents)

print(dispatchEvents.Addition(123, 456))

Matlab:

classdef COMCalculatorTest

    properties
        Calculator
    end

    methods
        function OnAdditionDone(varargin)
            disp('Addition done.');
        end

        function DoAddition(f)
            e = actxserver('ComCalculatorTest');
            f.Calculator = e;

            registerevent(f.Calculator, {('OnAdditionDone') @f.OnAdditionDone});
            sum = f.Calculator.invoke('Addition', 1, 2);
            disp(sum)
        end

    end

end

[Spikxzy]

@AaronRobinsonMSFT writing this comment, because you might not have gotten notified.

[AaronRobinsonMSFT]

@Spikxzy I did miss this, apologies. This looks like the same issue as in Excel. The issue is the cross-apartment marshalling logic. The Python and MATLAB code are going to need to do the same thing as Excel to work around this issue. I will try and make some time to figure out if there is another way to enable this.

[AaronRobinsonMSFT]

Appreciate the patience with me on this. I'm sorry I've been poorly engaged with this issue. I didn't pay all that much attention to the repo and that hasn't been fair to you and sadly all the tools were there to make this "just work". I alluded to the recommended fix in https://github.com/dotnet/runtime/issues/82628#issuecomment-1572985524 but didn't make the effort to prove it out. Let me correct that here.

The issue is indeed about threading models (STA vs MTA). COM Events are annoying and historically used for active scripting scenarios like VBScript, VB6 and VBA. This is where IDispatch (late-bound) is normally used. The point I am trying to make here is COM Events across apartments is uncommon in practice but .NET Framework made it too easy to not fully setup the environment and fallback with .NET Remoting. The official solution for COM is to perform the Type Library (TLB) registration using the RegisterTypeLib() Win32 API. Once this is done the events interface defined in the example (that is, dispinterface CalculatorEvents) will be known to COM and the standard marshallers will kick in when needed.

There is no standalone TLB registration tool anymore, at least not one I am aware of. The code to make this work in C++ is below and using a tool like ProcMon.exe, you can collect the registry keys needed and fold them into some setup tool chain. In order to register system wide, the registering process must be elevated.

// From developer command prompt run> cl.exe regtlb.cpp
#include <cassert>
#include <Windows.h>
#include <oleauto.h>
#pragma comment(lib, "OleAut32")
int main(int ac, char const** av)
{
    BSTR p = SysAllocString(LR"**(<PATH_TO_TLB>\COMCalculatorNet6Versionx64.comhost.tlb)**");
    assert(p != nullptr);
    ITypeLib* tlb{};
    HRESULT hr = LoadTypeLib(p, &tlb);
    if (SUCCEEDED(hr))
        hr = RegisterTypeLib(tlb, p, nullptr);
    if (tlb != nullptr)
        tlb->Release();
    SysFreeString(p);
    return EXIT_SUCCESS;
}

After the TLB is registered the triggering logic, as written below, should work. I've validated this in Excel, not Python or MATLAB.

public void TriggerAdditionEvent()
{
    Task.Run(() =>
    {
        OnAdditionDone?.Invoke();
    });
}

[Spikxzy]

@AaronRobinsonMSFT I have tried the solution you have suggested, however it seems that it only works for Excel. It doesn't seem to work for Python or Matlab. Maybe you still have an idea what could be done? I was also not able to do this in C++ since including the header file, containing RegisterTypeLib function (I assumed it is "oleauto.h"), leads to 200+ errors when trying to build it. I have used the C# equivalent instead.

internal class Program
    {
        static void Main()
        {
            string path = "PathToTLB";

            RegisterTypeLib(LoadTypeLib(path), path, string.Empty);
        }

        [DllImport("oleaut32.dll")]
        static extern int RegisterTypeLib(ITypeLib ptlib, [MarshalAs(UnmanagedType.BStr)] string szFullPath, [MarshalAs(UnmanagedType.BStr)] string szHelpDir);

        [DllImport("oleaut32.dll", PreserveSig = false)]
        public static extern ITypeLib LoadTypeLib([In, MarshalAs(UnmanagedType.LPWStr)] string typelib);
    }

Does this make a difference or does it work the same?

[AaronRobinsonMSFT]

I was also not able to do this in C++ since including the header file

I've updated the C++ example to be complete.

I have used the C# equivalent instead.

That is broadly the same thing. A few nits, but shouldn't impact the actual scenario.

It doesn't seem to work for Python or Matlab. Maybe you still have an idea what could be done?

I've no idea at this point. We would need to debug the Python or MATLAB scenario to understand where this is failing. I don't have access to MATLAB. Is there a simple repro I can set up for Python?

[Spikxzy]

@AaronRobinsonMSFT The following Python code can be used to check whether or not it works:

import win32com.client
import pythoncom
import pywintypes
from win32com.client import Dispatch

unkownObject = pythoncom.CoCreateInstance(pywintypes.IID('ComCalculatorTest'), None, pythoncom.CLSCTX_ALL, pythoncom.IID_IUnknown)
dispatchEvents = Dispatch(unkownObject.QueryInterface(pythoncom.IID_IDispatch))

class CalculatorEvents:
    def OnAdditionDone(self):
        print('Addition done.')

EventListener = win32com.client.WithEvents(dispatchEvents, CalculatorEvents)

print(dispatchEvents.Addition(123, 456))

Before the solution of the addition is printed the event text "Addition done." should be printed but isn't.

[AaronRobinsonMSFT]

Before the solution of the addition is printed the event text "Addition done." should be printed but isn't.

If the managed COM server is still exectuing the event on a new Task (that is, Task.Run()), that isn't true. There is no telling when the Task will be scheduled and it could be quite some time after the result is printed.

Also, apartment issues are going to be difficult with this scripting since the main thread could be blocked performing some other action. For Python and perhaps MATLAB, messages will need to be pumped. There appears to be an API for that in pywin32 - PumpWaitingMessages. I've no idea how MATLAB handles message pumping from the main thread.

// Synchronous and the event will be fired prior to the result being printed.
public void TriggerAdditionEvent()
{
    OnAdditionDone?.Invoke();
}

// Asynchronous and the event can be fired prior to or after the result is printed.
// This approach will also require the main thread to pump messages so the python
// event handler can be executed in the correct apartment (STA).
public void TriggerAdditionEvent()
{
    Task.Run(() =>
    {
        OnAdditionDone?.Invoke();
    });
}

[AaronRobinsonMSFT]

I was able to have the event fired for both scenarios using the following script.

import win32com.client
import pythoncom
import pywintypes
from win32com.client import Dispatch

unkownObject = pythoncom.CoCreateInstance(pywintypes.IID('ComCalculatorTest'), None, pythoncom.CLSCTX_ALL, pythoncom.IID_IUnknown)
dispatchEvents = Dispatch(unkownObject.QueryInterface(pythoncom.IID_IDispatch))

class CalculatorEvents:
    def OnAdditionDone(self):
        print('Addition done.')

EventListener = win32com.client.WithEvents(dispatchEvents, CalculatorEvents)

print(dispatchEvents.Addition(123, 456))

# Comment out if the event is running from a .NET Task.
#while 1:
#    pythoncom.PumpWaitingMessages()