Open jkoritzinsky opened 2 years ago
Tagging subscribers to this area: @dotnet/interop-contrib See info in area-owners.md if you want to be subscribed.
Author: | jkoritzinsky |
---|---|
Assignees: | - |
Labels: | `Epic`, `area-System.Runtime.InteropServices`, `Bottom Up Work` |
Milestone: | - |
I'm currently writing a managed library that wraps the COM interfaces that CoreCLR supports for diagnostics (clrdata
, cordebug
, corprof
) so I have some questions:
ComObject
to hold pointers to every potential COM interface supported by the project's ComWrappers
type. This is probably fine for small-to-medium sized COM interop scenarios, but in my case, I'm looking at some 400+ interfaces. I'd rather not have ComObject
s that huge. The strategy I'm currently using is to have a cache of QI'd pointers sitting on my ComObject
equivalent, which I lazily populate on casts. (I think this is kind of similar to what CsWinRT is doing.) Could this strategy, or something like it, be supported?HRESULT
exception translation, ref
/out
/in
modifiers, passing an out
parameter as return value where sensible, etc?I'm currently writing a managed library that wraps the COM interfaces that CoreCLR supports for diagnostics (
clrdata
,cordebug
,corprof
) so I have some questions:
- I see the idea is for the generated
ComObject
to hold pointers to every potential COM interface supported by the project'sComWrappers
type. This is probably fine for small-to-medium sized COM interop scenarios, but in my case, I'm looking at some 400+ interfaces. I'd rather not haveComObject
s that huge. The strategy I'm currently using is to have a cache of QI'd pointers sitting on myComObject
equivalent, which I lazily populate on casts. (I think this is kind of similar to what CsWinRT is doing.) Could this strategy, or something like it, be supported?
We've considered such a strategy, but we want to get some performance numbers before we decide to go one direction or the other (or a blend of both).
- With C# interfaces being the source of truth, how much support would there be for high-level constructs like spans instead of pointer/length pairs,
HRESULT
exception translation,ref
/out
/in
modifiers, passing anout
parameter as return value where sensible, etc?
Yes, we plan to support the same features we support with LibraryImportGenerator
plus a small number of COM-specific ones like HRESULT
exception translation.
- Is checkpoint 1 likely to be done in the .NET 7 time frame? I'm currently writing a project-specific generator that handles just enough for my needs, but even that is a fair amount of work. If it's likely to be wasted effort in a matter of months anyway, perhaps I should just put my project on ice until the official generator is available.
We're hoping to get checkpoint 1 done in the .NET 7 time frame, but I don't want to make any promises.
Yes, we plan to support the same features we support with
LibraryImportGenerator
plus a small number of COM-specific ones likeHRESULT
exception translation.
How would this look in practice, by the way? E.g. if I define
[GeneratedComInterface(typeof(MyComWrappers))]
[Guid("4b69d271-5c99-4f95-b1eb-381e6e689f1a")]
partial interface IMyComInterface
{
void Foo();
}
and Foo
actually has a HRESULT
return on the native side? Would I need to attach [return: MarshalAs(UnmanagedType.Error)]
to communicate that to the generator?
I'm not familiar with the exact set of constructs supported by LibraryImportGenerator
. Does it include spans (where the native side expects pointer and length in separate parameters)?
We're hoping to get checkpoint 1 done in the .NET 7 time frame, but I don't want to make any promises.
That's fine, I was just wondering what the plan was, with the understanding that plans can change.
Yes, we plan to support the same features we support with
LibraryImportGenerator
plus a small number of COM-specific ones likeHRESULT
exception translation.How would this look in practice, by the way? E.g. if I define
[GeneratedComInterface(typeof(MyComWrappers))] [Guid("4b69d271-5c99-4f95-b1eb-381e6e689f1a")] partial interface IMyComInterface { void Foo(); }
and
Foo
actually has aHRESULT
return on the native side? Would I need to attach[return: MarshalAs(UnmanagedType.Error)]
to communicate that to the generator?
This is still under discussion. The current prevailing idea is to match the built-in system where we do HRESULT checking by default unless the user provides the [PreserveSig]
attribute on the method.
I'm not familiar with the exact set of constructs supported by
LibraryImportGenerator
. Does it include spans (where the native side expects pointer and length in separate parameters)?
We have some custom marshallers for Span<T>
that marshal them to a pointer, and the length still needs to be passed as a separate parameter. These custom marshallers still need to go through API review.
Additional ideas (like synthesizing a length parameter into the native signature) have not been proposed or implemented yet. If you're interested in that feature, I think it's worthwhile filing a separate issue for the team to discuss it.
We're hoping to get checkpoint 1 done in the .NET 7 time frame, but I don't want to make any promises.
That's fine, I was just wondering what the plan was, with the understanding that plans can change.
:+1:
Is there an ETA for the ComWrappers source generator? (say's 'future' in the project?)
We are working on a prototype now that might be used for some internal parts of the dotnet/runtime repo. The plan would be to iterate early .NET 8 and hopefully have something in one of the later .NET 8 previews for public consumption.
I'd appreciate if this was usable for creating an out-of-process COM server.
I have a question about the Source Generator COM design doc, and the intended scope of the COM wrappers for IDispatch
interfaces when integrating with C#.
In the design doc under Checkpoint 4: IDispatch compatibility is stated (my emphasis):
We do not plan on supporting IDispatch integration with C# dynamic, at least for the first release of the COM source generator. Although the built-in system currently supports it, the integration is primarily used with the PIAs provided for Office, which we do not plan on regenerating with this tooling.
This is the scenario I'm concerned with - creating NativeAOT outputs (add-in libraries or standalone programs) that call Office COM APIs.
The problem with a lack of IDispatch
/ dynamic
integration is not about regenerating the Office PIA assemblies with new tooling, but in consuming them in the same way they are currently used, particularly with the enhanced 'embedded PIA' C# support. This is similar in motivation to an earlier discussion about the need for IDispatch
/ dynamic
support which was then implemented for .NET 5. Is this a limitation of the source generator, the COM Wrappers code or the dynamic
runtime support infrastructure?
Even when using a pre-compiled PIA, there are common code patterns that require the IDispatch
/ dynamic
support in C# to work as expected. For example, will we be able to make this code work in a NativeAOT setting?
using Microsoft.Office.Interop.Excel;
class Program
{
static void Main(string[] args)
{
Application app = new Application();
app.Visible = true;
Workbook wb = app.Workbooks.Add();
wb.Sheets[1].Name = "FirstSheet";
// ^^^^ The code in the PIA for the indexed property wb.Sheets[...] returns "object"
// Under current compiler, with PIA embedding, this compiles and runs as 'dynamic'
// since that kicks in the dynamic / IDispatch support.
}
}
The one part I can't see yet (and have not tried myself) is how to fill in, by source generation or extra wrapper code, a way to make calls like the above work when changing to NativeAOT. There are some related quirks around the PIA code and how it is called from C# - for example related to indexed properties (which appear in PIA code and can be called from C# for embedded PIA scenarios but can't be authored in normal (or source generated) C# code).
Due to the .NET versioning limitations for in-process add-ins, we will be stuck on .NET 6 (together with .NET Framework) for making Office add-ins until we have a viable NativeAOT approach. From the discussion around COM Wrappers, and the source generators here, it looks like we will have a comprehensive approach for exposing COM objects from NativeAOT libraries (perhaps with third-party tools like dscom for type library generation). Hence, I want to campaign for a higher priority also for supporting existing C# code that calls COM libraries, unlocked by some plan for IDispatch
/ dynamic
support.
Speaking of dscom, there is currently the issue, that there is no support for Serializable and Exception, which introduces a strange behavior at runtime even if a COM serializable Exception is thrown.
This is due to the fact, that the exception types and serializable implementations in the runtime are not exposed to COM as they were in .NET FullFramework.
We have completed the features we plan to complete in the .NET 8 timeframe. Moving this Epic to .NET 9 to track any future work we do in this space.
@AaronRobinsonMSFT @jkoritzinsky Could you please help me understand the intended scope of the COM / AOT support in .NET 8 and in near future versions relating to the Office applications (particularly Excel). If I intend for an AOT compiled library to talk to Excel via COM, my current understanding is that
If we want to do Office interop with .NET 8 AOT, how do you recommend we approach this?
@AaronRobinsonMSFT @jkoritzinsky, we're trying to build wrappers around Excel C/COM APIs (example project) it would be very helpful to shed some light on @govert's questions. Thanks in advance!
- the ComWrappers helpers provide the only mechanism for interacting with a COM object model when AOT compiled.
- the ComWrappers helpers themselves are rich enough to implement marshalling for all COM scenarios, in both directions - consuming COM object via RCW, and exposing CCW from an AOT .NET library
Correct.
- there is no support for C# 'dynamic' types under AOT, so even with a manual implementation using ComWrappers, one could not implement the dynamic-style IDispatch integration introduced with the PIA embedding work in .NET 4.0. I.e. there is no way to make a "MyComDispatch" type that will be able to implement the PIA embedding style code. Or is the IDispatch / dynamic limitation mentioned here just relating to the source generator?
There is no built-in support, but one could imagine an AOT scenario where all the known IDispatch
based interfaces were known and retain them. This isn't something the .NET team is keen to support at present and we are deferring to the community and up-stack teams to add the appropriate marshallers. For example, WinForms will need some support here, but they are in the process of determining how they want to express that. The .NET team is looking at providing a in-box VARIANT
marshaller. Something like SAFEARRAY
or IRecordInfo
though is going to be much harder to justify from the runtime side. Both are possible, but we need a substantial business case to take on that effort. We are very open to community proposed marshallers though, feel free to propose needed APIs.
- for simple COM interface models, the COM source generator could take type definitions that look like those inside a PIA interop assembly, and generate ComWrapper-based marshaling code under .NET 8
Yes.
- for complicated COM interface models, including those that are IDispatch-rich and with COM Events (like the Office COM models) the .NET 8 generator will provide no or minimal help
Yes.
- to support the Excel COM object model with .NET 8, we would need to write our own ComWrapper based wrapper library, either by hand or with our own PIA -> ComWrappers code generation implementation
The interfaces would currently need to be redefined using GeneratedComInterface
. If these are IDispatch
based, a definition of IDispatch
would also need to be defined.
- future versions of .NET are likely to improve the COM source generator scope
Yes, but narrowly. Support for IDispatch
in-box will likely be a very low priority for the .NET interop team. We want to provide the tools we can, but providing a full solution is incredibly expensive and Office's focus on .NET has shifted considerably so has become lower priority.
If we want to do Office interop with .NET 8 AOT, how do you recommend we approach this?
Redefine the needed interfaces using GeneratedComInterface
and start implementing them.
@AaronRobinsonMSFT Thank you very much for the reply and setting our expectations for the current COM / AOT support.
For the COM objects we need to expose as CCWs to Excel (e.g. to support Ribbon extensions or real-time data sources) we would need to implement the IDispatch
support, but only for classes known at compile time and relatively friendly signatures. Consuming the Office object models, including events, seems more daunting, especially if we want to end up in the post-.NET 4.0 world where COM worked easily from C# due to the improved compiler integration.
Could you please clarify the status of the C# 'dynamic' infrastructure under Native AOT? Expression trees seem to work under AOT. However, adding simple 'dynamic' expressions compiles to a larger binary, but then crashes when called. Is this a trimming problem, or totally unsupported? (I do recall some post-RC2 issue that might be related but can't find it again.)
I'm trying to understand whether one might revisit the work from https://github.com/dotnet/runtime/pull/33060 (which implements the COM / dynamic binder support), but replace the built-in COM features with ComWrappers. It might help to focus on the (hopefully) simpler cases that the Office COM model requires. But this would still need to the C# dynamic binder to work under AOT against the user-code implemented 'RCW's.
For reference (please redirect if this is off-topic), here's a snippet where runtime expression trees work, but dynamic
fails:
using System.Linq.Expressions;
Console.WriteLine("Hello, World!");
// This works under NativeAOT
var lambda2 = Expression.Lambda(
Expression.Add(
Expression.Constant(1.3, typeof(double)),
Expression.Constant(2.7, typeof(double))
)
);
Func<double> getResult = (Func<double>)lambda2.Compile();
Console.WriteLine(getResult());
// This crashes under NativeAOT
dynamic d1 = 1.3;
dynamic d2 = 2.7;
double d = (double)(d1 + d2); // <<<<< NRE or access violation here
Console.WriteLine(d);
// This crashes under NativeAOT
dynamic d1 = 1.3;
dynamic d2 = 2.7;
double d = (double)(d1 + d2); // <<<<< NRE or access violation here
Console.WriteLine(d);
dynamic
does not work with trimming and AOT. You should see warnings like this when building this snippet.
C:\repro\Program.cs(4): Trim analysis warning IL2026: Program.<Main>$(String[]): Using member 'Microsoft.CSharp.Runtime
Binder.Binder.Convert(CSharpBinderFlags,Type,Type)' which has 'RequiresUnreferencedCodeAttribute' can break functionali
ty when trimming application code. Using dynamic types might cause types or members to be removed by trimmer. [C:\repro
\repro.csproj]
C:\repro\Program.cs(4): Trim analysis warning IL2026: Program.<Main>$(String[]): Using member 'Microsoft.CSharp.Runtime
Binder.Binder.BinaryOperation(CSharpBinderFlags,ExpressionType,Type,IEnumerable`1<CSharpArgumentInfo>)' which has 'Requ
iresUnreferencedCodeAttribute' can break functionality when trimming application code. Using dynamic types might cause
types or members to be removed by trimmer. [C:\repro\repro.csproj]
@jkotas Indeed I do get such warnings, both for the dynamic
code that fails at runtime, and also for the expression trees that work (in this case). Evidently the add method we're trying to invoke is not trimmed, nor is (all of) the binder infrastructure.
I suppose my question is whether dynamic
is completely unsupported under AOT, or could one somehow give trimming hints to make it work? In our COM case we might know what binders and related code would be needed, so could generate hints if there is a way.
You can make it work by providing trimming and AOT hints, but it is very error prone. We do not recommend it.
To confuse things more, the dynamic
code runs fine with 'full' trimming, single file, but no AOT, using these settings:
<PublishAot>false</PublishAot>
<PublishSingleFile>true</PublishSingleFile>
<PublishTrimmed>true</PublishTrimmed>
<SelfContained>true</SelfContained>
<TrimMode>full</TrimMode>
<DynamicCodeSupport>false</DynamicCodeSupport>
<EventSourceSupport>false</EventSourceSupport>
Should we expect code to fail with AOT when it is working with this configuration?
Trim warnings indicate that some code may not work as expected after trimming. The trimmed app can work today and start failing after next .NET SDK update. It can work with JIT and fail with native AOT. This unpredictable behavior is by-design.
Hi! I'd like to add to this list of work items the need to properly support ActiveX interop bindings - the ones generated by aximp.exe on top of the COM interop bindings generated by tlbimp.exe in current .NET build tooling.
We need this in Remote Desktop Manager for the Microsoft RDP ActiveX control on Windows, the only exhaustive interface for third-party integration of the first-party RDP client. Even if we integrate FreeRDP, a project which I have created years ago, it will likely never reach 100% feature parity with the original, so customers really want the option to use official RDP client. For this reason, continued support for the RDP ActiveX control in .NET is business critical to us, as RDP is the most important protocol we support in our product.
This being said, we still ship .NET with regular JIT assemblies, so we're not in a rush to try Native AOT. However, the Remote Desktop Manager application startup time is an annoyance we've had since forever, so I would be lying if I didn't at least have my eye on experimenting with Native AOT for Remote Desktop Manager at some point in the future. I'm just looking at what would be blocking it, and lack of RDP ActiveX support would be a non-starter for us for the reasons listed above.
Enough of the business aspect of things: when it comes to the RDP ActiveX, I maintain MsRdpEx, a project which extends the original RDP ActiveX with additional features our customers need, but Microsoft doesn't provide. The RDP ActiveX is probably the most widely used ActiveX component in .NET today, and you'll find AxMSTSCLib.dll, MSTSCLib.dll in a lot of Windows .NET applications that use RDP.
We ship a code signed nuget package with everything needed to load and use the Microsoft RDP ActiveX component from .NET, which includes the ability for manual COM activation of the ActiveX DLL by path. COM Activation is mentioned earlier - we need manual COM Active by DLL path to load rdclientax.dll from MSRDC, which contains an unregistered RDP ActiveX interface compatible with the one from mstscax.dll. Even then, being able to load COM interfaces without registration is a huge plus, and in this case, it allows us to load both mstscax.dll and rdclientax.dll on a per-session basis inside Remote Desktop Manager, based on connection preferences.
As for the build tooling, I took notes on how to regenerate everything from the type library for both C# and C/C++. This should come in handy when comparing the old tooling with the new for experimentation. While aximp.exe has a way to generate source code, tlbimp.exe only generates an assembly straight as MSIL, which unfortunately doesn't decompile to clean C# code with tools like ILSpy. This currently limits the ability to control how the assembly is built, and it has annoying things like a hardcoded reference to net40 as the target framework. It doesn't break the build, but it does cause build warnings, even if the assembly is in fact compatible with modern .NET.
As for the ActiveX interop method, one can do dynamic calls through IDispatch, or call the vtable functions directly. In the case of the RDP ActiveX, I am fairly certainly that the calls are always done through vtable with the generated interop bindings. I don't know of an ActiveX component that would not support the vtable calls, and even then, it would still be possible to generate stubs wrapping IDispatch if it's ever needed. However, for our needs, we are more than happy to use the vtable calls without the runtime calling through IDispatch, it's much faster anyway.
Last but not least: the ActiveX bindings are meant to be consumed by WinForms. We're WinForms users, and we'd like to keep using WinForms in the future. I don't mind if the new build tooling can generate ActiveX bindings for other .NET UI frameworks, but it would be important to coordinate with the WinForms team to ensure compatibility of the newer bindings.
One last thing: when adding ActiveX components in WinForms from Visual Studio, it generates a .resx with a serialized BinaryFormatter OcxState object which is then loaded in the designer-generated source code. Needless to say that this causes deprecation warnings in .NET 8 as this kind of object serialization is scheduled to be entirely removed in .NET 9. I manually deleted those resources and set OcxState to null in the designer-generated source code, it didn't seem to cause problems. However, someone from the Visual Studio team should definitely look into fixing this issue in the WinForms designer. I have no idea what this OcxState object is used for, and how to create it dynamically instead of deserializing it from a resource.
I hope this clarifies the need for modernized ActiveX support in .NET! Feel free to reach out for any assistance, I'll be happy to help.
I have no idea what this OcxState object is used for, and how to create it dynamically instead of deserializing it from a resource.
Its serialized state of the ActiveX control, basically a binary blob for the control to store whatever designer state it wants to preserve to runtime thats not configurable through control properties. More complex ActiveX controls may have custom UI to configure things instead of just using properties. If you don't provide one when instantiating it the control implicitely uses default state, which should work fine as long as you didn't configure anything in the designer of the ActiveX control that affected its internal state.
It should be possible for the new designer to just store it as a byte array or something along those lines.
@awakecoding I took a look at the TLB through the oleview tool (shows the IDL source for a TLB, is part of the Windows SDK installed by VS) and it doesn't look very complicated, should be possible to write the interop manually for me. Is there an open source example for Desktop Framework using that ActiveX control? I'd consider doing a proof of concept port if you want to see how well it works in .NET 8 - I've been working with the WinForms project in the past to improve interop support and having another ActiveX control thats used in practice could help interop quality, we've been struggling to find relevant usecases to test the ActiveX support against last I worked with them.
PS: feel free to take the conversation about a PoC to somewhere else on git (if you have a project), or to mail (see my git profile), to not get offtopic on this one
@weltkante I have a sample .NET application just to test the .NET interop, it's nothing fancy but it's there in the MsRdpEx repository. I have build instructions for MsRdpEx, it's not too difficult to get started, and I'd be happy to assist with experiments on that repository if you want to take the discussion there. I would welcome an experimentation branch from you if you have an idea how to try it! You can also reach me on Twitter/X.
It is a bit disappointing that properties aren't supported at all by COM source generators. This makes it impossible to use generated interop as a (mostly) source-compatible drop-in replacement. It would definitely be desireable to consider property support for the future, maybe annotate the property methods with an attribute if you need it to define VTable position. I saw something about a VirtualMethodIndexAttribute
but it doesn't seem to be available in .NET 8 so I suppose its a .NET 9 thing, or am I missing something? Maybe that kind of attribute could be used for property support if its not possible to automatically order the setter/getter in order of source as the old interop did? (Whether you write { set; get; }
or { get; set; }
is significant for the old interop since it generates methods in different vtable order.)
It is a bit disappointing that properties aren't supported at all by COM source generators.
All part of a V1 release. I think it is worth having the conversation in .NET 9. I don't think we have an issue for property support, @jkoritzinsky?
I don't think we have one yet. I'll open one today.
Edit: Opened at https://github.com/dotnet/runtime/issues/96502
We plan to provide a COM source generator to enable developers to interoperate with unmanaged COM interfaces without using the built-in COM Interop subsystem as per our Source Generator COM design.
This issue tracks the various tasks that implementing this source generator will require. Not all of these tasks will be completed in .NET 7 (in fact this list intentionally includes significantly more work than we will do in .NET 7 to provide a roadmap over a few versions).
Checkpoint 1: MCG-replacement
ComObject
type that implements any required infrastructure to provide implementations of a COM interface.IDynamicInterfaceCastable
-related)ComObject
instance that wraps a nativeIUnknown*
through a user-definedpartial
ComWrappers
-derived type.ComObject
using the above functionality.IUnknown*
that represents a managed non-ComObject
object through a user-definedpartial
ComWrappers
-derived type.IUnknown*
wrapper using the above functionalityCheckpoint 2: WinForms compatibility
WinForms is primarily
IUnknown
-based COM, but there is some usage ofIDispatch
-based COM, primarily in the accessibility space. We would like to help support making WinForms trim-friendly, so we want to provide these features in the generator to support them.BSTR
strings. https://github.com/dotnet/runtime/pull/69213VARIANT
s without typelib related support (no user-defined record types): #93635SAFEARRAY
sIDispatch
, or provide guidance for implementing stubbed-out support.IDispatch
implementation can be stubbed out withE_NOTIMPL
errors: https://docs.microsoft.com/en-us/windows/win32/winauto/dual-interfaces--iaccessible-and-idispatchCheckpoint 3: Activation support
For parity with the built-in system, providing an easy mechanism for activating a COM object is desireable, if lower priority.
EnableComHosting
switch in the SDK for built-in COM.Checkpoint 4: IDispatch support
If we find enough demand, we should consider providing more extensive
IDispatch
support (excluding TLB-related dependencies)IDispatch
-based interfaces with well-definedDispId
valuesIDispatch
-based interfaces with generator-definedDispId
valuesdynamic
compatibility forIDispatch
-based COM object wrappers.Checkpoint 5: TLB support
VARIANT
orIDispatch
-related scenarios.Features to consider