Dynamic CoreCLR embedding?

[LegionMammal978]

I have a .NET 8 assembly written by a third party, filled with objects and methods that I want to access from native code on Linux. I've been interested in using the CoreCLR hosting APIs for this purpose. However, doing this in the style of dotnet/docs#18174 still doesn't seem to be possible outside the built-in Windows-only COM implementation. The current suggestion is to use ComWrappers, but its API is primarily geared toward being AOT-friendly, and I can't find any solid implementation that just lowers straight to reflection. In particular, as far as I can tell, the newer ComWrappers source generation creates only static APIs, and needs the assembly to be filled with special annotations at compile time.

The alternative here is to write my own assembly as a shim to access the third-party assembly, using reflection or source generation or whatever other means. But I've been attempting to avoid that, to simplify building and distributing the code. (And if I can't avoid it, then ComWrappers are redundant in the first place, since I can just set up a few reflection-based unmanaged entry points. Performance across this API boundary is not a particular concern to me.) Are there no plans to create such dynamic hosting tools, that wouldn't be AOT-friendly like ComWrappers are?

[janvorli]

The current .NET hosting API doesn't use COM and it is cross platform. The document you've mentioned above has a link to it: https://github.com/dotnet/docs/blob/main/docs/core/tutorials/netcore-hosting.md.

[huoyaoyuan]

The request looks similar to #109486, calling (third-party) instance methods from native hosting.

[AaronRobinsonMSFT]

@LegionMammal978 The interop team recommends using DNNE if you'd like to create a native projection for a .NET type instance. A basic example can be found at https://github.com/AaronRobinsonMSFT/DNNE/blob/master/test/ExportingAssembly/InstanceExports.cs. DNNE is a relatively simple library that handles all the hosting for you and does very basic code generation. ComWrappers is a fine choice if you'd like to adopt the COM model, but it will require you to create and respect the IUnknown contract in C++. This can be complicated on non-Windows platforms, but it is possible.

The current suggestion is to use ComWrappers, but its API is primarily geared toward being AOT-friendly, and I can't find any solid implementation that just lowers straight to reflection.

This isn't accurate. The ComWrappers API is about providing a source generated solution that reduces runtime overhead. A consequence of that is being AOT friendly, but that isn't a requirement nor should it be considered the primary use case.

There is a sample involving IDispatch, IUnknown based interface, on Windows with ComWrappers here. I do not recommend this approach on non-Windows unless one has experience with IUnknown on non-Windows and the project in question is already using IUnknown in some manner.

[LegionMammal978]

@LegionMammal978 The interop team recommends using DNNE if you'd like to create a native projection for a .NET type instance. A basic example can be found at https://github.com/AaronRobinsonMSFT/DNNE/blob/master/test/ExportingAssembly/InstanceExports.cs. DNNE is a relatively simple library that handles all the hosting for you and does very basic code generation. ComWrappers is a fine choice if you'd like to adopt the COM model, but it will require you to create and respect the IUnknown contract in C++. This can be complicated on non-Windows platforms, but it is possible.

The current suggestion is to use ComWrappers, but its API is primarily geared toward being AOT-friendly, and I can't find any solid implementation that just lowers straight to reflection.

This isn't accurate. The ComWrappers API is about providing a source generated solution that reduces runtime overhead. A consequence of that is being AOT friendly, but that isn't a requirement nor should it be considered the primary use case.

Thank you for the clarifications. Though what I was really trying to do here is avoid having a managed wrapper assembly that needs its own particular source generation and whatnot. I'm especially trying to avoid modifying the original library and its build process with special annotations, which DNNE seems to require with [DNNE.Export]. I don't particularly mind if this creates some extra legwork on the unmanaged side (e.g., setting up hostfxr properly), as long as I don't need to repeat that work for every type and method in the library. The goal is to have something approaching a subset of the Mono embedding API (at least handling objects and method calls), from the perspective of the host program.

There is a sample involving IDispatch, IUnknown based interface, on Windows with ComWrappers here. I do not recommend this approach on non-Windows unless one has experience with IUnknown on non-Windows and the project in question is already using IUnknown in some manner.

Thank you for that link, I must have missed it while searching. At worst, I could just stick a copy of that into the host program and load it with the hdt_load_in_memory_assembly delegate, which is ugly but workable. I've mainly been looking into IUnknown/IDispatch based solutions since ComWrappers seems to be the only supported way to interact with managed object instances from unmanaged code, without having to build a specialized source-generated managed wrapper. E.g., ComWrappers was listed as the only approach to dotnet/docs#18174 apart from writing your own unmanaged entry points. (Then again, I guess another alternative would be a manual reflection-based wrapper with unmanaged entry points, but ComWrappers looked promising as a way to avoid some of the wrapping and unwrapping on the managed side.)

Also, what are you referring to with IUnknown being difficult on non-Windows? Are you just talking about matching the vtable layout and HRESULT definitions, without the headers to help? It wouldn't be my first time doing that, I've had some practice using COM APIs from Rust without the benefit of source generation. Or are there particular differences between Windows' COM ABI and ComWrappers' pseudo-COM ABI to watch out for?

The request looks similar to #109486, calling (third-party) instance methods from native hosting.

Yes, that one does look similar. I'm mainly fishing around for alternatives to writing my own wrappers on the managed side, even if it comes at the cost of performance or terseness.

[AaronRobinsonMSFT]

Though what I was really trying to do here is avoid having a managed wrapper assembly that needs its own particular source generation and whatnot. I'm especially trying to avoid modifying the original library and its build process with special annotations, which DNNE seems to require with [DNNE.Export].

Even with a Reflection approch you will generally need this in .NET Core. Unless all the method signatures you are using are built-in and instantiated. DNNE does the absolute minimum needed and what you've just described means you can either (a) take or rewrite the platform.c file in DNNE and then create a few Delegate types to query into or place UnmanagedCallersOnly on it, the primary DNNE use case is through that - see any of the samples or doc.

The goal is to have something approaching a subset of the Mono embedding API (at least handling objects and method calls), from the perspective of the host program.

The Mono embedding API wasn't adopted by CoreCLR because it made more sense for people to use tools like DNNE or their own bespoke solution to write the embedding APIs they need rather than the runtime expose a large API surface that is chatty. Writing your own API and then exporting it via DNNE helps people "get it right" the first time. A matter of preference of course, but from the runtime side the embedding API is a huge cost to own and maintain.

Also, what are you referring to with IUnknown being difficult on non-Windows? Are you just talking about matching the vtable layout and HRESULT definitions, without the headers to help?

Yes. All of the COM goo that is provided by Windows headers. I wrote DNCP to help with this when .NET is involved.

It wouldn't be my first time doing that, I've had some practice using COM APIs from Rust without the benefit of source generation. Or are there particular differences between Windows' COM ABI and ComWrappers' pseudo-COM ABI to watch out for?

If you already have experience then go for it. There are loads of COM ABI issues because of Windows rules and how people often define interfaces. We created CallConvInstanceMember, specifically for Windows idioms and COM. Even with COM, you will still need some kind of native export via DNNE or another exported API to "activate" the instance.

If you're going down the COM route, I highly recommend using the COM source generator. It will help you get the interop correct.

[dotnet-policy-service[bot]]

This issue has been marked needs-author-action and may be missing some important information.

[LegionMammal978]

Thank you for the information.