search

Azure / azure-cosmos-dotnet-v2

Contains samples and utilities relating to the Azure Cosmos DB .NET SDK
MIT License
579 stars 837 forks source link

TypeLoadException with .NET Native compiled UWP app #625

Open Ricciolo opened 5 years ago

Ricciolo commented 5 years ago

You get some warnings when you build Microsoft.Azure.DocumentDB.Core NuGet package in a UWP project with .NET Native toolchain compilation enabled.

To Reproduce

Expected behavior Document shoud be read and no warnings should be listed.

Actual behavior I get this warning

MCG : warning MCG0007: Unresolved P/Invoke method 'Microsoft.Azure.Documents.ServiceInterop.dll!GetPartitionKeyRangesFromQuery' for method 'System.UInt32 Microsoft.Azure.Documents.ServiceInteropWrapper.GetPartitionKeyRangesFromQuery(System.IntPtr, System.String, System.Boolean, System.Boolean, System.String[], System.UInt32[], System.UInt32, Microsoft.Azure.Documents.PartitionKind, System.IntPtr, System.UInt32, System.UInt32)'.

Calling this method would throw exception at runtime. Please make sure the P/Invoke either points to a Windows API allowed in UWP applications, or a native DLL that is part of the package. If for some reason your P/Invoke does not satisfy those requirements, please use [DllImport(ExactSpelling=true) to indicate that you understand the implications of using non-UWP APIs.

The same warning occurs also for:

At runtime then I receive a TypeLoadException, as expected by the warning.

System.TypeLoadException: 'Unresolved P/Invoke method 'ntdll!RtlGetVersion' from this method. Please look for this method in build warnings for more details. '

Environment summary SDK Version: Windows 10 SDK (10.0.17134.0) OS Version: Windows 10 Pro (1803)

Additional context Add any other context about the problem here (for example, complete stack traces or logs).

ausfeldt commented 5 years ago

@Ricciolo In project file try adding false or the project properties -> Build -> uncheck "Compile with .NET Native tool chain". Also Microsoft.Azure.Documents.ServiceInterop.dll is for Windows x64 specific optimizations, it will not work on Windows on ARM64. Currently is not checking for ARM in the graceful fallback logic.

Ricciolo commented 5 years ago

@ausfeldt of course, if I disable .NET Native compilation I have no issue, but obviously I need it :-) I'm compiling for x86 or x64 and not for ARM64.

ausfeldt commented 5 years ago

@Ricciolo Added to the backlog for triage.

mnikonov commented 5 years ago

I'm having same issue. MS Store not submitting UWP applications with disabled "Compile with .NET Native tool chain". Also need at least compile application in x86 and x64. And, be honest, I dont know what to do right now, application not passing Store Submission Verification because of this issues and I have dead line for a project

MattWhilden commented 5 years ago

I work on the .NET Native team so I can shed some light on this that may help get to a faster resolution.

The issue is that Microsoft.Azure.Documents.NativeMethods.Windows.RtlGetVersion() relies on being able to call ntdll!RtlGetVersion which isn't available in the UWP API surface area. It's actually somewhat surprising that is works in DEBUG! I haven't tracked down why the debug runtime (CoreCLR) is more permissive but that's somewhat interesting.

As you can imagine, the ahead of time compiler (.NET Native) is much more particular about knowing where imports come from at compile time. In fact, it's the interop code generating portion of the compiler (MCG) that is warning that it cannot find various imports. We're somewhat permissive here because if you never end up calling those P/Invokes at runtime, you'll never have an issue. However, this particular error happens on the OpenConnection path so is unavoidable.

To get this to work properly in UWP we'd need to get an update to Microsoft.Azure.DocumentDB.Core that uses another method to get the version information. A similar PR happened in the CoreFX codebase and was tracked by this issue: https://github.com/dotnet/corefx/issues/21024.

It's probably worth investigating the other MCG warnings to see what else may be an issue. A lot of the warnings are because a native dll (DocumentDB.LogStoreViewer) seems to not have been included at all. Here's one such warning: MSBUILD : warning : MCG : warning MCG0007: Unresolved P/Invoke method 'DocumentDB.LogStoreViewer.dll!EnumerateDeletedStreams' for method 'System.Int32 Microsoft.Azure.Documents.LogStoreViewerInterop.LogStoreViewerInteropWrapper.EnumerateDeletedStreams(System.IntPtr, System.String, System.IntPtr)'.

@ausfeldt Do you know who's best to work with to get everything back on track? Happy to lend expertise from the .NET side of things.

Joel-Intact commented 5 years ago

Hi @ausfeldt, has there been any progress on patching Microsoft.Azure.DocumentDB.Core? I can confirm that version 2.5.1 still has this issue.

Thanks!

ausfeldt commented 5 years ago

In the next version of the SDK will have an alternative for ntdll!RtlGetVersion.

Also Microsoft.Azure.Documents.ServiceInterop.dll!CreateServiceProvider and Microsoft.Azure.Documents.ServiceInterop.dll!UpdateServiceProvider will have a flag "AllowGatewayToParseQueries" in the next SDK version, which can be set to true or 1 in app.config or an environment variable to prevent Microsoft.Azure.Documents.ServiceInterop.dll P/Invoked.

DocumentDB.LogStoreViewer shlould be removed in 2.5.1.

libc!sysctl and libc!uname are not used when the platform is Windows.

Joel-Intact commented 5 years ago

Sounds good, we're looking forward to testing it out. Thank you!

Joel-Intact commented 5 years ago

Hi @ausfeldt, could you provide an ETA when the next version of the Microsoft.Azure.DocumentDB.Core package will be released or be in preview?

ausfeldt commented 5 years ago

hopefully before end of next week.

Joel-Intact commented 5 years ago

Awesome, thank you for letting me know. I'll be able to rearrange some priorities on my end 👍

mnikonov commented 5 years ago

THANKS

ausfeldt commented 5 years ago

2.6.0 is released. If the SDK is running in direct mode, Microsoft.Azure.Documents.ServiceInterop.dll will throw a TypeLoadException. To avoid this, set either in your app.config or environment variable AllowGatewayToParseQueries to true.

Joel-Intact commented 5 years ago

Hey @ausfeldt, thanks for the update to the SDK. I've tested and verified the fix. Thanks to you, @MattWhilden and all else who helped, for all the time and effort put into this. It is very much appreciated!

Joel-Intact commented 5 years ago

@ausfeldt, @MattWhilden - hey guys, before we close this out, there's one more issue related to this:

The UWP project compiled in release with the .net native tool chain, it also created the windows store package. However, it failed the Windows App Certification Kit.

It failed the AppContainerCheck:

Binary analyzer

File C:\Program Files\WindowsApps\IntactPartnersInc.ReAccess_0.1.20.0_x64__r2zdz7y5x4a76\Cosmos.CRTCompat.dll has failed the AppContainerCheck check.
File C:\Program Files\WindowsApps\IntactPartnersInc.ReAccess_0.1.20.0_x64__r2zdz7y5x4a76\Microsoft.Azure.Documents.ServiceInterop.dll has failed the AppContainerCheck check.

The "what to do if..." information of the AppContainerCheck page section was clear on making sure the latest compiler and linker are being used and I believe we're all good there.

However, the part about the /appcontainer flag on the linker, is this something that can be set as a runtime directive on my side or does it need to be handled in the DocumentDB.Core SDK?

MattWhilden commented 5 years ago

My understanding is that those errors will have to be corrected by the folks building those libraries but I'm not an expert on the WACK platform checks.

ausfeldt commented 5 years ago

Working with the owners of those assemblies to resolve this issue. As a workaround, using gateway mode and not shipping those native asseblies should work.

Joel-Intact commented 5 years ago

Thanks @ausfeldt, that would be great. Since the UWP apps do not have an app.config and do not have access to environment variables, our options are limited.

ausfeldt commented 5 years ago

@Joel-Intact does using gateway mode and not shipping those native asseblies work in the mean time?

Joel-Intact commented 5 years ago

@ausfeldt it does not work in the mean time.

Gateway mode is to set an app setting in the app.config or setting an environment variable with "AllowGatewayToParseQueries" to true, correct?

I created Windows Store package with the environment variable present (made sure to restart my computer) and the WACK platform check still failed with the same error.

MattWhilden commented 5 years ago

Presuming Gateway mode doesn't need those libraries, you'll also need to get them out of your package to pass WACK. Either a custom build step somewhere or just removing them from the underlying packages should do the trick.

Joel-Intact commented 5 years ago

That sounds promising. I'll try it out and let you all know how it goes...

Thanks again!

ausfeldt commented 5 years ago

AllowGatewayToParseQueries would not be needed in Gateway Mode. Gateway Mode is set in ConnectionPolicy ConnectionMode that is passed into the constructor of DocumentClient. Then make sure that Microsoft.Azure.Documents.ServiceInterop.dll and Cosmos.CRTCompat.dll are not included so that it can pass WACK.

MattWhilden commented 4 years ago

The packaging ended up being fussier than I prefer but Joel and I worked something out. I suspect other customers would love a package with appropriately stamped native binaries.

Joel-Intact commented 4 years ago

Indeed, that would be great. In the meantime, for those who need to package UWP apps right away, you'll need to add the /appcontainer flag to the following DLLs with editbin.exe:

In a Visual Studio 2019 Developer Command Prompt, switch the directory to the location of editbin.exe:

Now, update the source DLLs that will be copied and bundled into the UWP Windows Store package, (the location of the DLLs may vary depending on your setup, mine are here) :

Now package your application for the Windows Store (if the build fails, clean and try again until it succeeds). NOTE: when you build the package, make sure that x64 in the only architecture selected (more on this later on).

Then run the WACK, you should not get the /appcontainer errors. If there are no more errors, the app package is ready to be submitted to the Windows Store.

Why only x64? When I packaged my app for all 3 architectures (arm, x86 and x64), the WACK passed. However, the Windows Store validation failed in the "Package Sanity Check" with and error stating that the DLLs are not compatible with the arm and x86 architectures.

trevortirrell commented 4 years ago

@Joel-Intact @ausfeldt I got the x64 package to work, but still having issues with x86 and ARM. I've tried using the original cosmos.crtcompat.dll and Microsoft.Azure.Documents.ServiceInterop.dll files and I've also deleted them. Nothing works.

trevortirrell commented 3 years ago

@Joel-Intact @ausfeldt I got the x64 package to work, but still having issues with x86 and ARM. I've tried using the original cosmos.crtcompat.dll and Microsoft.Azure.Documents.ServiceInterop.dll files and I've also deleted them. Nothing works.

@MattWhilden @Joel-Intact @ausfeldt This is still an issue. Has any progress been made?