.NET Native - Perf: LINQ queries run slower on Release mode

[abdu292]

Friends, I was doing some performance testing before I publish the app on to the store. However here is one critical issue I am seeing.

The below code takes around 8 second to run in Release mode, but runs within just 2 seconds in debug mode. (where the related tables are very small tables and the number of records in the main table is 150)

If the code runs within 2 seconds in debug mode (with debug overhead), I would expect to run this within milliseconds in release mode. If I double the records (150=>300), the amount of time it takes also doubles (as one would expect). I know from the docs using Anonymous LINQ query can result in poor performance in RC1, and I believe it is fixed in RC2. Even it did not, wondering if the "Include" comes under this category, and this is the reason for the performance issue. If so, do we know how we can refactor the "include" that gives me better performance (may be more than debug)?

public async Task<IList<Student>> GetStudentsAsync() { var students = await _context.Students .Include(b => b.Addresses) .Include(b => b.Emails) .Include(b => b.Titles) .Include(b => b.Phones) .Include(b => b.Websites) .ToListAsync();

return students; }

I am using Visual Studio 2015, UWP 5.1.0, Native Tools 1.3.1, Entityframework Core RC2 and SQLite.

[DrewScoggins]

I was able to get this on a phone and get a trace. I am still seeing times on the phone of about 1300 milliseconds versus 350 milliseconds on CoreCLR. The majority of that time is being spent in the interpreter running the LINQ expressions. In particular, we are spending a large amount of time getting the return type of the MethodInfo inside the interpreter. The majority of this time is being spent in TryGetNamedTypeForTypeReference. This seems like a place were we could do some caching to speed this up, but I am not an expert on reflection. Adding @MichalStrehovsky.

[MichalStrehovsky]

TryGetNamedTypeForTypeReference

Yeah, this is a linear scan of a table that maps metadata to the runtime types that exist within the system. It wasn't expected to become hot, but since MethodInfo doesn't cache the return parameter (and the way ReturnParameter is implemented actually hits the slow method for each parameter, and then throws everything but the return parameter on the floor), this became hot.

There are two things we can do:

1. Get rid of the linear scan. We should be able to replace it with a hash table. I already have a bug tracking this internally, it just never was high pri enough. I'll make it higher pri.
2. If things are still not fast enough, add a cache for MethodInfo so that we don't recompute everything all the time. At minimum, split the return parameter from other parameters.
3. If things are still not fast enough, add a cache to the LINQ interpreter.

I would be really happy if 1 solves it completely because 2 and 3 would cause a regression in working set.

Unfortunately, I can't share any timelines as to when an update of .NET Native with this would be available, and no workarounds come to my mind.

[popcatalin81]

@MichalStrehovsky There's no reason to have linear scans anywhere in the EF metadata, because you will hit client a scenario with performance issues. (It's practically guaranteed considering the variety of databases out there).

Please remove all linear scans from the EF metadata.

P.S. I've seen databases with 1200 Tables and 16000 columns and 5000 Foreign keys. The funny thing is, in a typical EF6 query (with cached metadata) 95% of the execution time was spent navigating the metadata (Yup, I've seen methods called hundreds of millions of times during a single query materialization).

[MichalStrehovsky]

@popcatalin81 Thanks for the feedback. The issue at hand is about the metadata used for reflection, not EF metadata.

[divega]

@MichalStrehovsky @DrewScoggins this is very good (and promising) information. Re the different alternative ways to get the performance back, I agree it would be nice if you can get it all without regressing the working set, but if the gains with a cache were much greater, would you see this as something users could choose to enable? Or even plugging in a cache that can evict based on memory pressure?

[MichalStrehovsky]

OK, I have a workaround for you that makes things better. The performance bug is in a code path that is only used when the shared library feature of .NET Native is enabled. At the time the blog post I'm linking to was written, this feature was opt in, but since Update 2 (?) it's opt out (because most people would want it).

If you opt out, you'll avoid the perf bug. It's still not as fast as CoreCLR, but from what I measured on my machine, we're within 25% (we haven't done focused performance work on .NET Native yet - for v1, our focus was "get it working!" - we plan to beat CoreCLR on all perf aspects eventually).

As to the workaround:

1. Right click your app's project in Visual Studio and select Unload project.
2. Right click again and select Edit.
3. Paste this in the first PropertyGroup you see:

<UseDotNetNativeSharedAssemblyFrameworkPackage>false</UseDotNetNativeSharedAssemblyFrameworkPackage>

Save the project, right click and select Reload. Rebuild and check out the perf.

[mvermef]

gonna check this out cause this BUG is a no release for me till its fixed, will absolutely kill tablets trying to view a single query of 260 records.

[MattWhilden]

@mvermef have you tried the workaround listed by @MichalStrehovsky? If that doesn't clear things up for you, there may be another issue we need to get tracked down.

Happy to help, let us know.

[mvermef]

Will find out later this morning when I get up... 😀

[mvermef]

YES!!!!! Instantaneous like in debug build no toolchain, 1 second for 260 records compared to 25seconds... magnitudes faster.

[MattWhilden]

Excellent. Do let us know if you see anything else out of the ordinary.

[abdu292]

I am not sure how it works for @mvermef . It doesn't work for me and the app crashes in release mode. Unfortunately with the latest insider build my Lumia 535 has issues in dumping the crash report (I am unable to set the crash report count to >0 and because of this I have no other way to find out what is the root cause of this crash). I apologize for the delay. I worked hard on two different UWP apps for several months but had to put on hold because of different issues (most of them are related to Native Toolchain). I invested a lot of time and effort and feeling pain in mind. However I still love Windows and will continue working. For now I just started with Xamarin. I am surprised that it looks almost matured (because it was a paid one before MS acquisition?) and I was able to create almost an app (mainly for Android, and then iOS + Windows) within a few weeks without much issues.

Why is that we have everything slow in the Windows world? See this request was created almost 4 months ago but with a very slow move so far. Even we have a great tool set (I confirm that Windows Dev tools are the best exists today) why the support and feature developments are too slow?

Alright. Please let me know if you have any workaround to get the dump report in release mode or I will wait till next insider build and will send across the dump report as to what was the issue.

I appreciate your support.

[MattWhilden]

@abdu292 can you send details (build numbers etc) to me at dotnetnative@microsoft.com? Happy to help chase down the crashing issues with you.

[abdu292]

Thank you! I sent across. Please let me know if you need further info.

[abdu292]

Just to clarify, what I meant is if I use the solution given by @MichalStrehovsky my app crashes in release mode, but works if I remove that code from projcs file. I am unable to debug the issue that causes this crash, due to not being able to get the crash dump after the latest insider build on my Windows Phone. (See attached. After the latest insider build the "Error Reporting" count is always zero and you cannot change it.)

[MattWhilden]

Sure. I'll send that build number to some folks to see if there's something broken with crash collection for that build. In the mean time, if you deploy your app via Visual Studio, my expectation is that the debugger should help us make progress.

[abdu292]

Sure. Here is the exception when we enable the code suggested by @MichalStrehovsky

@MattWhilden Thanks for giving the tip (in your mail) to break at the exception in release mode via exception settings. I thought this is used only to break within a try catch block.

Let me know if I can get further info from this release debug session. Thank you!

[aflx]

@abdu292 I had this problem too (access violation on db migrate). I disabled "Optimzie code" in project build settings and the error was gone...

Maybe not the best option, but for me the only one :(

[aflx]

Damn... Without code optimization, WACK fails with "API CoCreateInstance in api-ms-win-core-com-l1-1-0.dll is not supported..."

Don't know, how to fix this... :(

[aflx]

Ok, I guess I have a solution.

Keep "Optimize code" enabled. Add following lines to Default.rd.xml (in Application section):

<Type Name="System.Nullable{T}" DoNotInline="true" DoNotOptimize="true"/>
<Type Name="System.Collections.Generic.List{T}" Dynamic="Required All" />
<Type Name="System.Collections.Generic.IEnumerator{T}" Dynamic="Required All"/>

Add this line to the first PropertyGroup in your .csproj project file (like written in a previous comment):

<UseDotNetNativeSharedAssemblyFrameworkPackage>false</UseDotNetNativeSharedAssemblyFrameworkPackage>

(I had to remove my app from the device in order to get it up and running.)

[abdu292]

Hey @aflx Thank you for taking time to respond to this. Yes, it works with the solution you provided and with the configuration what @MichalStrehovsky gave, now the it takes around 6 seconds to load 150 records (it was around 11 seconds before). Though it is a good news that we make some progress, the bad news for me is that I still cannot productionize my app. I am expecting to have this loaded within half a second (like in debug mode). Hope to see a resolution sooner on this. Please share if you find something more. Thanks again!

[abdu292]

@aflx Wondering if you see the performance that you expect. Do you see it is faster as it is in debug mode and ready for production?

[aflx]

@abdu292 Hm. It seems to be as fast as in debug. But I didn't have numbers yet (I will test it later)... At least the performance is good enough for me to have a private beta online :)

[abdu292]

@aflx Good to know. Thanks for the response!

[abdu292]

@MattWhilden @divega Team - Wondering if we have any update on this. At least a hint of direction that we are headed to, in resolving this issue? As per my understanding there is not even a work around for this issue.

It is very tough to move forward with our UWP development without you helping us. As we all know working with data is very common. If resolution for this issue gets delayed that can make our development extremely painful. Hope to see a response at your earliest convenience.

[abdu292]

@MattWhilden @divega @MichalStrehovsky A response would be helpful. Thanks!

[janabimustafa]

This really is a breaking issue. I am experiencing slow performance from EF+SQLite when deploying to the phone with the release profile. It takes upwards of 30 seconds to execute a DbSet.ToList opertation on the phone compared to around 3 on the desktop. I've tried every suggestion in this thread and it's still only marginally better.

[mstahl77]

I have had lot of problems with EF+SQLite and .NET Native also. I had to rewrite many queries to not use Include or joins to make it work with .NET Native. And now when I have simple selects without joins there are still performance problems. Trying to do workarounds by caching data in memory now.

When can we expect any performance improvements?

[MichalStrehovsky]

The shipping schedule of .NET Native compiler and tools is locked to the shipping schedule of Visual Studio UWP SDK. We understand the frustration this causes - it's not a good place to be in for us, or our users. Visual Studio is a big project and it takes time to get fixes through.

We are working on solving both problems - to bring the bug fixes and performance improvements required for this particular issue, and to put the .NET Native compiler tools in a position where we can roll out newer versions at a faster cadence. GitHub is not the place where we would disclose timelines for unreleased products though.

[abdu292]

Thanks @MichalStrehovsky for the update. At least we can relax knowing it is being looked at. The silence was pretty annoying with no update.

[MichalStrehovsky]

@abdu292 If you would like to, we have a beta version of the .NET Native compiler toolchain ready to try out.

You'll need Visual Studio 2017 RC. By default, VS 2017 RC comes with pretty much the same .NET Native compiler that shipped with VS 2015 Update 3, but your project can opt in to get the beta:

• Right click the project and select Manage NuGet Packages...
• Make sure to check the Include prerelease checkbox
• Set the Package source to All or Microsoft Visual Studio Offline Packages
• You should see an update is available for Microsoft.NETCore.UniversalWindowsPlatform (version 5.3.0-beta). Install that.

After rebuilding your app, you should see the perf improvements. For your SQLitePerf.zip demo, I'm seeing these numbers:

• 20% faster than CoreCLR on first click on Tap to Load
• About 5% slower than CoreCLR on subsequent clicks

[abdu292]

@MichalStrehovsky Hey, thanks for the response. In reality with your workaround that you suggested in the past (see my comments posted on 26 Sep), I get a better performance from 11 seconds to 6 seconds to load 150 records. However this is still too much from an user's stand point. I would like to have a debug like experience (half a second) before I can productionize my app.

Apologies, 20% with this beta version is not so exciting to me. Thanks and all of the hard work of the team is much appreciated!

[MichalStrehovsky]

Apologies, 20% with this beta version is not so exciting to me.

Sorry I wasn't clear. CoreCLR is the "debug experience". 20% faster than CoreCLR means 20% faster than Debug.

[abdu292]

Oh. Sorry. I should have been more attentive in reading your comment. I misunderstood that it is 20% faster than the release mode. It is a great news. I'm going to try this out. I don't have VS 2017 on my dev box however I can spin up an Azure VM to test this.

I'll post my findings at my earliest convenience. Thanks a bunch.

[abdu292]

@MichalStrehovsky Okay. So here is my findings on my project. Please note that I have 4 projects in my solution (excluding test projects) Main, Business, Data & Shared and I upgraded all of the four projects to 5.3.0-beta version.

VS 2017 RC Debug Mode - Loaded 300 records in 6 to 7 seconds Release Mode - Loaded 300 records in 6 to 7 seconds (Same length)

VS 2015 Update 3 Debug Mode - Loaded 300 records in 4 to 5 seconds Release Mode - Loaded 300 records (in the past) in 8 seconds

It appears that in the new version even the debug mode takes a little more than what it used to be in the older version. I would expect the new version of release mode to take less then the time of debug mode of older version. That is 2 to 3 seconds for 300 records. Please let me know your thoughts.

Thanks!

[mstahl77]

@MichalStrehovsky I did one loading test with my App also.

VS 2015 Update 3 (and UniversalWindowsPlatform 5.2.2) Debug Mode - 1.0 s Release Mode - 3.7-3.9 s

VS 2017 RC (and UniversalWindowsPlatform 5.3.0-beta) Debug Mode - 1.5 s Release Mode - 0.9-1.3 s

Looks like it's a huge improvement in VS 2017 RC with Release Mode. Good work! But interesting that it seems to be a bit slower in Debug mode than before.

[VladimirAkopyan]

I've been struggling with this myself, and tested a few workarounds,since this is blocked maybe it would help someone. I tried materialising the data into tuples was the fastest way to read the data, see I described this on Stackoverflow And second I found a no-sql db DBreeze is extremely simple to use, and although it's no-sql, the perf (200,000 records per second) is worth the trade-off in some cases. It's written in c# and support UWP

[divega]

Based on @MichalStrehovsky's update and data reported by users we believe this issue is resolved by the 5.3.0 version of .NET Native currently in beta.

Note that a user also reported the performance on Debug being 50% worse in the new version. If this becomes an issue for development of it turns out to affect other kinds of application we will deal with it as a new bug.

[aflx]

@divega Is it also working with VS2015?

[divega]

@aflx I will let @MichalStrehovsky respond if an improvement is planned that will work with VS2015.

[MichalStrehovsky]

@aflx The 5.3.0-beta .NET Native compiler requires VS2017. Consuming the compiler from a NuGet package requires Visual Studio features that are not present in VS2015.

[aflx]

@MichalStrehovsky Thx! Just downgraded back to 2015 because of some trouble/missing extensions ;)

Just for info: would this work with store releases, too? As fa as I understood, everything is compiled in the cloud, right?

[MichalStrehovsky]

would this work with store releases, too?

Yes, this will work for Store scenarios too, eventually. 5.3.0-beta is a preview release though and can't be used to submit apps to the Store. We're working on having something available soon.

[abdu292]

@divega @MichalStrehovsky Sorry for the late response. I was on my vacation. I am unsure if this issue can be closed. I was expecting a performance improvement of 4 seconds as opposed to 6 to 7 seconds for 300 records (Please see my comments on Nov 26, 11.20 AM IST - 11 days ago). This issue doesn't necessarily mean we are expecting the performance same as debug any more and that is because in this new version, even the debug mode is affected with the performance issue. I believe debug was affected a couple of times. Because if you see my first post (very first post of this issue) I am saying I was able to run 150 records within 2 seconds (less than 4 seconds for 300 records) with Visual Studio 2015, UWP 5.1.0. However unfortunately that's not the case anymore. Currently VS2015 with Update 3 takes 4-5 seconds and VS2017 RC takes 6-7 seconds (see my comments posted 11 days ago). Hope to see your responses.

[abdu292]

@divega @MichalStrehovsky I want to re-open this due to reason mentioned in my previous post. I don't see an option to do so. Can any one help please?

[MichalStrehovsky]

@abdu292 Please open a new issue. The original issue (.NET Native being several times slower than CoreCLR) has been fixed and the updated compiler will ship with a go-live license soon. This thread is pretty convoluted and it is not clear what baselines the various numbers are referring to. The EF team will need to have a look at it and determine if the root cause is still the runtime (but since the times are now pretty similar between CoreCLR and .NET Native, it may well be an issue in EF instead).

If you would like to pursue the CoreCLR ("Debug build configuration") regression, please open a separate issue for that too - it's a different set of people who would act on it. I can't speak for the CoreCLR team, but since CoreCLR is not a shipping scenario for UWP apps, the they may only act on it if:

1. The performance is so bad that it makes the Debug builds hard to use (doesn't seem to be case here)
2. The repro case is an ASP.NET scenario (where CoreCLR is the shipping runtime and they take perf very seriously)

[abdu292]

@MichalStrehovsky That makes sense to me. Thanks for the clarification.

[saurabha5]

@abdu292 , @MichalStrehovsky can you please link the new bug here so that we know the actual status. I still see lot of performance lag with .Net native compilation.

[abdu292]

@saurabha5 This issue is not actually on the performance of the compilation, but on the querying of numerous records in release mode. I tested this (I think a week ago), with the latest UWP version, and it looks like it works and no more performance issues. I am going to publish my app now (after holding for a long time) The .Native toolchain compilation still takes a while however much better than how it was before.