Open rolfbjarne opened 1 year ago
Is there a list of known deficiencies in the current release mode tooling? I have experienced similar issues.
I would like to know which features require the Interpreter. Because we have run into the same thing. UseIntepreter works for release mode, otherwise it crashes at launch. I'd like to try the app without the interpreter because our client is not really happy with the performance compared to how the app ran in Xamarin Forms.
@jonathanpeppers this feels like a nice way to make customers happy and still have all the power to get that performance?
@rolfbjarne do we know what part of entity framework needs the interpreter? Technically we can make all platforms use a slower code path because a popular library needs it. However, maybe entity framework can provide an alternate code path for AOT-only environments?
@mattleibow just don't do this for Android, because it disables the JIT. iOS/MacCatalyst can't JIT, so the only downside is whatever the increase to app size. Plus there might be some things about weird stack traces Rolf mentions above.
@mattleibow
@rolfbjarne do we know what part of entity framework needs the interpreter? Technically we can make all platforms use a slower code path because a popular library needs it. However, maybe entity framework can provide an alternate code path for AOT-only environments?
I haven't investigated why entity framework needs the interpreter. I also believe there are other libraries that needs it, so it's not only entity framework.
Linking this bug for context. It is what I have marked on the line that I set the interpreter on for iOS. A couple of you have already commented there, but others might not be aware.
I am using Entity Framework, and that is likely a main reason why I set it.
Here is an example of something that seems to work for me on Debug mode, but fails on Release mode
In this branch there is a converter that is marked as internal. On Windows and/or in Debug mode it works, but on Android release mode it crashes with:
System.MethodAccessException: Method
Maui.DataGrid.SortDataTypeConverter..ctor()' is inaccessible from method
YourProject.Views.UsersPage.InitializeComponent()' at YourProject.Views.UsersPage.InitializeComponent...
To reproduce the error, just run the sample project in Android release mode. It should crash on launch.
Changing SortDataTypeConverter to public fixes it, but I don't see why Release mode should be special, so it seems like a bug.
To retrieve the exception in Release mode I have subscribed to AppDomain.CurrentDomain.UnhandledException and TaskScheduler.UnobservedTaskException from the MainActivity in the Android platform code, and logged to the console from there.
https://github.com/symbiogenesis/Maui.DataGrid/tree/android-release-mode
Is it possible in new dotnet to do the equivalent of the Xamarin.iOS --interpreter=-all
(aot everything, but allow the interpreter to handle codegen)?
Is it possible in new dotnet to do the equivalent of the Xamarin.iOS
--interpreter=-all
(aot everything, but allow the interpreter to handle codegen)?
This should work the same:
<PropertyGroup>
<MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>
I was experiencing crashes when writing to sqlite in release mode on ios. Using the interpreter stopped that happening. It also made some other minor things work again like colour changes to controls. I only experienced these problems with ios, which worked fine in debug mode. Other platforms all worked OK in both debug and release. I used the example from above and changed Debug to Release to make it work.
<PropertyGroup>
<UseInterpreter Condition="'$(UseInterpreter)' == '' and '$(Configuration)' == 'Release'">True</UseInterpreter>
<MtouchLink Condition="'$(MtouchLink)' == '' and '$(Configuration)' == 'Release' and '$(UseInterpreter)' == 'true'">None</MtouchLink>
</PropertyGroup>
Using the interpreter by default may cause perf issues for everyone - even those who don't need it.
The opposite is that some portion of the users may have issues in release and waste time trying to fix it.
Both are bad, but which is the most significant number? Is there a way to have partial interpretation so that the libraries that have issues can use the interpreter but the rest of the app is AOT?
Also, have we been able to determine what causes the issues? Maybe there is some method use in EF for example that needs to have an AOT friendly implementation on some platforms?
TL;DR: I am suggesting that if there is an issue with a few libraries for a few people then maybe we can fix those instead of negatively impacting performance for all users.
"The opposite is that some portion of the users may have issues in release and waste time trying to fix it."
That is my case. My iOS app CacheAll worked perfectly in debug and so I published the release version on the app store. This also worked fine until you went to save and then it would silently crash with no crash report.
Fortunately, a little Googling revealed the
Subsequently I uploaded to the app store and downloaded the app via TestFlight, which also showed that the crash was fixed.
BTW, so far I have not noticed a performance issue. @rolfbjarne Does UseInterpreter only affect the code that was crashing or does it affect the execution of the app overall?
I've just hit this myself after developing on mac for months but having no mac users, until today, precisely because it works in debug but not release.
While I understand the concerns for performance, it's also not a great out of the box experience for MAUI, and given that it's not an obvious tradeoff to consider (I don't remember seeing it in any documentation or tutorials I hit, anecdotally) it feels like it could be quite devastating to run into for people planning to use MAUI for mac and iOS if they depend on a library that needs it (for me it was Microsoft.Data.Sqlite I believe).
Tough call, but I would like this to be more obvious/easier to deal with than it is today. The Microsoft MAUI csproj templates have a bespoke commented-out-by-default section for tizen, in a similar vein, it might be nice to see something like this added there if not enabled by default:
<!-- Uncomment to enable the interpreter. This is needed in release if you or your dependencies use reflection -->
<!-- <PropertyGroup Condition="$(TargetFramework.Contains('-maccatalyst')) Or $(TargetFramework.Contains('-ios'))">-->
<!-- <UseInterpreter>true</UseInterpreter>-->
<!-- </PropertyGroup>-->
Yeah,you changed me :) I like this idea... We can have it enabled actually and then maybe have some publishing doc that focuses on optimization where we can talk linking, reflection and interpreters...
Should this be on from the workloads and you opt out... This will affect existing users. Or should this be a templates thing and only be something for new users.
Today we enable for release builds implicitly so maybe we just always set it for ios/maccatalyst and just release for Android. That is semi inconsistent but might be the best. The the project can override however they like but at least the default always works.
@davidbritch @rolfbjarne @jonathanpeppers do we have a doc on all the options that are good for optimizing maui apps? Obviously there are millions of buttons, but more the ones that most people use.
Should the interpreter be there along with the pros and cons?
To the comment;
Some customers don't even test their apps in Release mode, they go straight to trying to publish and they're obviously rather upset when their apps are rejected because they crash at launch.
This sort of points the finger at customers, rightly so in many cases I'm sure - however there are a few nuances here;
Quick question: Does the interpreter exist in Xamarin? If not, with it switched on does this negate any of the performance enhancements of MAUI vs. Xamarin? I don't ever recall it being mentioned prior to MAUI.
Rightly or wrongly, we as .NET developers have been trained to think that building in release mode is intrinsically safe
The trimmer does output the message:
Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
This is one source of when Release
builds go wrong, but maybe this could be a warning instead of a message?
The interpreter is more of a setting for iOS, as it enables more System.Reflection.Emit-related APIs to work in a JIT-not-allowed environment. There really isn't a reason to turn the interpreter on for Android, as it is only used for hot reload scenarios.
To the comment;
Some customers don't even test their apps in Release mode, they go straight to trying to publish and they're obviously rather upset when their apps are rejected because they crash at launch.
This sort of points the finger at customers, rightly so in many cases I'm sure
I fully agree with all your comments, and I didn't mean to point any fingers at anyone but ourselves, and I'm sorry if it came off that way.
Quick question: Does the interpreter exist in Xamarin? If not, with it switched on does this negate any of the performance enhancements of MAUI vs. Xamarin? I don't ever recall it being mentioned prior to MAUI.
Yes, the interpreter has been around for a few years now, but it was an experimental feature in Xamarin (you had to opt in).
This meant few people tried it, and even fewer ran into differences between Debug and Release.
One reason this has become such an annoying problem is because the interpreter is required for Hot Reload (which is new in MAUI), so it was turned on by default for Debug, and thus a lot of people would happily (and rightfully so) code away, using features that would require (by accident or by design) the interpreter.
This is one source of when
Release
builds go wrong, but maybe this could be a warning instead of a message?
That's not very useful for iOS, because for internal reasons the linker is always enabled, and a warning you can't fix is a warning that will be ignored.
One reason this has become such an annoying problem is because the interpreter is required for Hot Reload
Thanks @rolfbjarne,
Couple of observations;
If you mean hot reload for code (as opposed to XAML which I assume doesn't need it as it worked prior to the interpreter?) then I'd say I'd rather not have the code hot reload and get rid of the interpreter problem altogether, as the moment I diverge away from the "hello world" MAUI example, code-level hot-reload stops working anyway and none of my team use it. The XAML hot reload is much more important for iterating UIs, but as I say that seems to have been working in Xamarin long before the interpreter. Edit: Ah, I realise you might be hesitant to do this as MAUI pushes code-based (as opposed to XAML based) UI as a first class concept, is that correct?
Secondly, there seems to be a thing of third party libraries accidentally requiring the interpreter (such as EF Core), but if Xamarin didn't use the interpreter why did this never crop up? and how did libraries such as EF Core work ok back then? Is there something fundamentally different about how these libraries work now or is simply the presence of the interpreter enough to create the potential for such a requirement?
Also, I'm guessing my team never used the Xamarin interpreter (as we never knowingly opted in), but with MAUI we will need to use the interpreter for release builds, so are there any figures on how performance compares between non-interpreted Xamarin vs. interpreted MAUI? Just want to get a feel for whether we should expect to see performance degradation in any app ports whilst we migrate.
Two questions around crash report weirdness:
@gkarabin That's a good point, additionally in some cases the issue is the opposite - I think the issue isn't often the interpreter itself, but lack of the interpreter when an app tries to execute code that requires it, which can't be known ahead of time.
You almost need two error scenarios and appropriate messages;
InterpreterNotEnabledException: Failed to execute abc due to xyz. This operation requires the MAUI interpreter to be enabled on this platform, see http://aka.ms/maui-interpreter for more information.
InterpreterFailedException: Failed to execute abc due to xyz. An error occurred within the MAUI interpreter.
Diagnostics:
-------------
<something here - last instruction, method name, etc.>
Secondly, there seems to be a thing of third party libraries accidentally requiring the interpreter (such as EF Core), but if Xamarin didn't use the interpreter why did this never crop up? and how did libraries such as EF Core work ok back then? Is there something fundamentally different about how these libraries work now or is simply the presence of the interpreter enough to create the potential for such a requirement?
Each of these cases are different, and would have to be investigated separately. I know that there were some regressions in some cases when merging old Mono-based code from Xamarin with Microsoft-based code (because the Mono-based code was implemented to be safe in a FullAOT scenario - i.e. no interpreter/jit - while Microsoft historically never had FullAOT support, so the code was written differently). That said, I don't know if EFCore is in that bucket. In either case, I would recommend filing issues in dotnet/runtime about these cases, and we'll investigate and try to figure out what can be done.
Also, I'm guessing my team never used the Xamarin interpreter (as we never knowingly opted in), but with MAUI we will need to use the interpreter for release builds, so are there any figures on how performance compares between non-interpreted Xamarin vs. interpreted MAUI? Just want to get a feel for whether we should expect to see performance degradation in any app ports whilst we migrate.
That's extremely dependent on the app. If your app is a game doing a lot of math in C#, it will be much slower (probably excruciatingly so). If you're app does very little CPU heavy code in C#, then the performance slowdown might even be unmeasurably small.
Also note that it's possible to disable the interpreter on an assembly basis, so if you find out that you have a particular assembly where you're doing a lot of math, you can disable the interpreter for that assembly.
- If the interpreter is enabled, is it obscuring all stack traces or just ones where a code path requires interpretation (e.g., some reflection emit call). I had assumed that parts of the app remain AOT compiled and are not interpreted, with normal stack traces. If it’s just some crash reports that are useless it’s not quite so bad.
Any stack frame executed by the interpreter will be rather useless. The interpreter can be disabled on a per-assembly basis, so it's possible to have some frames from some assemblies accurately depicted in any crash reports, while others will just say the interpreter is doing stuff.
- If the app crashes when in the interpreter, does the interpreter itself have useful diagnostics that could be added to the crash dump to compensate?
We can't add anything to the built-in crash report on iOS at runtime. The crash reporter is an out-of-process executable that examines the process right before it's terminated by the OS, and writes out the results (i.e. the crash report).
If the crash originates from an unhandled managed exception, then that exception might contain more information, and in fact it does sometimes. Example exception:
System.ExecutionEngineException: Attempting to JIT compile method ... while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.
There's a link there, but unfortunately it doesn't say anything about the interpreter. However, I've filed https://github.com/MicrosoftDocs/xamarin-docs/issues/3546 to try to rectify that.
Thanks for the awesomely thorough explanations, @rolfbjarne. The per-assembly approach has legs for my apps - the performance critical stuff is limited to native code (C++) and just a few assemblies.
I wonder if there's room in the iOS Release build process to look for assemblies that would require the interpreter to run correctly, and fail the build if the assembly doesn't use the interpreter? Dunno if it would just be heuristics or if you could be 100% sure.
That would give people a way to know about the problem before it's found in production. I realize its scope creep compared to the original point of this issue, but it might hit the sweet spot if it can be done without blowing up build times.
To my understanding, in most (all?) situations, none of the assemblies available at compile time should require the interpreter to execute. "Problematic" libraries emit new code into new dynamic assemblies that only exist at runtime. These would typically be JIT'd, but here need to be interpreted, and (probably/practically) can't be named in your csproj ahead of time.
So I think the solution in dotnet ios, is the same as in Xamarin.iOS - this:
<PropertyGroup>
<MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>
and it might be a reasonable project default.
I became aware of this fix in the EF Core 8 RC1 blog post https://github.com/dotnet/efcore/issues/29725 and now I'm wondering whether this might be related to this topic? Does this make UseInterpreter obsolete, at least for using EF Core Sqlite for iOS?
Just for the record, it's not just EF Core. I have also experienced problems that only appear in Release mode when using Autofac with its autogenerated Factory registrations
https://autofac.readthedocs.io/en/latest/resolve/relationships.html#dynamic-instantiation-func-b https://github.com/autofac/Autofac/issues/1398#issuecomment-1771008802
I'll be switching over to Microsoft DI
Any news on this one? Wouldn't the setting below be a sensible default for iOS as @rdavisau and @rolfbjarne pointed out?
<PropertyGroup Condition="$(TargetFramework.Contains('-ios'))">
<MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>
Another common scenario that fails is Linq Expressions with more that 2 parameters: https://github.com/dotnet/runtime/issues/94063
Any news on this one? Wouldn't the setting below be a sensible default for iOS as @rdavisau and @rolfbjarne pointed out?
<PropertyGroup Condition="$(TargetFramework.Contains('-ios'))"> <MtouchInterpreter>-all</MtouchInterpreter> </PropertyGroup>
Another common scenario that fails is Linq Expressions with more that 2 parameters: dotnet/runtime#94063
Actually, after some more troubles with that parameter I'd advise against that as a default.
Even with -all
we still experienced crashes in our iOS-Release builds, the latest one in the Maui Community Toolkit :/
Finding the root culprit/assembly is a very manual & cumbersome process, even after @rolfbjarne gave me some tips on Discord to speed things up (Kudos for that 👍)
We still put up with <MtouchInterpreter>-all</MtouchInterpreter>
because our migrated Maui app is still slow as ass, compared to the old Xamarin version and we need to squeeze out every ounce of performance we can get 😕
NativeAOT is an even bigger pita. The crashes are even more numerous and the stacktraces/hints next to nonexistant. At the current pace I expect NativeAOT not to be usable for anything but console apps before .Net 11, but I digress...
I would recommend to stick with <UseInterpreter>true</UseInterpreter>
as a default for Debug & Release builds unless you consider all MAUI users to be either compiler-experts or masochists.
I would like to point out one big benefit of AOT that doesn't seem to be discussed: code protection. IL code is stripped when using AOT, and this helps protect the app from reverse engineering. When enabling the interpreter, the IL code is not stripped from the compiled assemblies. I don't think many users are aware of this difference. Some users probably don't care, but others might. I have been down the road of using obfuscation for protection, but it is honestly really painful. In my opinion, this is one of the biggest benefits of using AOT.
In a related point, it appears that IL Stripping and MtouchInterpreter is all or nothing. For example, if you want to use the interpreter for a single assembly and AOT everything else (e.g.
@mjo151 do you know if IL is not stripped even if you set -all
alone?
Whatever the benefits of a AOT, if the downside of it being enabled is that my app doesn't run they don't matter too much.
We view AOT/IL stripping etc. as very, very dodgy once you get out of "hello world" territory and smells more of the result of an academic exercise done within the CLR team than something that should be relied upon in production. Automated tooling that results in potentially changing the meaning of a program resulting in requiring deep manual testing to ensure it hasn't broken anything has massive inherent risk. Its USP seems to be "making stuff faster" but in reality, my customers will be happier with an app which takes another 0.5 seconds to launch but is stable than something that just fails more quickly.
We've had to completely disable AOT in Android production due to weird errors preventing our users booting the app at all. The underlying issue is still open.
In a related point, it appears that IL Stripping and MtouchInterpreter is all or nothing. For example, if you want to use the interpreter for a single assembly and AOT everything else (e.g. -all,System.Collections.Immutable), it appears that IL is NOT stripped from the AOTed assemblies. Maybe that is by design, but I did not expect that to be the case.
That's by design. If that single interpreted assembly references another assembly that's AOT-compiled, the AOT-compiled assembly might not have AOT-compiled everything the interpreted assembly needs. In these cases the interpreter needs to interpret code from the AOT-compiled assembly, and as such the actual IL to interpret must be present.
@mjo151 do you know if IL is not stripped even if you set
-all
alone?
I just tried only the -all option and it looks like the IL code was not stripped.
In a related point, it appears that IL Stripping and MtouchInterpreter is all or nothing. For example, if you want to use the interpreter for a single assembly and AOT everything else (e.g. -all,System.Collections.Immutable), it appears that IL is NOT stripped from the AOTed assemblies. Maybe that is by design, but I did not expect that to be the case.
That's by design. If that single interpreted assembly references another assembly that's AOT-compiled, the AOT-compiled assembly might not have AOT-compiled everything the interpreted assembly needs. In these cases the interpreter needs to interpret code from the AOT-compiled assembly, and as such the actual IL to interpret must be present.
OK, I figured there was a good reason.
Whatever the benefits of a AOT, if the downside of it being enabled is that my app doesn't run they don't matter too much.
We view AOT/IL stripping etc. as very, very dodgy once you get out of "hello world" territory and smells more of the result of an academic exercise done within the CLR team than something that should be relied upon in production. Automated tooling that results in potentially changing the meaning of a program resulting in requiring deep manual testing to ensure it hasn't broken anything has massive inherent risk. Its USP seems to be "making stuff faster" but in reality, my customers will be happier with an app which takes another 0.5 seconds to launch but is stable than something that just fails more quickly.
We've had to completely disable AOT in Android production due to weird errors preventing our users booting the app at all. The underlying issue is still open.
Xamarin.iOS apps have always been AOTed and have worked perfectly. I've had a Xamarin.iOS in production since 2013, and never once had to use the interpreter.
I proposed a new Development configuration for interpreter and other stuff speeding up e.g. XAML development here #21678 .
Just another vote to enable the interpreter by default on release IOS. It took me a few days to happen on to this thread and figure out why it was crashing in release. It doesn't seem a great decision to need to find an obscure configuration option to make a relatively trivial application work because some downstream package requires it
Couldn't agree more with Martin. Can you please consider updating this in an upcoming future release? This seriously took some time debug and was such a slap in the face when it was resolved.
I’m developing a high-performance app in .NET MAUI, so definitely do not want the interpreter to be on in general (it is roughly 50x slower in my case), but as a general principle, Debug and Release should have a very similar setup, hence I earlier proposed a separate Development configuration for the interpreter. Debug using interpreter and Release not does not make a whole lot of sense.
We've had to turn off the Interpreter to stop the app from crashing in Debug. One major issue is that that even for minor changes we're now waiting about 4.5 minutes before the app builds and runs on a phone. We're using
<MtouchInterpreter>-all,OurApp.dll</MtouchInterpreter>
in the csproj. Before we did this the build was about 2 minutes which still isn't great but it's better than 4.5 minutes. Is there any advice about how to stop the app from crashing by turning off the Interpreter but also not end up with a major increase in the build time?
We've had to turn off the Interpreter to stop the app from crashing in Debug. One major issue is that that even for minor changes we're now waiting about 4.5 minutes before the app builds and runs on a phone. We're using
<MtouchInterpreter>-all,OurApp.dll</MtouchInterpreter>
in the csproj. Before we did this the build was about 2 minutes which still isn't great but it's better than 4.5 minutes. Is there any advice about how to stop the app from crashing by turning off the Interpreter but also not end up with a major increase in the build time?
You'd have to diagnose exactly why it's crashing, and then try to find a solution based on that.
Feel free to open a new issue and we'll help you along the way.
I don't want to be asked to produce a reproducible repo, since that's not possible, and then just have the issue closed or ignored.
In our case it's third party dlls that are the problem. How do I go about diagnosing what the issue may be in those dlls? I've tried adding the dlls to the MtouchInterpreter csproj entry but now I just get InvalidProgramException when calling one of the methods. Are there any best practices defined somewhere so that third party dlls don't end up with these problems or some list of coding practices that could be problematic, or other information on how to work out the problem?
@JRosanowski I would start by looking for any crash reports and see if the device log shows anything useful.
Description
Currently the interpreter is enabled for Debug builds:
https://github.com/dotnet/maui/blob/6b43399d3f1b9520925bebdab680076787ab5e60/.nuspec/Microsoft.Maui.Controls.props#L8-L9
At least for iOS and Mac Catalyst, we might want to make this the default for Release builds as well, because I'm seeing a lot of customers running into problems where their app crashes once they switch to Release builds, because they used some feature in their app that requires the interpreter.
Some customers don't even test their apps in Release mode, they go straight to trying to publish and they're obviously rather upset when their apps are rejected because they crash at launch.
While the interpreter might slow things down, the performance will be fine for most apps, and if someone really needs the additional performance they can disable the interpreter.
The main downside might be that crash reports will be rather useless, because all managed stack frames will just show up as the interpreter doing "stuff".
Steps to Reproduce
Triage bugs for a while.
Alternatively read about customer's experiences online (Twitter, blog posts, etc.).
Example blog post: https://www.andreasnesheim.no/my-experience-with-migrating-my-app-from-xamarin-forms-to-net-maui/
Link to public reproduction project repository
N/A
Version with bug
7.0 (current)
Last version that worked well
Unknown/Other
Affected platforms
iOS, macOS
Affected platform versions
All so far
Did you find any workaround?
Relevant log output
No response
References