Webassembly AOT Crashes Android Mobile (v96)

legistek commented 2 years ago

Hi, unfortunately this closed issue seems to have re-emerged.

AOT .NET6 works perfectly in Chrome on Windows but on mobile Android the Webassembly piece starts to load, then gives the "Aw Snap" reload screen of death. Also works perfectly on mobile Chrome when using interpreted mode so this is limited to AOT on Chrome Mobile.

Other info: Chrome 96.0.4664.45 Android 11 Pixel 4 XL Build RQ3A.210905.001

My co-worker had the same problem on his Pixel 5, but NOT when using the Adblock browser - only Chrome.

I'm afraid I'm not in a position to post code and have not found the minimal reproducible steps yet. I'm hoping someone might understand the basics of what might be going on sufficiently to suggest some diagnostic steps.

BTW, it took me awhile to get around to upgrading to NET6 to try AOT but it's really fantastic. My application is MUCH MUCH faster across the board and honestly the file sizes and startup times aren't any worse than our legacy Angular app primarily because our main JS payload is so much smaller now.

So I really think AOT will be important for mobile devices and will do what I can to assist in finding a solution to this.

Originally posted by @danroth27 in https://github.com/dotnet/aspnetcore/issues/33559#issuecomment-885895324

daniel-p-tech commented 2 years ago

@legistek my comment that you've linked was related to an issue with a Blazor app running in the interpreted mode, which had been fixed by Google (at least for the time being). You may be experiencing a different problem.

legistek commented 2 years ago

Ohh, thanks @daniel-p-tech sorry I missed that detail. Could be a different issue then.

I will confess I'm doing some, let's say, unconventional stuff with WASM-Javascript interop. But, since everything works fine in interpreted mode and AOT in Windows Chrome, it seems reasonably likely that it's not anything I'm doing wrong and that this is affecting other people/applications.

mkArtakMSFT commented 2 years ago

Can you please create a new project, publish with AOT and try to run on the same device and see whether you hit that issue or not?

ghost commented 2 years ago

Hi @legistek. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

ghost commented 2 years ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.

legistek commented 2 years ago

I will get you an answer shortly. Please don't close it. Thanks.

legistek commented 2 years ago

Ok, so not surprisingly, just compiling the default project with AOT and publishing it on a server works fine with my Android Pixel.

I definitely want to try to do what I can do diagnose this, but I'm going to need a little bit of help/guidance. After all, publishing AOT is pretty slow, so trying to find the exact thing in my code that causes it is going to be excrutiatingly difficult without some guidance.

So the question is what sorts of things could possibly be different between AOT and interpreted that could cause the browser to totally crash in one and not the other?

As I mentioned I am pushing the envelope a bit. For example I'm using some of the low-level JS interop (the same things that Blazor itself uses), such as providing a custom argument signature to bind_static_method, returning pure Mono objects to my C# code,, etc. The stuff that the Mono source code says not to use unless I "know what [I'm] doing". (Well, I thought I did...)

But this bears repeating:

It works fine interpreted on all devices
It works fine AOT on Windows Chrome
It works fine AOT on Android Adblocker Browser
It crashes on AOT on Android Chrome

There must be a finite number of things it could be.

Any suggestions of things to try to diagnose this would be welcome! Thanks!

legistek commented 2 years ago

I managed to do a remote debug on my phone and see this in the browser console:

Streaming compilation failed. Falling back to ArrayBuffer instantiation. RangeError: WebAssembly.instantiate(): Out of memory: wasm memory

Hope that means something?

legistek commented 2 years ago

I can also confirm that my Program.cs never actually gets executed, as I have a Console.WriteLine message as the first line of Main. In other words my C# code never gets executed before the browser crashes.

legistek commented 2 years ago

I found this related issue and so added this to my WASM project's CSPROJ to reduce the initial memory allocation: <EmccTotalMemory>134217728</EmccTotalMemory>

Now instead I get this console error:

Streaming compilation failed. Falling back to ArrayBuffer instantiation.  TypeError: Could not download wasm module
blazor.webassembly.js:1 TypeError: Failed to execute 'arrayBuffer' on 'Response': body stream is locked
window.Module.s.printErr.e @ blazor.webassembly.js:1
(anonymous) @ blazor.webassembly.js:1
async function (async)
(anonymous) @ blazor.webassembly.js:1
window.Module.s.instantiateWasm @ blazor.webassembly.js:1
createWasm @ dotnet.6.0.0.qb2grxhl7x.js:1
(anonymous) @ dotnet.6.0.0.qb2grxhl7x.js:1
blazor.webassembly.js:1 Uncaught (in promise) TypeError: Failed to fetch
async function (async)
addToCacheAsync @ blazor.webassembly.js:1
loadResourceWithCaching @ blazor.webassembly.js:1
async function (async)
loadResourceWithCaching @ blazor.webassembly.js:1
loadResource @ blazor.webassembly.js:1
(anonymous) @ blazor.webassembly.js:1
window.__wasmmodulecallback__ @ blazor.webassembly.js:1
(anonymous) @ VM20:1
(anonymous) @ blazor.webassembly.js:1
Promise @ blazor.webassembly.js:1
start @ blazor.webassembly.js:1
St @ blazor.webassembly.js:1
async function (async)
St @ blazor.webassembly.js:1
(anonymous) @ blazor.webassembly.js:1
(anonymous) @ blazor.webassembly.js:1
blazor.webassembly.js:1 Uncaught (in promise) TypeError: Failed to execute 'arrayBuffer' on 'Response': body stream is locked
    at e.response.then.e (blazor.webassembly.js:1)

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

mkArtakMSFT commented 2 years ago

@lewing can you guys please look into this? Thanks!

legistek commented 2 years ago

Just a little more information that may help:

I tried a second project that shares a lot of the code with the first one, but is quite a bit smaller. This one originally crashed with the same issue, prior to me using the EmccTotalMemory tag.

Once I added EmccTotalMemory to 128MB, it worked fine.

My main project, however, crashes no matter what memory setting I used. (Well, I've tried 128MB and 256MB).

It does seem likely that the EmccTotalMemory issue has something to do with this though.

Perhaps someone close to that issue could provide some tips on how to set it properly? Or is there any way of knowing how much memory a particular Chrome version will allow for the array buffer on a particular device?

Thanks!

ghost commented 2 years ago

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details

Hi, unfortunately this closed issue seems to have re-emerged. AOT .NET6 works perfectly in Chrome on Windows but on mobile Android the Webassembly piece starts to load, then gives the "Aw Snap" reload screen of death. Also works perfectly on mobile Chrome when using interpreted mode so this is limited to AOT on Chrome Mobile. Other info: Chrome 96.0.4664.45 Android 11 Pixel 4 XL Build RQ3A.210905.001 My co-worker had the same problem on his Pixel 5, but NOT when using the Adblock browser - only Chrome. I'm afraid I'm not in a position to post code and have not found the minimal reproducible steps yet. I'm hoping someone might understand the basics of what might be going on sufficiently to suggest some diagnostic steps. BTW, it took me awhile to get around to upgrading to NET6 to try AOT but it's really fantastic. My application is MUCH MUCH faster across the board and honestly the file sizes and startup times aren't any worse than our legacy Angular app primarily because our main JS payload is so much smaller now. So I really think AOT will be important for mobile devices and will do what I can to assist in finding a solution to this. _Originally posted by @danroth27 in https://github.com/dotnet/aspnetcore/issues/33559#issuecomment-885895324_

Author:	legistek
Assignees:	-
Labels:	`arch-wasm`, `area-Codegen-AOT-mono`
Milestone:	-

radical commented 2 years ago

cc @radekdoulik

legistek commented 2 years ago

Hi all - I realize it may be awhile before anyone can turn to this, but I figured I'd offer this update -

My application works fine, AOT or interpreted, on a new iPad.

Thus, so far, the only issue is AOT on Android.

Jack-Edwards commented 2 years ago

I am experiencing the same issue with several browsers on Android: Chrome, Firefox, and DuckDuckGo. Either the page is stuck in "Loading..." mode (Firefox and DuckDuckGo) or Chrome shows the "Aw, Snap!" error page. This only occurs when I enable AOT compilation. Everything works fine after disabling AOT compilation and republishing/redeploying.

legistek commented 2 years ago

Sorry for your trouble but glad to know it's not just me. Is anyone on the Blazor team able to look into this? Unfortunately it makes AOT unusable for us.

Norsotec commented 2 years ago

i need this to be fixed asap tbh. I bet on blazor and my app is running on >100 Android Devices, now without AOT. My client reported me already about the performance decrease. Hope to get any fix asap.

lewing commented 2 years ago

It sounds like the apps are hitting chrome android wasm memory limits when aot compiled. Since this is likely application specific we'd need a way to reproduce the error to even begin to understand why.

Jack-Edwards commented 2 years ago

My application is open source. Feel free to compile and run it. When I run the app locally using dotnet run, the ArrayBuffer memory consumption reported by desktop browsers is under 90 MiB.

legistek commented 2 years ago

chrome android wasm memory limits

Do we have any way of knowing what those are? Do they vary by device?

Since this is likely application specific we'd need a way to reproduce the error to even begin to understand why.

The "why" is the applications are complex. Since they work perfectly in interpreted mode and other devices this is plainly not an issue with our code. If we have to keep our app sizes low to avoid crashing, then we need some way of knowing what the size limits are.

legistek commented 2 years ago

Also just to be clear, whether it crashes is based solely on the size of the executables and not any memory allocation done by our applications. I know because my application's WASM never actually gets invoked (I have console.logs etc. to trigger when main() is executed).

lewing commented 2 years ago

can you please attach the output of chrome://version from a failing device

legistek commented 2 years ago

Of course. Hope this helps.

Google LLC
Copyright 2022 Google LLC. All rights reserved.
Google Chrome   100.0.4896.127 (Official Build) (32-bit) 
Revision    ff0d0695743e65305d7194f9bd309e5e1c824aa0-refs/branch-heads/4896_88@{#4}
OS  Android 12; Pixel 4 XL Build/SP2A.220305.012
Google Play services    SDK=220680000; Installed=221514044; Access=1p
JavaScript  V8 10.0.139.15
User Agent  Mozilla/5.0 (Linux; Android 12; Pixel 4 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Mobile Safari/537.36
Command Line    --use-mobile-user-agent --top-controls-show-threshold=0.5 --top-controls-hide-threshold=0.5 --use-mobile-user-agent --enable-viewport --validate-input-event-stream --enable-longpress-drag-selection --touch-selection-strategy=direction --disable-composited-antialiasing --enable-dom-distiller --flag-switches-begin --flag-switches-end --top-controls-show-threshold=0.5 --top-controls-hide-threshold=0.5
Executable Path No such file or directory
Profile Path    /data/user/0/com.android.chrome/app_chrome/Default
Variations  4f59692b-ca7d8d80
65ac254a-377be55a
5c56da73-ca7d8d80
e306c0f3-ca7d8d80
62f1d3ce-fd5dd07c
3a66a871-377be55a
4547826d-ea65074f
d382db9b-377be55a
61704cf4-5df7abeb
38213490-33c3eba5
112b64df-ca7d8d80
47722f0d-ca7d8d80
3e7d7783-ca7d8d80
7e184ca7-33c3eba5
f908d585-377be55a
b515b249-ca7d8d80
a112f012-6edc92c7
16b16054-377be55a
50d45193-ca7d8d80
59b6f412-377be55a
60d4b352-377be55a
3e8fe123-ca7d8d80
15e85316-ca7d8d80
8fb64725-12ede6a2
534138de-ca7d8d80
72233f8b-ca7d8d80
273268f8-935c8049
6cb5e962-377be55a
a77166f2-ca7d8d80
6e08fc3e-42aa9e1d
eddd0d82-107edff7
53879ff4-ca7d8d80
f3763f99-ca7d8d80
826f3281-ca7d8d80
87f33ad6-ca7d8d80
91c89351-377be55a
421c54-377be55a
3095aa95-3f4a17df
da05e4eb-3e0c486f
18ed1312-b89512ca
2f63bb4b-ee68549f
38262571-6edc92c7
c2fcf59c-f29a9da2
fa0fb40b-ca7d8d80
f26e9e11-ca7d8d80
7aa01420-6dea9697
e548d538-ca7d8d80
41e765a5-65c2de82
44c2e2c0-fc111c29
56fb8525-3f4a17df
84ef82df-ca7d8d80
f6c8ea70-ca7d8d80
65570806-ebfb497f
440a1764-377be55a
53fd5552-5f66ab04
dd8d67e-a9080253
363012bc-ca7d8d80
a582a1b8-ad75ce17
8d569531-62714bf0
a5b9261b-219af806
a37c6974-ca7d8d80
f2e6ae40-dfec247a
da2d8531-ca7d8d80
1f5e8c3a-cfdfb84f
62c71f3-d88bc9c1
6cd75d26-ca7d8d80
e4a357e9-a8ca353b
d9384a1d-ca7d8d80
2ba47366-ca7d8d80
248c3fbd-ca7d8d80
6ea69886-ca7d8d80
25ad6029-ca7d8d80
abf5aad6-5bd15e37
b4d9badf-40e2b871
8835ae81-e6e520d9
906834f5-9da5e2a0
8bf75a63-ca7d8d80
5e3a236d-4113a79e
8bccc03b-ca7d8d80
69d4ebd5-1f8c5973
eaae7a36-667038f7
72e4712b-33c3eba5
8500acb0-f0b68a85
94fbee8a-ca7d8d80
f7a3ec9-377be55a
470b3a8b-12ede6a2
e153f4cb-377be55a
f4187eec-ca7d8d80
ca5a2953-ca7d8d80
da493d3c-ca7d8d80
64a3b934-ca7d8d80
a284ff81-ca7d8d80
f677f007-73bd376b
47de1263-3d47f4f4
4ea303a6-ecbb250e
c5dfd685-ca7d8d80
89f54299-93fbae77
bfaf635f-ca7d8d80
fc7e4d22-a2d707c6
7d74eac0-ca7d8d80
597963a0-891c71fc
56bf6ea2-12ede6a2
9af490f6-ee63ae02
622d1a3c-ca7d8d80
62e55d5e-377be55a
5404a50f-28ad44a
22723ee1-ca7d8d80
2afb0ba9-ca7d8d80
3aa1e8b7-377be55a
931c5f72-5c0c1f00
494d8760-52325d43
7b2873c2-5f08a1ab
3ac60855-486e2a9c
f48c01d3-2a0e5f33
63dcb6a3-45641d72
e706e746-8282a32d
f296190c-eaceac83
4442aae2-7158671e
f690cf64-6e3b1976
ed1d377-e1cc0f14
75f0f0a0-4ad60575
710c3f90-6bdfffe7
e2b18481-cdc3d902
e7e71889-4ad60575
3a8271ac-12c226
2d3dfd19-377be55a
6979ab2b-ca7d8d80
248e3a0-377be55a
4ebfaf88-ca7d8d80
595f5eb0-ca7d8d80
bef5c006-ca7d8d80
1166396-ab02a1cf
4c973162-64762731
c2f68a06-ca7d8d80
c3859e82-93b196c8
636e2847-ca7d8d80
5e31bb48-ca7d8d80
c4c291c4-50cfd6a4

Jack-Edwards commented 2 years ago

I think @legistek is on the right track with large file sizes being the problem. As I recall when testing, Chrome reported there was only 10-12 MB worth of stuff in the browser cache after a crash occurred. dotnet.wasm is 53 MB uncompressed when using AOT compilation.

lewing commented 2 years ago

does the app work in edge on android?

legistek commented 2 years ago

does the app work in edge on android?

I didn't know there was such a thing. ;) I'll try it out. Like Jack I gave up on AOT awhile ago so will have to rebuild. It DID work on the Adblock browser on Android, which is not Chromium based.

lewing commented 2 years ago

does the app work in edge on android?

I didn't know there was such a thing. ;) I'll try it out. Like Jack I gave up on AOT awhile ago so will have to rebuild. It DID work on the Adblock browser on Android, which is not Chromium based.

edge is chromium based but there is an important difference that suspect is the cause of the problem here

legistek commented 2 years ago

AOT seems to work fine in Edge for Android. Man I forgot how much faster it was; would really be great if this could be sorted out.

Just for kicks I tried it again in Chrome Android and this time - for the first time - I was able to get the app running for awhile before I got "Aw Snap". But I did get "Aw Snap" eventually. I must be at the borderline of RAM usage?

lewing commented 2 years ago

The behavior you are experiencing is due to a 32bit build of Chrome and the resulting lack of address space. The issue and the comment here https://github.com/WebAssembly/design/issues/1397#issuecomment-963561649 describe the situation. Edge on android is a 64bit build so it doesn't have the same issue. We're discussing if there anything we can do on the runtime side to make the problem less acute.

legistek commented 2 years ago

Whoa, that issue comment you linked to seems to be saying WASM across the board is not suitable for mobile devices.

Why doesn't this happen in interpreted mode? And should we expect it will if the application gets complex enough?

Webreaper commented 2 years ago

Also hitting this - my .Net 7-preview-7 app runs fine on Android unless I enable AOT compilation, at which point it breaks (never gets to the point of executing any code in Program.cs in WASM).

This is running on Android 12, Chrome 105. It runs fine in Edge. According to the earlier comment, the issue is a lack of 64-bit Chrome, because that isn't available for devices with less than 8GB. I'm running this on an S21FE which has 8GB....

Looking forward to seeing what can be done to make this a) work and b) easier to debug.

legistek commented 1 year ago

Looking forward to seeing what can be done to make this a) work and b) easier to debug.

I imagine if Microsoft ever decided to port its Office apps to Blazor it would be addressed REAL fast. Until then there doesn't seem to be much attention being placed on mobile device compatibility.

sengiv commented 1 year ago

Problem with Chrome Android still persists in .NET 7.0.100-rc.1 preview. Based on this stat, it looks like if you use Blazor AoT, you will only abandon about 50% of users out there.

2022-09-30 08_36_29-Mobile browsers market shares _ world

Jack-Edwards commented 1 year ago

A pretty straightforward work-around is to host a separate, non-AOT compiled version of your application for mobile browsers. Feel free to use the Nginx configurations I am making available in the pull request linked above (or here). The redirect criteria were pulled from http://detectmobilebrowsers.com/

praschl commented 1 year ago

We are also experiencing this issue:

AOT works on PC with Chrome, Edge, Firefox
AOT works on Android with Edge, Firefox, Brave and at least one more I cant remember
AOT fails on Android with Chrome
AOT fails on iOS (all browsers, since all use the same engine)

Hosting a non-AOT version for mobile is no option. The performance gain is crucial even more for mobile devices.

pavelsavara commented 1 year ago

Once I added EmccTotalMemory to 128MB, it worked fine.

That would be probably hotfixed by https://github.com/dotnet/runtime/pull/80849 for next Net7 service release. In Net8 is even better fix for that. The property is now called EmccInitialHeapSize

My main project, however, crashes no matter what memory setting I used. (Well, I've tried 128MB and 256MB).

Large projects produce large AOT images. There are good improvements for AOT size by @vargaz in Net8 under "dedup" theme. https://github.com/dotnet/runtime/pulls?q=is%3Apr+author%3Avargaz+sort%3Aupdated-desc+is%3Aclosed+dedup

dotnet / runtime

Webassembly AOT Crashes Android Mobile (v96) #62149