search

dotnet / maui

.NET MAUI is the .NET Multi-platform App UI, a framework for building native device applications spanning mobile, tablet, and desktop.
https://dot.net/maui
MIT License
21.95k stars 1.7k forks source link

Memory leaks - After running 24 hours it crashes #17049

Open mgungorchamp opened 12 months ago

mgungorchamp commented 12 months ago

Description

I have a very simple APP - which shows web pages and images in turn. I am using MS VS Profiler and Objects and Heapisize gradually increases and at the end OOM - Out of Memory happens and the app crashes. I thought it was on my part, I was using Singleton, and I removed all singleton, no help. Second I was using a recursive call, I went to use while(true), but no help. I was using SemaphoreSlim, and removed it, no help. I took all the blame, but none was helping me.

I though it was Grid view, though Grid view causes a faster OOM crash, but removing it also did not help

The profiler does not show anything about my code doing wrong... This app should run all the time on screens, but it's not usable, I am planning to move from Net MAUI to Native Android Development. I was hoping to get one stone for 4 birds, but I have no bird at all.

I am not sure if anybody will hear my voice... I feel I wasted my months on this app.

Steps to Reproduce

1- Create an app with an Image and Webview on the content page - 2- Then just show images in rotation and websites, Wait ~24 hours on the Android Device will crash, but Windows will survive since there are vast amount of memory 3-Open profiler and observe Objects and Heapisize grow gradually until it can no longer get tolerated by the OS

Link to public reproduction project repository

https://github.com/mgungorchamp/InfoBoard/tree/Semaphore-NoSingletonNoRecursive/InfoBoard

Version with bug

7.0.92

Is this a regression from previous behavior?

No, this is something new, Not sure, did not test other versions

Last version that worked well

Unknown/Other

Affected platforms

Android, Windows

Affected platform versions

All Android

Did you find any workaround?

No workaround

Relevant log output

At the end you will see OOM message and crash.
[cr_ChildProcessConn] onServiceDisconnected (crash or killed by oom): pid=10851 bindings:W  S
akhanalcs commented 12 months ago

This is wild. I thought they fixed all memory leaks, but apparently not. I hope they look at this soon and get your app working.

kcrg commented 12 months ago

@affableashish I think the vast majority of memory leak issues have only been resolved in .NET 8 branches.

samhouts commented 12 months ago

Yeah, we've put most of the memory leak fixes in .NET 8 and not in .NET 7. Before you give up, you might want to test with the latest previews to see if it improves. If it doesn't, let us know! We're still working diligently on tracking these down. Thanks!!!!

ghost commented 12 months ago

Hi @mgungorchamp. We have added the "s/try-latest-version" label to this issue, which indicates that we'd like you to try and reproduce this issue on the latest available public version. This can happen because we think that this issue was fixed in a version that has just been released, or the information provided by you indicates that you might be working with an older version.

You can install the latest version by installing the latest Visual Studio (Preview) with the .NET MAUI workload installed. If the issue still persists, please let us know with any additional details and ideally a reproduction project provided through a GitHub repository.

This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

mgungorchamp commented 12 months ago

I will test with the new version of NET MAUI, for now I wanted to put this here. More information on this:

debug-console-crash-report.txt

ShiftyTR commented 11 months ago

My application crashes after running for about 4 to 5 hours. However, the console app, which uses the same library, works without any issues. I attempted to use .NET 8 to check for memory leaks, but in release mode, it gets stuck on loading. I started a new project from scratch and tried to build it in release mode, but it gets stuck during loading. Later, when I initiated a standard Maui project and installed the SignalR client, it keeps connecting and disconnecting continuously. It seems that SignalR has some issues when operating in release mode on .NET 8. At this point, I'm at a loss and have exhausted all potential solutions. I've decided to switch to Java and develop a native application. Unfortunately, this means that my 5 to 6 month-long project has gone to waste.

mgungorchamp commented 11 months ago

Yeah, we've put most of the memory leak fixes in .NET 8 and not in .NET 7. Before you give up, you might want to test with the latest previews to see if it improves. If it doesn't, let us know! We're still working diligently on tracking these down. Thanks!!!!

I am trying with the .NET 8. The profiler is not available in the PREV, and debugging to the Android connection gets lost somehow, therefore I do not see when it crashed and why. I opened with log cat.

I see these two messages... Seems NET 8 is not there yet. If I am doing something wrong I am open... but it looks I am exhausted the options. Time Device Name Type PID Tag Message 08-31 01:12:48.099 Amazon AFTKA Error 6411 libc Fatal signal 7 (SIGBUS), code 2 (BUS_ADRERR), fault addr 0xb0a45456 in tid 6411 (board.infoboard), pid 6411 (board.infoboard)

Time Device Name Type PID Tag Message 08-31 12:31:46.842 Amazon AFTKA Error 24328 libc Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 24719 (RenderThread), pid 24328 (board.infoboard)

And from debug [Choreographer] Skipped 298 frames! The application may be doing too much work on its main thread.

I am clueless ...

samhouts commented 11 months ago

@jonathanpeppers any recommendations?

jonathanpeppers commented 11 months ago

The app above (Semaphore-NoSingletonNoRecursive branch) runs for me using a local build of dotnet/maui/main:

image

@mgungorchamp what steps do I take to see the problem? Do I need to login?

This page probably has a lot of info on how to troubleshoot issues like this:

But if I can just run your app and see for myself, I'm happy to investigate. Thanks!

mgungorchamp commented 11 months ago

To get the image and URL to show you need to add them via the https://infopanel.vermontic.com/ It's free to create an account and add any web content or upload images then it will start showing them in an ordered fashion.
I have been here at https://github.com/dotnet/maui/wiki/Memory-Leaks and tried to apply most of the stuff, but I could not solve the problem...

mgungorchamp commented 11 months ago

Could you try this branch which has the latest NET 8 https://github.com/mgungorchamp/InfoBoard/tree/2023-08-31-IndPageByCodeNoGrid/InfoBoard It works but after some hours it crashes... You need to keep the app running... and it will stop sometimes while you are not paying attention.

I use FireTV Stick to test and also Chromecast... it works on tablets/PCs longer time since they have so much memory.

jonathanpeppers commented 11 months ago

Which do you think is the problem <WebView/> or <Image/>? I don't see how replacing the source on these controls would leak, is that basically all the app does? Can you reproduce this in a smaller sample with a single page?

One thought looking at this code: can you clear the WebView's source when it is not visible?

I keep getting this popup, so I'm not sure if I'm able to see the issue or not:

image

mgungorchamp commented 11 months ago

This is a caught exception, I could not find the solution for it, but seems harmless - due to socket communication - you can ignore it. I tested it on Image and it worked for several days. The image is ok. Webview with Grid faster crash. That's why I tried to tie page Content to webview to eliminate any middleman.

By clearing you mean this webView.Source = null; I did not but did not see the behavior change.

protected override void OnNavigatedFrom(NavigatedFromEventArgs args) { base.OnNavigatedFrom(args); webView.Source = null; }

For your question: Can you reproduce this in a smaller sample with a single page? I am testing just that case. In NET 8 PREV, the profiler is not available therefore I am unable to monitor the object count and heap size changes.

jonathanpeppers commented 11 months ago

In NET 8 PREV, the profiler is not available

What does this mean? Basically, every day I use dotnet-trace and dotnet-dsrouter to profile .NET 8 Android apps without issue.

For your question: Can you reproduce this in a smaller sample with a single page? I am testing just that case.

An app with only one page with one control, for example, would be great. Right now, there is a lot going on in your app, so it is difficult to pin down where the exact problem is.

mgungorchamp commented 11 months ago

VS 2022 has this option Debug / Windows / Show Diagnostic Tools But PREV does not have - I will try to use the tools you have mentioned image

I will try to do that, but testing could take more than a day since it crashes after a day. Currently, with image / and webview, it crashes less than a day.

jonathanpeppers commented 11 months ago

Should I be looking at this issue running on Android or Windows? I don't believe the Diagnostic Tools have ever worked for mobile, but we'd like them to one day.

mgungorchamp commented 11 months ago

Android!!! Please

jonathanpeppers commented 11 months ago

Many places you are converting JSON web responses to string, such as:

https://github.com/mgungorchamp/InfoBoard/blob/c0cacffa91721ea79eaf40d3dfe442160fe8e6c6/InfoBoard/Services/RestService.cs#L290-L291

This is creating a string in memory the size of every JSON response. Can you use a Stream directly to avoid this?

Example here:

Are you able to reproduce the issue without any HttpClient usage involved?

I have not seen the app crash yet on my Pixel 5, I took a diff several minutes apart and there are is only 90kb more memory used:

image

I suspect this is not showing a leak for me.

Do I need to setup many images on the site? (https://infopanel.vermontic.com/) Or will a single one show the issue?

jonathanpeppers commented 11 months ago

I have it cycling between two images now and the memory goes up and down:

image

Now, keep in mind, I am basically using .NET 8 (dotnet/maui/main), so maybe we have resolved the issue?

This one comes to mind, which actually fixed a leak in Image on all platforms:

mgungorchamp commented 11 months ago

I appreciate so much with your support Could you try it with webview too … I will follow your suggestions to convert to stream, not sure how much it will help.
and w/o HttpClient I didn’t test.

Crash occurs after hours…FYI

mgungorchamp commented 11 months ago

That code crashed less than 3 hours at FireTV ANDROID

jonathanpeppers commented 11 months ago

Can you get a detailed adb logcat output of the crash? The fact memory goes up and down for me, doesn't seem like I'm going to see the problem.

For example:

# enable more logging
adb shell setprop debug.mono.log default,debugger,assembly,mono_log_level=debug,mono_log_mask=all
# clear the log, just to make sure old message are not in there
adb logcat -c
# run the app until it crashes
# save the log
adb logcat -d > log.txt
mgungorchamp commented 11 months ago

Sure I will do that post it here

mgungorchamp commented 11 months ago

Here you go log-AmazonFireTV.txt

log-XiomiMI-TVStick.txt

log-AnotherAmazonFireTV.txt

Thank you so much for your attention, looking forward to hearing from you with possible solution/diagnosis

jonathanpeppers commented 11 months ago

Were these caused by a new simpler example? or your full application?

Is adb shell setprop debug.mono.log default,debugger,assembly,mono_log_level=debug,mono_log_mask=all enabled for all three? I don't see any extra log messages in here.

One of them crashed with SIGBUS:

08-31 22:20:43.623 24580 24580 F DEBUG   : backtrace:
08-31 22:20:43.623 24580 24580 F DEBUG   :     #00 pc 0005359a  /system/lib/libc.so (strcasecmp+6)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #01 pc 00168c71  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #02 pc 001d9ac9  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #03 pc 00169f69  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #04 pc 00169aa9  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_assembly_request_byname+324)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #05 pc 00169793  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_assembly_load_reference+306)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #06 pc 0016d455  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_class_from_typeref_checked+212)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #07 pc 00170367  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_class_get_checked+114)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #08 pc 001a57bb  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #09 pc 001727c3  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_field_resolve_type+498)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #10 pc 0017460b  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #11 pc 00174979  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_class_init_internal+348)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #12 pc 00138553  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #13 pc 0013e71b  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #14 pc 0013e203  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #15 pc 0012b40f  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #16 pc 0012225d  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #17 pc 00120801  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #18 pc 001b047f  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_runtime_invoke_checked+90)
08-31 22:20:43.623 24580 24580 F DEBUG   :     #19 pc 001b6695  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #20 pc 00185d83  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.623 24580 24580 F DEBUG   :     #21 pc 0018c6cb  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.624 24580 24580 F DEBUG   :     #22 pc 0012c1e9  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.624 24580 24580 F DEBUG   :     #23 pc 0012b613  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.624 24580 24580 F DEBUG   :     #24 pc 00121fe9  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.624 24580 24580 F DEBUG   :     #25 pc 0012023b  /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
08-31 22:20:43.624 24580 24580 F DEBUG   :     #26 pc 000013ba  <anonymous:ad821000>

I can maybe get a line number from this from the Mono runtime. Can you check what version you have? There will be a folder in C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Runtime.Mono.android-arm for example with the version number.

mgungorchamp commented 11 months ago

These are for full application I enabled all three of them - I don't know why extra log messages are not there. Here you go image

jonathanpeppers commented 11 months ago

Ok, the symbolicated crash is:

********** Crash dump: **********
Build fingerprint: 'Amazon/kara/kara:9/PS7646.3554N/0028085969408:user/amz-p,release-keys'
#00 0x0005359a /system/lib/libc.so (strcasecmp+6)
#01 0x00168c71 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_get_got_var
/__w/1/s/src/mono/mono/mini/method-to-ir.c:0:0
mono_method_to_ir
/__w/1/s/src/mono/mono/mini/method-to-ir.c:7291:5
#02 0x001d9ac9 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
emit_unsafe_intrinsics
/__w/1/s/src/mono/mono/mini/intrinsics.c:479:3
mini_emit_inst_for_method
/__w/1/s/src/mono/mono/mini/intrinsics.c:2009:10
#03 0x00169f69 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_method_to_ir
/__w/1/s/src/mono/mono/mini/method-to-ir.c:11210:6
#04 0x00169aa9 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_assembly_request_byname+324)
                                                                                                       emit_pop_lmf
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:1825:2
                                                                                                       mono_method_to_ir
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:8365:6
#05 0x00169793 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_assembly_load_reference+306)
                                                                                                       mono_method_to_ir
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:10033:7
#06 0x0016d455 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_class_from_typeref_checked+212)
                                                                                                       mono_method_to_ir
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:10141:7
#07 0x00170367 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_class_get_checked+114)
                                                                                                       initialize_array_data
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:5231:12
                                                                                                       mono_method_to_ir
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:10274:21
#08 0x001a57bb /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
add_generic_class_with_depth
/__w/1/s/src/mono/mono/mini/aot-compiler.c:5585:57
#09 0x001727c3 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_field_resolve_type+498)
                                                                                                       handle_gsharedvt_ldaddr
                                                                                                       /__w/1/s/src/mono/mono/mini/ir-emit.h:357:2
                                                                                                       mono_method_to_ir
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:6662:3
#10 0x0017460b /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
handle_gsharedvt_ldaddr
/__w/1/s/src/mono/mono/mini/ir-emit.h:0:2
emit_llvmonly_interp_entry
/__w/1/s/src/mono/mono/mini/method-to-ir.c:6027:4
#11 0x00174979 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_class_init_internal+348)
                                                                                                       emit_set_deopt_il_offset
                                                                                                       /__w/1/s/src/mono/mono/mini/method-to-ir.c:3599:2
#12 0x00138553 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_os_mutex_init_type
/__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:56:3
mono_os_mutex_init
/__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:81:2
sgen_thread_pool_start
/__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:287:2
#13 0x0013e71b /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
GlobalizationNative_LoadICU
/__w/1/s/src/native/libs/System.Globalization.Native/pal_icushim.c:481:5
#14 0x0013e203 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
FixupTimeZoneGenericDisplayName
/__w/1/s/src/native/libs/System.Globalization.Native/pal_timeZoneInfo.c:0:5
GlobalizationNative_GetTimeZoneDisplayName
/__w/1/s/src/native/libs/System.Globalization.Native/pal_timeZoneInfo.c:327:17
#15 0x0012b40f /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
sgen_ptr_in_nursery
/__w/1/s/src/mono/mono/mini/../sgen/sgen-gc.h:206:9
major_copy_or_mark_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:71:11
major_scan_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:74:3
drain_gray_stack_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack_par
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1344:10
#16 0x0012225d /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
sgen_vtable_get_descriptor
/__w/1/s/src/mono/mono/mini/../metadata/sgen-client-mono.h:29:33
major_copy_or_mark_object_concurrent_with_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:208:9
major_scan_vtype_concurrent_with_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:52:3
#17 0x00120801 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_copy_or_mark_object_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:122:11
major_scan_object_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:59:3
drain_gray_stack_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1335:10
#18 0x001b047f /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_runtime_invoke_checked+90)
                                                                                                       init_method
                                                                                                       /__w/1/s/src/mono/mono/mini/aot-runtime.c:4682:44
#19 0x001b6695 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
m_class_get_byval_arg
/__w/1/s/src/mono/mono/mini/../../mono/metadata/class-getters.h:71:1
decode_generic_inst
/__w/1/s/src/mono/mono/mini/aot-runtime.c:444:19
#20 0x00185d83 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_lldiv
/__w/1/s/src/mono/mono/mini/../../mono/utils/mono-error-internals.h:0:2
#21 0x0018c6cb /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_ssa_cprop
/__w/1/s/src/mono/mono/mini/ssa.c:0:4
#22 0x0012c1e9 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_scan_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:93:3
drain_gray_stack_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack_par
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1344:10
#23 0x0012b613 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
sgen_safe_object_is_small
/__w/1/s/src/mono/mono/mini/../sgen/sgen-gc.h:828:6
major_copy_or_mark_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:161:7
major_scan_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:59:3
drain_gray_stack_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack_par
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1344:10
#24 0x00121fe9 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_copy_or_mark_object_concurrent_with_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:158:48
major_scan_vtype_concurrent_with_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:74:3
#25 0x0012023b /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_copy_or_mark_object_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:192:4
major_scan_object_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:74:3
drain_gray_stack_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1335:10
#26 0x000013ba <anonymous:ad821000>
Crash dump is completed

********** Crash dump: **********
Build fingerprint: 'Amazon/kara/kara:9/PS7646.3554N/0028085969408:user/amz-p,release-keys'
#00 0x0001ce9e /system/lib/libc.so (abort+58)
#01 0x0001df95 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonodroid.so (xamarin::android::Helpers::abort_application()+4)
#02 0x00035377 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonodroid.so (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+66)
#03 0x001587d5 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_method_to_ir
/__w/1/s/src/mono/mono/mini/method-to-ir.c:6323:25
#04 0x0014f0fd /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_jit_runtime_invoke
/__w/1/s/src/mono/mono/mini/mini-runtime.c:3591:21
#05 0x0014f145 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_jit_runtime_invoke
/__w/1/s/src/mono/mono/mini/mini-runtime.c:0:3
#06 0x0014eebb /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mono_jit_runtime_invoke
/__w/1/s/src/mono/mono/mini/mini-runtime.c:0:7
#07 0x00110443 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
SGEN_LOAD_VTABLE_UNCHECKED
/__w/1/s/src/mono/mono/mini/../metadata/sgen-client-mono.h:23:14
sgen_safe_object_get_size
/__w/1/s/src/mono/mono/mini/../sgen/sgen-gc.h:812:20
scan_object_for_specific_ref
/__w/1/s/src/mono/mono/sgen/sgen-debug.c:735:17
scan_object_for_specific_ref_callback
/__w/1/s/src/mono/mono/sgen/sgen-debug.c:749:2
#08 0x00110e35 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
sgen_dump_occupied
/__w/1/s/src/mono/mono/sgen/sgen-debug.c:1030:2
#09 0x0012b863 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_copy_or_mark_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:0:3
major_scan_object_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:59:3
drain_gray_stack_par_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack_par
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1344:10
#10 0x001220ab /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_copy_or_mark_object_concurrent_with_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:0:0
major_scan_vtype_concurrent_with_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:74:3
#11 0x00120801 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
major_copy_or_mark_object_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:122:11
major_scan_object_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:59:3
drain_gray_stack_no_evacuation
/__w/1/s/src/mono/mono/mini/../sgen/sgen-marksweep-drain-gray-stack.h:347:3
drain_gray_stack
/__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1335:10
#12 0x001b047f /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so (mono_runtime_invoke_checked+90)
                                                                                                       init_method
                                                                                                       /__w/1/s/src/mono/mono/mini/aot-runtime.c:4682:44
#13 0x001c42b5 /data/app/com.guzelboard.infoboard-5I3FA4yjDrmAzVJ71FlmLg==/lib/arm/libmonosgen-2.0.so
mini_get_gsharedvt_wrapper
/__w/1/s/src/mono/mono/mini/mini-generic-sharing.c:2115:4
#14 0x00063775 /system/lib/libc.so (__pthread_start(void*)+22)
#15 0x0001e061 /system/lib/libc.so (__start_thread+24)
Crash dump is completed

Which looks to be around this line? (for the first crash, there are two above)

https://github.com/dotnet/runtime/blob/45161fb0b325fdf02d7e9a2a794b0780b50dc173/src/mono/mono/mini/method-to-ir.c#L7291

Is this crash not really about memory leaks at all, but perhaps a bug in the runtime when running on arm for a very long time? Do you see the same issue on an arm64 device?

We can send this issue to the dotnet/runtime team to look at this, but I think they will need a smaller example to solve it? Have you made it any further to get this to happen with a single page & single control?

mgungorchamp commented 11 months ago

The devices I tested are: The Chromecast with Google TV has 2GB of RAM and 8GB of storage and is powered by an Amlogic S905X3 CPU based on an Arm Cortex-A55

Fire TV Family ARM Cortex-A53

Seems I don't have an arm64 device to test.

jonathanpeppers commented 11 months ago

So, I think we're we are at, is to try to make a smaller sample project that has the same problem.

Unfortunately, I do not have an arm device, only arm64 and x64 devices.

ghost commented 11 months ago

Hi @mgungorchamp. We have added the "s/needs-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

mgungorchamp commented 11 months ago

Thank you for your diagnosis. I am not sure what other surprises waiting for me down the road. I spent so much time on this. I am moving to Android Studio and Java. Thanks for the help.

akhanalcs commented 11 months ago

C'mon @mgungorchamp, don't give up on MAUI just yet.

mgungorchamp commented 11 months ago

Thank you for the kind word @affableashish but I am not sure when they are going to fix this memory leak - no timeline - this app should work days without restart... I don't want to move away after this many months of learning investment...

akhanalcs commented 11 months ago

I completely understand you @mgungorchamp. Maybe @jonathanpeppers will provide some further guidance here.

In the meantime, can you implement a workaround? The workaround I can think of is: Write a very simple console app or worker service that runs 24hrs in the background that will restart your MAUI app everyday at midnight or something. This way your app will run forever with periodic restarts.

After MAUI team puts a reliable solution, you can get rid of that worker service. I can probably get you an example of how to do this for windows apps tomorrow because I've done this kind of thing before. Haven't done this for android apps but there's probably a way.

Also please reopen this issue. That way, this issue can make it to MAUI team's backlog and will get worked on (hopefully soon).

mgungorchamp commented 11 months ago

Thanks again @affableashish for the message. But this app will be distributed via the app store, ordinary users cannot do this separate procedure. If there is a way to implement that feature, like when the app crashes, it restarts itself! I don't think this is logically possible, since the app will not be aware of its death. I have started a parallel project, but if this gets solved sooner, I will go back to MAUI before I go so far with Android Studio.

akhanalcs commented 11 months ago

Not sure how much it's worth it but you can create a simple watchdog application and deploy it alongside your app. I asked ChatGPT so it might not be all correct, but it's a start.

References: https://medium.com/@deepakgahlot98/a-never-ending-service-android-263ec5d5dc3b

ChatGPT answer:

Step 1: Create a Watchdog Android App

Define Permissions: In your AndroidManifest.xml, define the necessary permissions. Since your watchdog app needs to restart other apps, you should include the RECEIVE_BOOT_COMPLETED permission for automatic startup.

<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />

Create a Service: Create a background service that will periodically check and restart your target app.

public class WatchdogService extends Service {

    private static final long RESTART_INTERVAL = 24 * 60 * 60 * 1000; // 24 hours in milliseconds

    @Override
    public int onStartCommand(Intent intent, int flags, int startId) {
        // Create a timer to periodically restart the target app
        Timer timer = new Timer();
        timer.schedule(new TimerTask() {
            @Override
            public void run() {
                restartTargetApp();
            }
        }, 0, RESTART_INTERVAL);

        return START_STICKY;
    }

    private void restartTargetApp() {
        // Code to restart your target app (the one with the defect)
        try {
            Intent launchIntent = getPackageManager().getLaunchIntentForPackage("com.example.targetapp");
            if (launchIntent != null) {
                startActivity(launchIntent);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Nullable
    @Override
    public IBinder onBind(Intent intent) {
        return null;
    }
}

Add Receiver in Manifest: Register your service to receive the BOOT_COMPLETED broadcast, so it starts automatically when the device boots up.

<receiver
    android:name=".BootReceiver"
    android:enabled="true"
    android:exported="true">
    <intent-filter>
        <action android:name="android.intent.action.BOOT_COMPLETED" />
    </intent-filter>
</receiver>

Create a BootReceiver: Create a broadcast receiver to start your watchdog service when the device boots up.

public class BootReceiver extends BroadcastReceiver {
    @Override
    public void onReceive(Context context, Intent intent) {
        if (intent.getAction() != null && intent.getAction().equals(Intent.ACTION_BOOT_COMPLETED)) {
            Intent serviceIntent = new Intent(context, WatchdogService.class);
            context.startService(serviceIntent);
        }
    }
}

Step 2: Deploy Your Watchdog App

Build Your Watchdog App: Build your watchdog app using Android Studio. Make sure you have the APK file ready.

Install the Watchdog App: You can install the watchdog app on your device using ADB or by uploading it to a file-sharing service and downloading it on your device. Alternatively, you can publish it on the Google Play Store if necessary.

Step 3: Configure Your Watchdog App

Configure Package Name: In the restartTargetApp method of your WatchdogService, replace "com.example.targetapp" with the package name of your crashing app.

Set Restart Interval: Adjust the RESTART_INTERVAL to your preferred duration for restarting the target app.

Step 4: Run Your Watchdog App Start your watchdog app. It will automatically start monitoring and periodically restart your target app.

Ensure that your target app is also installed on the same device.

akhanalcs commented 11 months ago

I agree it's one hell of a workaround and would be nice when MAUI/ runtime team fixes this.

jonathanpeppers commented 11 months ago

Maybe just to recap where we are on this issue:

  1. I did not observe any memory leaks using dotnet/maui/main. Maybe @mgungorchamp can comment if .NET8 fares any better?

  2. I did see @mgungorchamp's logs, where the Mono runtime appeared to crash on arm (32-bit) devices in .NET 7. It feels like it could be a runtime issue.

  3. I'm not able to observe the app crashing as @mgungorchamp is. However, the best I have are older Pixel devices that are arm64.

To move further on this, there are two paths @mgungorchamp can take to help us investigate further:

  1. Try .NET 8.
  2. Try to create a smaller sample with the same issue -- @mgungorchamp basically sent their entire application where you have to navigate a web application to create an account, upload photos, to even use the app.

If we had a smaller repro, it would help me investigate any memory leaks (as there should be objects less in memory) -- and we would have something the runtime team would have an easier time investigating. Thanks!

ghost commented 11 months ago

Hi @mgungorchamp. We have added the "s/needs-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

akhanalcs commented 11 months ago

@mgungorchamp Can you try the 2 paths that Jonathan recommended? I'm quite interested in this issue.

akhanalcs commented 11 months ago

@mgungorchamp Any update on this?

mgungorchamp commented 11 months ago

I paused the development... thank you for the follow-up, though!
Update 1- .NET 8. did not solve my problem - it crashed after several hours 2- Smaller app size - I need to find time to implement and test -

Zhanglirong-Winnie commented 6 months ago

Verified this issue with Visual Studio Enterprise 17.10.0 Preview 1. Can repro on Windows platform with sample project. Memory leak causes VS to crash. https://github.com/mgungorchamp/InfoBoard/tree/Semaphore-NoSingletonNoRecursive/InfoBoard