SebastianKleu commented 1 month ago

Android framework version

net8.0-android, net9.0-android

Affected platform version

.NET 8 & .NET 9

Description

Hi,

We have been running our .Net Maui Android app on .Net 7 for some time now and upgraded to .Net 8.

After the upgrade, we realized that the app crashes after a few hours of use.

I have narrowed down the culprit to EF call on Sqllite. I am able to induce this problem by just reading from a table. See below:

private async Task TesterLoop()
    {
        while (true)
        {
            var testDataMessages = await GetTestDataMessages(new CancellationToken());
        }
    }

private async Task<List<TestData>> GetTestDataMessages(CancellationToken cancellationToken)
    {
    await using var dbContext = await _dbContextFactory.CreateDbContextAsync(cancellationToken);

        List<TestData> returnResult = await dbContext.TestData.AsNoTracking()
            .ToListAsync(cancellationToken);

        await dbContext.DisposeAsync().ConfigureAwait(false);

        return returnResult;
 }

Please see attached zip of test application where you can clearly see the memory rise in .Net 8 and not using .Net 7.

See below memory usage of .Net 7 app after 10 min.

See below memory usage of .Net 8 app after 10 min.

See below test application.

TestMaui.zip

Steps to Reproduce

Run Test.Maui.7 on Android device
Notice memory stays constant
Run Test.Maui.8 on Android device
Notice native memory climb fast

Did you find any workaround?

No

Relevant log output

No response

SebastianKleu commented 1 month ago

Just an update. I was able to narrow down even more by completely removing the Maui UI from the test applications. With the Maui UI removed, the memory still climbs rapidly for .Net 8 Android projects. See updated test app: TestMaui.zip

See results below: Running Test.Android.7 for 3 min

Running Test.Android.8 for 3 min

roji commented 1 month ago

Note https://github.com/dotnet/efcore/issues/34836 which tracks this on the EF side. As the problematic memory behavior appears when upgrading from .NET 7 to 8 - without any EF changes - we're assuming this is a non-EF problem for now... But please let me know if help is needed from the EF side!

julealgon commented 1 month ago

@roji any chance of this being related to that other problem with async TransactionScopes?

roji commented 1 month ago

@julealgon #50683 indeed seems like a duplicate of this. However, as with many "GC isn't working" issues that users open, I can't see any evidence of a leak - the complaint is that memory usage is growing, not that there's an actual OutOfMemoryException. So it may very well be that TransactionScopeAsyncFlowOption.Enabled causes increased memory usage, but that by no means implies a leak - the GC may simply not be reclaiming the memory yet (but would as memory usage approaches the limit). This is a point we keep need to explain to users.

dotnet-policy-service[bot] commented 1 month ago

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] commented 1 month ago

Tagging subscribers to this area: @roji, @ajcvickers See info in area-owners.md if you want to be subscribed.

roji commented 1 month ago

@akoeplinger @jeffschwMSFT who's the right person to look at this? This is currently labeled as System.Data; the repro uses TransactionScope (which is System.Transactions rather than System.Data), but in any case at this point it seems like there's a memory-related issue on android only, which is why this was in the dotnet/android repo.

vitek-karas commented 1 month ago

@BrzVlad for ideas

BrzVlad commented 1 month ago

@SebastianKleu @roji Could you quickly port this to a console application? It is fine if it not going to leak, since it would be using CoreCLR, but I could test it with mono and it should be easier to investigate.

SebastianKleu commented 1 month ago

TestMaui.zip @BrzVlad - Please see updated test application. It now includes the following projects: Test.Maui.7 -> DotNet 7 Maui app (no leak) Test.Maui.8 -> DotNet 8 Maui app (leak detected) Test.Android.7 -> DotNet 7 Android app (no leak) Test.Android.8 -> DotNet 8 Android app (leak detected) Test.Console.8 -> DotNet 8 Console app (no leak)

Please let me know if there's anything else I can provide to assist with the investigation.

BrzVlad commented 1 month ago

@SebastianKleu Seems like you uploaded the wrong archive since it doesn't contain the Console project.

SebastianKleu commented 1 month ago

@BrzVlad - Sorry about that. Seems I have reached the size limit of .zip folders for Github.

See link with updated test applications here: https://github.com/SebastianKleu/TestMaui

markples commented 1 month ago

I've been looking at TransactionScope (and ConditionalWeakTable and finalizers) and have at least a partial explanation. I'll try to write it up later today.

markples commented 1 month ago

50683 contains my notes for ConditionalWeakTable (CWT). I don't understand this code enough to say whether CWT is a complete, partial, or irrelevant explanation for this case.

There are two aspects of CWT that may be relevant.

The internal use of finalization can lead to very delayed collection of internal objects used by CWT - basically needing gen2 collections to make progress, so anything that the apps does to trigger gen2s (via allocation or manually) helps mitigate it.
The finalization queue can get overloaded (entries added faster than they can be processed). For CWT this is probably only going to show up in microbenchmarks or if many threads are generating finalizable objects. Otherwise, it doesn't take much time where the app is doing non-finalization-creating work for the queue to drain.

BrzVlad commented 1 month ago

@SebastianKleu Note that I haven't been able to reproduce the leak on desktop so this issue would appear to be specific to android.

SebastianKleu commented 1 month ago

@BrzVlad - Yes, during my testing as well, it seems to only happen on Android devices.

SebastianKleu commented 3 weeks ago

Hi guys,

Is there anything we can do in the meantime to get around this problem? We really want to upgrade to the LTS version in .Net 8 but this issue is preventing us from doing so.

I would appreciate if there is some workarounds we can implement?

SebastianKleu commented 1 week ago

Hi guys,

Any update on this?

BrzVlad commented 4 days ago

Unfortunately I cannot reproduce this issue. I cloned https://github.com/SebastianKleu/TestMaui. Test.Android.8 doesn't leak for me (I included in the manifset <profileable android:shell="true" />, then built the app, opened the apk in android studio and used View Live Telemetry profiler option. App stayed stable at around 150 MB used memory). Test.Maui.8 crashes shortly after startup with MonoDroid: Java.Lang.RuntimeException: Context.startForegroundService() did not then call Service.startForeground(): ServiceRecord{e652e1f u0 com.test.maui.x8/com.maui.eight c:com.test.maui.x8} which seems like a problem with the sample.

I have 8.0.303 installed and was running on a Pixel 7a with android 14.

GendacAI commented 3 days ago

I am working with @SebastianKleu on the project. We are able to reproduce the issue on an Android 10 device. Android 12 and 13 seems to not have the issue. Running the app on an emulator also does not produce the memory issues.

The information for the device we are getting the issue looks as follow:

System Information:

CPU Information:

OS Information:

dotnet / runtime

Memory leak in .Net 8 using Entity Framework and Sqllite (.Net Android) #108762

Android framework version

Affected platform version

Description

Steps to Reproduce

Did you find any workaround?

Relevant log output

50683 contains my notes for ConditionalWeakTable (CWT). I don't understand this code enough to say whether CWT is a complete, partial, or irrelevant explanation for this case.