Closed pekspro closed 7 months ago
Looks similar to #3119 thought 7GB is a lot.
/cc @vitek-karas
Might be worth trying to pass /maxcpucount:1
to dotnet build as a workaround to make sure it only tries to trim one app at a time.
Thanks @akoeplinger, that is an interesting idea. I'm trying that right now.
/maxcpucount:1
didn't help. It run for about 90 minutes and then it run out of memory.
If it ran for 90 minutes "in the linker" (or anywhere in msbuild) then this is definitely a bug. I'll look into this tomorrow (sorry, too late today).
I tried /maxcpucount:1
in .NET 6 as well. Took about 20 minutes in Github Actions. Without this parameter, I think it is about 15 minutes.
Tried it locally - it is really bad in 7.0:
Publishing the Android project spins up 4 linkers each of which takes 25 minutes to run on my machine (which is reasonably fast, so this is really bad) and they each consume 7-10 GB of memory (it oscillates in this range, so my guess is that it needs around 7GB, the rest is GC doing its thing).
When I reran only one of the illink invocations it is a bit faster - around 17 minutes, but still really slow. And the memory consumption was as high as 11 GB (but there was less memory pressure on the system, so GC didn't need to work as hard).
Using latest linker from main (8.0) it is much better - I reran one of the illink invocations and it took only 1 minute and used a little bit over 4 GB of memory. So it might still not work in 7GB limited environment if running 4 at once, but it's definitely a LOT better.
This should be fixed by https://github.com/dotnet/linker/pull/3094 once it's merged into 7.0 and shipped. I tried locally with a build from https://github.com/dotnet/linker/pull/3094 and it took less than 1 minute and consumed max 4.5 GB of memory.
Using latest linker from main (8.0) it is much better - I reran one of the illink invocations and it took only 1 minute and used a little bit over 4 GB of memory.
That's still quite a lot, any pointers into what is taking that much?
I havened looked at the memory consumption details. I'll leave that to @jtschuster (I'll send you the simple repro offline).
When running the repro on my machine, the process only ever got up to about 3 GB, but I was using a different version of the Android sdk.
It looks like Cecil Instructions, the MarkStep._methods queue, and ParameterDefinitions take up the most space in our heap.
Type | Size (mb) |
---|---|
Instructions | 300 |
_methods (Queue<ValueTuple<MethodDefinition, DependencyInfo, MessageOrigin>>) | 110 |
ParameterDefinition | 100 |
MethodDefinition | 80 |
GenericInstanceType | 60 |
MethodReturnType | 50 |
ParameterDefinitionCollection | 44 |
MethodReference | 42 |
It looks like Cecil Instructions, the MarkStep._methods queue, and ParameterDefinitions take up the most space in our heap.
It could be also useful to check if we could "free" some memory earlier.
Another idea is to process the methods in different order. If you imagine how typical programs work:
Each level is bigger in size (number of different methods) than the one above it, so it looks like a tree. There will be direct calls to low-level methods spread around there as well, but they're not the ones which hurt. Since we use a simple queue, this will effectively do a breath first walk of the tree - where it basically keeps the entire lower level of the tree in the queue before it gets to it (probably more than just one actually).
Another way to look at it:
Our current algorithm effectively prioritizes processing high-level methods over the low-level ones (see the tree description above). This makes the queue long.
I prototyped this real quick by changing the Queue
to Stack
(we could probably do better than that still) and for hello world the max size of the collection were:
Stack
and #3139 - 805I would expect the effect to get bigger for larger apps.
Another thing to check would be if we keep MarkStep
around after it's done. In theory the driver should free steps which are processed, but it's a complex system, so hard to tell.
- Current main - 3735
- With Call CheckProcessed before enqueuing to MarkStep._methods #3139 - 833
- With
Stack
and Call CheckProcessed before enqueuing to MarkStep._methods #3139 - 805I would expect the effect to get bigger for larger apps.
Unfortunately, with #3139, _methods is about the same size as a queue or stack in the repro for this issue (~14mb).
I had seen this issue with my app NewsReader when I was using MAUI 7 (.NET 7).
After upgrading to MAUI 8 (.NET 8) I have re-checked this issue again - now the GitHub Actions for Android in Release mode are working again.
Note: It is still much slower than the build for iOS or Windows (WinUI 3).
I can confirm this is no longer an issue in .NET 8. Closing this.
If I’m trying to publish a .NET 7 MAUI application, targeting Android, with trimming enabled on GitHub actions, it runs out of memory:
I’m using the default Windows runner that has 7 GB of memory: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
With trimming disabled, it works fine. I could publish with trimming enabled on my local computer that has a lot more memory.
The source code for the application is available here: https://github.com/pekspro/RadioStorm
I also have an older version of the application targeting .NET 6. I have been able to trim this in GitHub without running out of memory.
It’s not a trivial app, but also not a very large one, I think. I just wanted to report this. I do not expect any solution for this :-)