Excessive memory usage

[alessaba]

Hi, I just wanted to let you know that this script, while it seems to be faster than Apple's implementation, used 80GB of RAM out of 32 i have installed, throwing more than half into swap, obviously. Is there a way to limit RAM usage to 75% of availbale memory, so it only rarely goes into swap (i believe macOS starts to use swap when 75% memory usage is reached)? Swapping slows down the process a lot too, so this would probably improve performance

[saagarjha]

Hmm…that's not good at all. The tool is designed to naturally throttle itself so as to not use too much memory, but so far I have only tested it on computers with fast disks and lots of cores so it's possible that I missed a bottleneck that you're now running into. What hardware are you using? Since you're using this tool I understand that you are familiar with developing for macOS/iOS to some extent, would you be willing to help me debug the performance problem by using the Allocations tool in Instruments against the process? Or perhaps a memgraph?

[alessaba]

I am using a Monterey 12.0 Hackintosh with a 4.8 GHz all core overclocked 8700k, 3400Mhz 32GB Ram, a NVMe SSD which does about 2GB/s

I will try later to do the debugging. Is a Leaks and Allocations a good combination of tests? Also, since i am not sure, should i first start the process from terminal, then hook up the Instruments tool, right?

Also, i thought it had finished because i saw disk and CPU usage drop, but it seems to be stuck and the process never ends...

[saagarjha]

Yeah, no, unxip should work well in that environment; I'm not sure what went wrong. For Instruments, you can just create a new document using the "Allocations" template, start the process, then attach to it. The memory usage shouldn't really cross more than a few gigabytes so once it does that just screenshot the window and send it over. Another thing you could do is compile the binary with an additional -g, run with MallocStackLogging=1, then run leaks -outputGraph $(pgrep unxip) when the memory issue occurs (and attach the file here).

[alessaba]

I tried recompiling with the latest Xcode beta (extracted with the normal method, alas) After about a minute it already was at about 30GBs of use. I did a short Allocations run. I stopped it almost as soon as it started swapping

It's difficult to upload the instuments log since 2 minutes did make a 1,5GB file :-/

[saagarjha]

You don't have to wait until it's swapping, it's broken as soon as it uses any more than a few gigabytes of RAM. From your graph I assume that should've happened just a few seconds in–can you take a screenshot of the entire window when that happens?

[alessaba]

The leaks instument alerts about a large leak, i did a screenshot of that,too If you need, i can send you a private icloud link to let you inspect the file for yourself. Few seconds of inspection made the file about 500mb

[saagarjha]

Sure, that would be helpful.

[alessaba]

if you give me your email address i'll invite you

is this your iCloud mail?: saagar@saagarjha.com

[saagarjha]

Yeah, that should work. Thanks!

[saagarjha]

Looking at the trace, the code appears to be leaking compression buffers for reasons that are not entirely clear. If you have disk space to spare and just want to be able to use unxip without its file compression feature, just put a "return false" at the top of the writeCompressedIfPossible function and see if that improves memory usage. I'll look into this more in the meantime and see what needs to be fixed.

[jshier]

I can't replicate this on 12.2 with Xcode 13.2.1 or 13.3. @alessaba If you're on 12.0, can you update to 12.2? It's possible this is a bug in the Swift concurrency runtime that was fixed as part of a later OS release. There have been a bunch, including fixes around AsyncStream lifetime management. I don't know if they separate runtime updates from Xcode updates, but it seems possible.

[alessaba]

That’s certainly possible. Since i have an Hackintosh (with Clover, even worse) it’s not as easy to update, but i am planning to uodate to either 12.2 or 12.3 beta soon. If something changes, I’ll let you know

[saagarjha]

Were you able to resolve this with a recent version of macOS and Swift?

[alessaba]

Sorry, i forgot i had the issue still open. Yes, updating to the release version of 12.3 solved the issue, now it's extremely fast, and while it still does consume about half the memory, it's not even close to reaching swap territory

sorry again for ieeping the issue open for longer than needed