NSCache missing auto-eviction policies

[triplef]

The current NSCache implementation will not evict entries unless:

• countLimit > 0 is set
• or totalCostLimit > 0 is set and a cost is assigned to objects in the cache (i.e. setObject:forKey:cost: is used with a cost parameter > 0)

Otherwise (i.e. if just setObject:forKey: is used) the cache will grow indefinitely.

In contrast, the reference platform implementation will auto-evict items in the cache based on system memory pressure:

The NSCache class incorporates various auto-eviction policies, which ensure that a cache doesn’t use too much of the system’s memory. If memory is needed by other applications, these policies remove some items from the cache, minimizing its memory footprint.

https://developer.apple.com/reference/foundation/nscache?language=objc#overview

[rajsesh]

@triplef on Windows, the resource manager will kick in under low memory conditions. This may not match 1:1 with what Mac or iOS does (nor would it make sense to do that). What we should really be doing in NSCache (among other places in the code where we can discard loaded pages) is to handle this notification and evict some entries.

A workaround for now would be for the app to implement applicationDidReceiveMemoryWarning in the app delegate and set the NSCache limits to some low value and set it right back to 0. Not ideal, but should unblock you in the interim.

[triplef]

Thanks @rajsesh-msft, we’ll give that workaround a try.

[DHowett-MSFT]

Plan

NSCache is specified as automatically evicting objects when there is memory pressure. Windows offers a high-level overview of application memory usage through Windows::System::MemoryManager.

NSCache can subscribe to the AppMemoryUsageIncreased and AppMemoryUsageDecreased events and implement a final line of defense against cache bloat.

Open Questions

I'm not sure what the best eviction approach is; perhaps there can be a global maximum object count across all caches? It'll be effectively infinite when the application's memory usage is Low, medium when it is Medium and low when it is High.

If we assume that 100,000 objects across all caches is generous, we can evict objects from caches opportunistically (based on least-recent-use) until we come below that limit on a memory change event. Which caches get hit?

How does size factor into this? Object "size" is an ill-defined notion--it has no bearing on the true size in bytes of memory the object occupies--so it cannot be used as a good eviction heuristic under memory pressure. Still, though, "expensive" items are expensive regardless of what that expense truly means.

[DHowett-MSFT]

If we hit the High usage level, we could simply evict all objects from all caches. That's a very heavy-handed approach.

[DHowett-MSFT]

@triplef

It looks like the Windows Resource Manager isn't engaged for applications on Desktop, even those subject to the 32-bit 2GB address space limit (!). That complicates things significantly for NSCache.

The official recommendation we've received is (paraphrased):

Let the application go to swap. The system will swap you as necessary. If you really want to do this, you'll have to poll for the working set on your own and void your caches when you believe it to be necessary. Remember that other applications will have a different view of "free memory" than you do, and might expand their use to consume the recently-freed memory.

We had a few discussions about adding this functionality to WinObjC, but didn't come up with a good conclusion. There's no place in Foundation that would truly "fit" a memory polling API, and we would need to expose knobs for frequency and limits (what's the absolute maximum, where do we make the "high" cutoff, "medium", ...). Additionally, any solution like that would need to interoperate with the Resource Manager on constrained platforms (phone) so that it doesn't make conflicting or overriding decisions.

That brings me to my recommendations! It's not the best solution, but you could do the following:

• Keep track of every NSCache you allocate
• Poll the application's working set through supported win32 store-partition APIs
• Call removeAllObjects explicitly on caches until you cross below the limits you set.

It may also be worthwhile to investigate the /LARGEADDRESSAWARE linker switch. It won't help the caching issue, but it will give you more headroom (up to the whole 4GB available to a 32-bit process) before you fail due to address space exhaustion!

WOCCatalog with /LARGEADDRESSAWARE

[DHowett-MSFT]

t;ldr this will have to be a WinObjC caveat :frowning:. Closing

[triplef]

Understood, thanks for looking into this @DHowett-MSFT!