[SDMemoryCache removeAllObjects] crash

[Robot2037]

New Issue Checklist

• [x] I have read and understood the CONTRIBUTING guide
• [x] I have read the Documentation
• [x] I have searched for a similar issue in the project and found none

Issue Info

Info	Value
Platform Name	iOS
Platform Version	11.4, 11.4.1, 12.0.0, 12.0.1
SDWebImage Version	e.g. 4.2.0 / 4.1.0
Integration Method	cocoapods
Xcode Version	Xcode 10.0
Repro rate	sometimes
Repro with our demo prj
Demo project link

Issue Description and Steps

I'm getting following error while application tries to clean memory cache after entering background state. This crash started to appear since SDWebImage 4.4.2, but doesn't happen every time.

-(void) applicationDidEnterBackground:(UIApplication *)application {
    [SDWebImageManager.sharedManager.imageCache clearMemory];
}

Here is crash stacktrace:

Crashed: com.apple.main-thread
0  libobjc.A.dylib                0x182bfc910 objc_msgSend + 16
1  Foundation                     0x18444f6b8 empty + 72
2  Foundation                     0x1843a1db8 -[NSConcreteMapTable removeAllItems] + 64
3  MyApp                          0x1005ce384 -[SDMemoryCache removeAllObjects] (SDImageCache.m:130)
4  MyApp                          0x1005d12c4 -[SDImageCache clearMemory] (SDImageCache.m:619)
5  MyApp                          0x1004de84c -[AppDelegate applicationDidEnterBackground:] (AppDelegate.m:269)
6  UIKit                          0x18da21984 __47-[UIApplication _applicationDidEnterBackground]_block_invoke + 184
7  UIKit                          0x18d6a89a0 +[UIViewController _performWithoutDeferringTransitions:] + 112
8  UIKit                          0x18da2185c -[UIApplication _applicationDidEnterBackground] + 96
9  UIKit                          0x18dc3b17c -[__UICanvasLifecycleMonitor_Compatability deactivateEventsOnly:withContext:forceExit:completion:] + 1000
10 UIKit                          0x18e29e89c __82-[_UIApplicationCanvas _transitionLifecycleStateWithTransitionContext:completion:]_block_invoke + 376
11 UIKit                          0x18d6081ec -[_UIApplicationCanvas _transitionLifecycleStateWithTransitionContext:completion:] + 432
12 UIKit                          0x18e083ac8 __125-[_UICanvasLifecycleSettingsDiffAction performActionsForCanvas:withUpdatedScene:settingsDiff:fromSettings:transitionContext:]_block_invoke + 220
13 UIKit                          0x18e1d1bf8 _performActionsWithDelayForTransitionContext + 112
14 UIKit                          0x18d607c0c -[_UICanvasLifecycleSettingsDiffAction performActionsForCanvas:withUpdatedScene:settingsDiff:fromSettings:transitionContext:] + 248
15 UIKit                          0x18d6075a8 -[_UICanvas scene:didUpdateWithDiff:transitionContext:completion:] + 368
16 UIKit                          0x18d645334 -[UIApplicationSceneClientAgent scene:handleEvent:withCompletion:] + 468
17 FrontBoardServices             0x186230f24 __80-[FBSSceneImpl updater:didUpdateSettings:withDiff:transitionContext:completion:]_block_invoke.362 + 212
18 libdispatch.dylib              0x183320a60 _dispatch_client_callout + 16
19 libdispatch.dylib              0x183328170 _dispatch_block_invoke_direct$VARIANT$mp + 224
20 FrontBoardServices             0x186264878 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 36
21 FrontBoardServices             0x18626451c -[FBSSerialQueue _performNext] + 404
22 FrontBoardServices             0x186264ab8 -[FBSSerialQueue _performNextFromRunLoopSource] + 56
23 CoreFoundation                 0x1839d7404 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
24 CoreFoundation                 0x1839d6c2c __CFRunLoopDoSources0 + 276
25 CoreFoundation                 0x1839d479c __CFRunLoopRun + 1204
26 CoreFoundation                 0x1838f4da8 CFRunLoopRunSpecific + 552
27 GraphicsServices               0x1858da020 GSEventRunModal + 100
28 UIKit                          0x18d914758 UIApplicationMain + 236
29 MyApp                          0x1004f6c3c main (main.m:14)
30 libdyld.dylib                  0x183385fc0 start + 4

[zhongwuzw]

@Robot2037 Seems crashed when map empty keys and values, can you reproduce in your development environment? if you can, please enable Zombie in scheme to catch released object.

And for temporary fix, you can disable the weak memory cache by [SDWebImageManager sharedManager].imageCache.config.shouldUseWeakMemoryCache = NO;.

By the way, you don't need to call clearMemory by yourself when enter to background. imageCache uses NSCache to store image, it would clear when enter background automatically.

[Robot2037]

@zhongwuzw There are 32 crashes reported during last 4 days in Crashlytics, which started to appear after new app release, which uses SDWebImage 4.4.2. However I'm not able to reproduce issue myself in Xcode.

Thanks for the info, I will remove clearMemory call when entering background, since is not needed. Anyway I call this function also in didReceiveMemoryWarning, which could cause crash as well.

[zhongwuzw]

@Robot2037 Emm, SD also handles the situation of memory warning. Theoretically even though you don't call it by yourself, the crash would also appear, because we call it in SD. And before we can find the issues, I suggest you to use [SDWebImageManager sharedManager].imageCache.config.shouldUseWeakMemoryCache = NO; to disable weak memory, and only use NSCache to manage images in memory.

Could you mind post some integrity symbolic crash logs?

[Robot2037]

@zhongwuzw Ok, I din't even know that. It should be stated somewhere in the documentation that clearMemory function is called automatically and you don't need to call it yourself.

I uploaded crash logs here.

[dreampiggy]

It seems the weak NSMapTable<NSString, id> cause the issue ? Strange...

I'm sure I place a lock between each read/write to that NSMapTable, but this still cause issue ?

Did you try to trigger a set method (-[SDImageCache setConfig:]) to the SDImageCache.config during runtime at different thread ? This API is actually not correct in 4.x. It's a readonly property, but however we can not change any public header because we follow semversion. In 5.x we fixed it...

[Robot2037]

I don't set any configuration for SD, I just use it as it is.

[dreampiggy]

@Robot2037 What previous version of SDWebImage did you use ? I remember this weak memory cache feature is available since SDWebImage 4.3.1 via #2228. Did you use any version between 4.3.1-4.4.2 version ?

For temporary workaround, try disable weak cache using shouldUseWeakMemoryCache = NO. I'll try to find out the reason and provide a better solution.

[Robot2037]

@dreampiggy Hard to say, I don't force any version in Cocoapods, so I always use the last available version. I just know date when I made release build and uploaded it to iTunes. Every time before making release build, I run pod update command to update all libraries.

There are dates for three last app builds:

app version	build date
2.4.5	2018-10-01
2.4.4	2018-07-20
2.4.3	2018-05-18

Regarding Relases page, SD 4.4.2 was released 2018-07-18, that means app version 2.4.4 and 2.4.5 uses SD 4.4.2. This crash started to appear 5 days ago for both versions. Strange there were no crashes until then and also there is no crash in our other apps using the same SD version. There is screenshot from Crashlytics.

[zhongwuzw]

@Robot2037 After some debug, I can confirm now, the crash happens when NSMapTable empty the keys, in SD, keys is NSString of the image url, when we remove the objects, runtime calls objc_msgSend, the selector is release to release the keys, in objc_msgSend, when try to get isa of the parameter self, then crash, because key already released.

The crash procedure just like above I describe. But the actual reason why it would be released in advance is unknown. In theory, we use strong to store the key, would not be released before NSMapTable release it.

@dreampiggy , I have a thought to try to fix it, before we call - setObject:forKey: of NSMapTable, we do a mutableCopy of key if key is NSString, let NSMapTable to manage the memory of key. I don't use NSPointerFunctionsCopyIn because it would not always do copy. What's your opinion? 🤔

[Robot2037]

Any news on this one? It is very strange there are no more crashes since I reported it here. It suddenly appeared on 15th October and lasted until 18th October, caused 36 crashes in total, and then disappeared. No more reports for last week and also there were no reports before 15th October. At first I thought it was related to new app release, but later I realized even the previous app version was using same SD version. This issue probably only happens during very specific application state or situation.

[zhongwuzw]

@Robot2037 Yeah, it's strange, because the keys is valid when remove objects in NSCache, but invalid when empty keys in NSMapTable. So seems it's a weird system bug? I don't know wether we need to fix it, but I have a thought in previous comment to try to fix it. Let's hear other guys opinion. cc @bpoplauschi .

[bpoplauschi]

From what I see, the keys have strong references, where the values are weak.

self.weakCache = [[NSMapTable alloc] initWithKeyOptions:NSPointerFunctionsStrongMemory valueOptions:NSPointerFunctionsWeakMemory capacity:0];

Any chance the UIImage stored in the NSMapTable is deallocated and then sent a message?

[zhongwuzw]

@bpoplauschi Yeah, at first, I suspect the issue happened in weak operation, but after @Robot2037 provide some crash log, I debug refer to the offset of crash line, the crash happened when empty keys. Detailed analyze I already comment here.

[dreampiggy]

Really strange. It seems a Apple implementation bug ? Because we use strong-weak table, the key should follow the strong (retain + 1) semantic behavior, and should not been released before we manual remove that.

Your attempt (or hack ?) may work from the theory and we can have a try. We just do a real copy of the original string for key (I think using +[NSSring stringWithFormat:] is better, mutable may not so safe, many of our API is avoid using NSMutableString...).

[dreampiggy]

@Robot2037 Is this appear for all firmware version ? (From iOS 8+ ~ iOS 12), or only some specify version or jailbroken device ? I want to check if this is the SDK bug or some designed behavior.

[kinarobin]

If the key is TaggedPointer, the key will not follow the strong (retain + 1) semantic behavior, According to https://opensource.apple.com/source/objc4/objc4-723/runtime/NSObject.mm.auto.html

objc_retain(id obj)
{
    if (!obj) return obj;
    if (obj->isTaggedPointer()) return obj;
    return obj->retain();
}

[dreampiggy]

But if it's a tagged pointer, the objc_release will also ignore it. And objc_msgSend will also handle this situation. I'm still confused what this is going on wrong with that map table usage.

__attribute__((aligned(16)))
void 
objc_release(id obj)
{
    if (!obj) return;
    if (obj->isTaggedPointer()) return;
    return obj->release();
}

It's not a 100% reproduable issue, maybe there are some other issue cause this crash ?

[kinarobin]

Yeah, This should not be a problem for TaggedPointer, I try to debug this issue on my iOS8 device for firmware reasons.

[Robot2037]

@dreampiggy Issue was reported on devices running iOS 10 up to iOS 12, both iPhone and iPad. There were no jailbroken devices reported.

[zhongwuzw]

@dreampiggy @bpoplauschi I want to try to fix it like below, what's your guys opinion? 🤔

[self.weakCache setObject:obj forKey:[[key mutableCopy] copy]];

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is still an issue, please make sure it is up to date and if so, add a comment that this is still an issue to keep it open. Thank you for your contributions.

[bpoplauschi]

@zhongwuzw we can try this idea :)

[bpoplauschi]

Should we reopen?