google / promises

Promises is a modern framework that provides a synchronization construct for Swift and Objective-C.
Apache License 2.0
3.79k stars 292 forks source link

SIGABRT due to malloc errors when using Promises from swift? #141

Closed tmm1 closed 4 years ago

tmm1 commented 4 years ago

We're experiencing some strange/confusing crashes and trying to figure out how to investigate further. Have you seen or heard of anything like this previously?

OS Version: tvOS 13.4 (17L256)
Report Version: 104

Exception Type: Unknown (SIGABRT)
Crashed Thread: 29

Application Specific Information:
Channels > Channels(2107,0x16d913000) malloc: *** error for object 0x282ffd2c0: pointer being freed was not allocated

Thread 29 Crashed:
0   libsystem_kernel.dylib          0x35486ed88         __pthread_kill
1   libsystem_pthread.dylib         0x34172d69c         pthread_kill
2   libsystem_c.dylib               0x354e78580         abort
3   libsystem_malloc.dylib          0x366857d30         malloc_vreport
4   libsystem_malloc.dylib          0x366857ee8         malloc_report
5   libsystem_malloc.dylib          0x36684d21c         free
6   CoreFoundation                  0x3427686f0         __rehashs
7   CoreFoundation                  0x3426c0314         -[__NSSetM addObject:]
8   Promises                        0x102e5b394         Promise.then (Promise+Then.swift:101)
ykjchen commented 4 years ago

I personally haven't seen this.

Are you able to provide more context on the triggering code or how to reproduce this?

tmm1 commented 4 years ago

We have not been able to reproduce ourselves, but are receiving many crash reports from users in the wild.

Here's the full backtrace if that helps:

Thread 0 Crashed:
0   libsystem_kernel.dylib          0x3489f1d88         __pthread_kill
1   libsystem_pthread.dylib         0x346ff469c         pthread_kill
2   libsystem_c.dylib               0x35a414580         abort
3   libsystem_malloc.dylib          0x346eb3d30         malloc_vreport
4   libsystem_malloc.dylib          0x346eb3ee8         malloc_report
5   libsystem_malloc.dylib          0x346ea921c         free
6   CoreFoundation                  0x3522ce6f0         __rehashs
7   CoreFoundation                  0x352226314         -[__NSSetM addObject:]
8   Promises                        0x100bea5fc         [inlined] <redacted> (Promise+Any.swift:62)
9   Promises                        0x100bea5fc         [inlined] <redacted>
10  Promises                        0x100bea5fc         [inlined] forEach
11  Promises                        0x100bea5fc         any<T> (Promise+Any.swift:61)
12  LTFramework                     0x100e0237c         firstSuccess<T> (LTEndpointProbe.swift:21)
13  LTFramework                     0x100e02c54         [inlined] ISO6937toUTF8
14  LTFramework                     0x100e02c54         [inlined] ISO6937toUTF8
15  LTFramework                     0x100e02c54         [inlined] ISO6937toUTF8
16  LTFramework                     0x100e02c54         [inlined] map
17  LTFramework                     0x100e02c54         LTEndpointProbe.possibleEndpoint (LTEndpointProbe.swift:66)
18  LTFramework                     0x100e23a80         [inlined] ISO6937toUTF8
19  LTFramework                     0x100e23a80         [inlined] ISO6937toUTF8
20  LTFramework                     0x100e23a80         @callee_guaranteed
21  Promises                        0x100beec34         Promise.then<T> (Promise+Then.swift:39)
22  Promises                        0x100beed94         @callee_guaranteed
23  FBLPromises                     0x100b11a2c         __56-[FBLPromise chainOnQueue:chainedFulfill:chainedReject:]_block_invoke.50 (FBLPromise.m:271)
24  libdispatch.dylib               0x34629a900         _dispatch_call_block_and_release
25  libdispatch.dylib               0x34629bcf4         _dispatch_client_callout
26  libdispatch.dylib               0x3462a7ba4         _dispatch_main_queue_callback_4CF
27  CoreFoundation                  0x3522c5c3c         __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__
28  CoreFoundation                  0x3522c0b10         __CFRunLoopRun
29  CoreFoundation                  0x3522c0128         CFRunLoopRunSpecific
30  Foundation                      0x34a271bb8         -[NSRunLoop(NSRunLoop) runMode:beforeDate:]
31  Foundation                      0x34a2aafcc         -[NSRunLoop(NSRunLoop) run]
32  libxpc.dylib                    0x347b41574         _xpc_objc_main
33  libxpc.dylib                    0x347b43b74         xpc_main
34  Foundation                      0x34a2ad0d0         -[NSXPCListener resume]
35  TVServices                      0x371fb08d0         TVExtensionMain.cold.1
36  TVServices                      0x371f8c5c0         TVExtensionMain
37  libdyld.dylib                   0x348eb9a84         start

Thread 12
0   libsystem_pthread.dylib         0x346ffad10         pthread_getspecific
1   libobjc.A.dylib                 0x365e0fcb4         _objc_fetch_pthread_data
2   libobjc.A.dylib                 0x365e10c8c         fetch_cache
3   libobjc.A.dylib                 0x365e109b0         id2data
4   libobjc.A.dylib                 0x365e1091c         objc_sync_enter
5   FBLPromises                     0x100b10798         +[FBLPromise defaultDispatchQueue] (FBLPromise.m:59)
6   Promises                        0x100bf06e4         OS_dispatch_queue.promises.getter (Promise.swift:138)
7   LTFramework                     0x100e13800         [inlined] always
8   LTFramework                     0x100e13800         LTPlusDiscoveryManager.performDiscovery (LTPlusDiscoveryManager.swift:328)
9   LTFramework                     0x100e12430         LTPlusDiscoveryManager.discover (LTPlusDiscoveryManager.swift:235)
10  Promises                        0x100bed924         Promise.init<T> (Promise+Do.swift:51)
11  Promises                        0x100bed6fc         @callee_guaranteed
12  FBLPromises                     0x100b0bc04         __38+[FBLPromise onQueue:do:]_block_invoke (FBLPromise+Do.m:33)
13  libdispatch.dylib               0x34629a900         _dispatch_call_block_and_release
14  libdispatch.dylib               0x34629bcf4         _dispatch_client_callout
15  libdispatch.dylib               0x3462aac04         _dispatch_root_queue_drain
16  libdispatch.dylib               0x3462ab120         _dispatch_worker_thread2
17  libsystem_pthread.dylib         0x346ff5498         _pthread_wqthread
tmm1 commented 4 years ago

The above backtrace was from an app extension, but we see the same issue in the same codepath inside our main app as well:

OS Version: tvOS 13.4 (17L256)
Report Version: 104

Exception Type: Unknown (SIGABRT)
Crashed Thread: 15

Application Specific Information:
Channels > Channels(1160,0x16fd73000) malloc: *** error for object 0x28314e660: pointer being freed was not allocated

Thread 15 Crashed:
0   libsystem_kernel.dylib          0x31db07d88         __pthread_kill
1   libsystem_pthread.dylib         0x30905869c         pthread_kill
2   libsystem_c.dylib               0x31c63d580         abort
3   libsystem_malloc.dylib          0x31db48d30         malloc_vreport
4   libsystem_malloc.dylib          0x31db48ee8         malloc_report
5   libsystem_malloc.dylib          0x31db3e21c         free
6   CoreFoundation                  0x3277a36f0         __rehashs
7   CoreFoundation                  0x3276fb314         -[__NSSetM addObject:]
8   Promises                        0x10168f394         Promise.then (Promise+Then.swift:101)
9   LTFramework                     0x102426214         [inlined] ISO6937toUTF8
10  LTFramework                     0x102426214         [inlined] ISO6937toUTF8
11  LTFramework                     0x102426214         [inlined] next
12  LTFramework                     0x102426214         firstSuccess<T> (LTEndpointProbe.swift:21)
13  LTFramework                     0x102444d24         [inlined] ISO6937toUTF8 (LTEndpointProbe.swift:60)
14  LTFramework                     0x102444d24         [inlined] ISO6937toUTF8
15  LTFramework                     0x102444d24         [inlined] ISO6937toUTF8
16  LTFramework                     0x102444d24         [inlined] synchronized (NSLocking+Synchronized.swift:15)
17  LTFramework                     0x102444d24         [inlined] possibleEndpointsPromise (LTEndpointProbe.swift:56)
18  LTFramework                     0x102444d24         LTPlusFullDiscoveryRound.discoverKnownUrls (LTPlusFullDiscoveryRound.swift:97)
19  LTFramework                     0x102448858         LTPlusFullDiscoveryRound.discover (LTPlusFullDiscoveryRound.swift:56)
20  LTFramework                     0x102437540         [inlined] discover
21  LTFramework                     0x102437540         [inlined] discoverEndpoint (LTDiscoveryRound.swift:44)
22  LTFramework                     0x102437540         LTPlusDiscoveryManager.performDiscovery (LTPlusDiscoveryManager.swift:300)
23  LTFramework                     0x102436404         LTPlusDiscoveryManager.discover (LTPlusDiscoveryManager.swift:235)
24  Promises                        0x10168d924         Promise.init<T> (Promise+Do.swift:51)
25  Promises                        0x10168d6fc         @callee_guaranteed
26  FBLPromises                     0x10145fc04         __38+[FBLPromise onQueue:do:]_block_invoke (FBLPromise+Do.m:33)
27  libdispatch.dylib               0x3082a2900         _dispatch_call_block_and_release
28  libdispatch.dylib               0x3082a3cf4         _dispatch_client_callout
29  libdispatch.dylib               0x3082a61dc         _dispatch_queue_override_invoke
30  libdispatch.dylib               0x3082b2964         _dispatch_root_queue_drain
31  libdispatch.dylib               0x3082b3120         _dispatch_worker_thread2
32  libsystem_pthread.dylib         0x309059498         _pthread_wqthread

Thread 0
0   libsystem_kernel.dylib          0x31db0787c         __open
1   libsystem_kernel.dylib          0x31daee778         open
2   LTFramework                     0x10245f608         CLSSDKFileLog (CLSInternalLogging.c:29)
3   LTFramework                     0x10246a39c         CLSFileInitWithPathMode
4   LTFramework                     0x102470e98         __CLSUserLoggingWriteAndCheckABFiles_block_invoke (CLSUserLogging.m:482)
5   libdispatch.dylib               0x3082a3cf4         _dispatch_client_callout
6   libdispatch.dylib               0x3082b05e8         _dispatch_lane_barrier_sync_invoke_and_complete
7   LTFramework                     0x1024707f8         CLSUserLoggingWriteAndCheckABFiles (CLSUserLogging.m:479)
8   LTFramework                     0x102470cc0         CLSLogInternal (CLSUserLogging.m:541)
9   LTFramework                     0x102470d70         CLSLog (CLSUserLogging.m:370)
10  LTFramework                     0x1023cce04         [inlined] _dispatch_once (once.h:83)
11  LTFramework                     0x1023cce04         LTLogIt (LTLog.m:25)
12  LTFramework                     0x1023cd040         LTLog (LTLog.m:55)
13  LTFramework                     0x1023e59f8         -[LTHTTPClient asyncRequestResponseWithRequest:method:params:] (LTHTTPClient.m:152)
14  LTFramework                     0x10244d7a4         LTPlusHTTPClient.asyncRequestResponse (LTPlusHTTPClient.swift:27)
15  LTFramework                     0x10242d910         [inlined] asyncRequestResponse
16  LTFramework                     0x10242d910         LTPlusDiscoveryClient.discoverEndpoint (LTPlusDiscoveryClient.swift:18)
17  Promises                        0x10168d924         Promise.init<T> (Promise+Do.swift:51)
18  Promises                        0x10168d6fc         @callee_guaranteed
19  FBLPromises                     0x10145fc04         __38+[FBLPromise onQueue:do:]_block_invoke (FBLPromise+Do.m:33)
20  libdispatch.dylib               0x3082a2900         _dispatch_call_block_and_release
21  libdispatch.dylib               0x3082a3cf4         _dispatch_client_callout
22  libdispatch.dylib               0x3082afba4         _dispatch_main_queue_callback_4CF
23  CoreFoundation                  0x32779ac3c         __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__
24  CoreFoundation                  0x327795b10         __CFRunLoopRun
25  CoreFoundation                  0x327795128         CFRunLoopRunSpecific
26  GraphicsServices                0x32cae9388         GSEventRunModal
27  UIKitCore                       0x344857da8         UIApplicationMain
28  Channels                        0x2010dabc8         main (main.m:14)
29  libdyld.dylib                   0x308920a84         start

(Note that some of the inlined frames are not being symbolized correctly).

eric commented 4 years ago

This has been in most or all of the stack traces. Is there anything in this code that would be not properly capturing all of the strong references that it should?

func firstSuccess<T>(_ promises: [Promise<T>]) -> Promise<T> {
  let promise = Promise<T>.pending()

  if promises.count == 0 {
    promise.reject(Error.nothingFound)
    return promise
  }

  // Return the first successful promise
  for p in promises {
    p.then { promise.fulfill($0) }
  }

  // If no promises are successful, wait until they have all failed
  // and return the first error.
  any(promises).catch { promise.reject($0) }

  return promise
}
shoumikhin commented 4 years ago

Hi guys, sorry for the late reply.

It's hard to tell what's going on w/o the surrounding context. Looks like NSMutableSet tries to add an already deallocated object? Not sure how exactly that's possible by looking at the Promise+Then.swift and Promise+Any.swift. Likely, some race condition is involved, which is hard to detect w/o the context.

I can see the race operator doesn't exactly fit your needs, because you're only interested in the success outcome. Perhaps, you may like to add another operator named first which would be similar to race, but resolve with the first success instead and ignore all failures until the last one? I believe it can be just a slight modification to the existing race, which should threat the Swift <-> ObjC interoperability correctly.

Another simpler way you may consider is to avoid pending promise, but use async instead, i.e. essentially do the same as you have in firstSuccess, just call the fulfill and reject closures passed into the work block.

Thanks.

eric commented 4 years ago

Thanks for the info.

As I was looking into this more, I realized that the promise operations aren't actually thread-safe themselves.

For instance, the line we're hitting with then:

    objCPromise.__pendingObjects?.add(promise)

...is calling [NSMutableSet addObject:], which from everything I have read, is not thread safe.

As I've reviewed all of our stack traces, it appears that most of them are in then or any calls that are trying to modify this set.

Should there be synchronization on this?