Closed Parkyprg closed 7 years ago
Nope, nothing to do at all.
In fact in the unit tests there are lots of examples of up to 50 threads all reading/writing/deleting at the same time.
There is a critical section around a write op to a database, but these will not block reads. And transactions are partitioned by thread as well, so unless there is a huge op within a large transaction, you should not see, much in the way of blocking.
Any additional information you could provide would be helpful. Is there anything special about what you are persisting? e.g. large BLOB objects, or very wide objects?
It seems I am not able to find a reason, but I suspect is somehow related to the framework (at least maybe to the way I am using it).
As a background - I am sending a few server requests from different threads to fetch some data (using AFNetworking). When the results come, I insert/update them in the database with SharkORM.
The problem is that the app simply freeze (like entering in an endless loop) - but not always at the same step and what is most frustrating - it happens a few times per each 100 runs (or so). If a pause it from xCode, I see different threads, stuck at some SharkORM methods. For example now I have the following: Thread 1 - freezed at SRKQuery whereWithFormat - when it sets "self.whereClause = format;". In the steps I see this object has been initialized from a removeAll method of SRKResultSet. Thread 4 - freezed at SRLObject setPropertyBoolIMP - when it sets a boolean value on a field.
I will try to note down where it happens each time but unfortunately I don't know how I can reproduce it properly.
Any thoughts are highly appreciated.
PS - it's not like a temporary block - it freezes forever, no matter what I do. I am storing only simple values - like NSNumber, NSString, BOOL, double and a few relations between tables.
Is it possible that I could enter in a deadlock or something (like referentiating A from B and also B from A)?
I don't think there is a deadlock - because in that case it would freeze all the time at the same place/moment.
Hi Sorry for not getting back to you yesterday, I would agree. The deadlocks always show themselves waiting for a semaphore, and we only have the one critical section as far as I can remember around write operations.
You have done quite a lot of investigation already. And we also use AFNetworking in the same kind of way. We modified AFNetworking to always use background threads and not all crashing in on the main thread, but that would be the only difference.
Are you seeing any errors on the delegate? Locked SQL file, index corruption, anything like that?
Also, i'd never expect a freeze on your example of thread 4. I'd be looking at what priority those dispatch items were, and if they got suspended by GCD for higher priority blocks? But i'm really scraping the barrel for ideas there.
Thank you for your reply. I managed to get it reproduced all the time now. This should make things easier (in theory :)). I have 2 relevant threads, always the same ones: Thread 1: Stuck at [SRKQuery setOrderBy] Thread 5: Stuck at setting a BOOL property. Runs in a concurrent queue.
I am attaching 2 images with what I think it might be relevant.
Thanks for the info, it's very weird. As the "setPropertyIMP", is only on a single object, and shares nothing with other objects or the ORM. There is quite literally no reason why it would ever block there.
What created the block and dispatched it? Was it an AFNetworking response block? Could you try dispatching one of those blocks on a different queue:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{ // loop through doing your update here. });
I appreciate that this is normally a nightmare to try and break your app apart to make a bit of it async. But might I suggest using an NSLock to hold a semaphore until the dispatched block has completed.
I'm just interested to see if you break out of this dispatch queue, whether or not it starts to work.
It might give us at least a direction to head in.
P.S. does this happen on an entirely repeatable basis, or is it intermittent?
Hy @Parkyprg,
Is there anything specific you would like me to look at, or a scenario I could setup that you think would be the best possible chance to replicate your issue? I have a couple of free hours today that can be dedicated to trying to work out this issue.
Thanks Adrian
Hi Adrian,
I managed to reproduce the problem in a separate project. Can you tell me an email address where I can send you the source files?
Thank you, Cristi
Hi,
devs@db-access.org.
Looking forward to fixing it!
Sent from my iPhone
On 12 Sep 2016, at 05:37, Parkyprg notifications@github.com wrote:
Hi Adrian,
I managed to reproduce the problem in a separate project. Can you tell me an email address where I can send you the source files?
Thank you, Cristi
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
Email sent. Forgot to mention it here :)
I was wondering if you were able to reproduce the problem with the test project that I sent.
Just looking now, but so far unable to reproduce. Does this happen on devices or the simulator, or both?
On 14 Sep 2016, at 09:46, Parkyprg notifications@github.com wrote:
I was wondering if you were able to reproduce the problem with the test project that I sent.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/sharksync/sharkorm/issues/30#issuecomment-246946492, or mute the thread https://github.com/notifications/unsubscribe-auth/ADUZ3u34rm9Ezue5Laoy172CjD0cFcNbks5qp7SAgaJpZM4JqlVI.
Mostly on device. Try to add many contacts, so it can spend more time on that particular job.
Hi, i'm just trying to get it on a phone, but in the simulator we added 1000 contacts to really stretch it.
On simulator I believe the time spent processing the contacts is too short. Try on a device to add some random contacts with the second project that I sent you.
Okay, I will try that when our testing team get here. I'd prefer not to create loads of addresses on my personal phone. As actually, the app just crashes on iOS 10, as it tries to access the contacts before it asks for permissions. I'm just going to add it in manually now.
UPDATE: The permission for Contacts is not present in iOS 10 (it crashes before it adds the entitlement I think), so I can't manually add it. I'll wait for an iOS 9 device. Will get back to you soon.
Right, i've managed to re-create the hang (it took a lot of trying), however, in this particular case it was not utilising shark at all, but had been blocked by two threads both creating a runloop for the contacts API at the same time. I will continue looking to see if I can't pin it down further.
Also, what device/iOS version are you using? I'll try and match that this end if I can for a better and more accurate test. I'm currently using an iPhone 6 running iOS 9. 1000 contacts.
I am using the same device - 6 with iOS 9.3.5
Hi, I'm going to look at this again today. Is there any further information/investigation for this issue?
Thank you for looking into this. However, there are no new information. It still freezes.
I am not 100% sure this is caused by the framework but every time it happens - 2 threads are locked on SharkORM processes.
Hi,
Were you able to find anything? At least did you managed to reproduce it on a test device?
I've only been testing for about 1/2 an hour, but so far it has not locked up once. When I did manage to get it to freeze last time it happened to not be doing anything in Shark. Another time, it froze whilst on the line.
idx++;
Which, really just has to be symptomatic of the co-ordinating thread locking. Knowledge of how iOS/OSx manages it's thread hierarchy is not known to me, but I would have thought there will be a base level runloop (thread) that controls all the others, but I would need to ask someone on SO who may be more knowledgeable.
Shark, as a general rule does nothing in any background threads, it merely executes it's code in-line with the developers intentions. We do use performSelectorInBackground: if you're performing async queries, but these are the specific differences.
So, although unable so far to identify the culprit, i'm struggling to find any reason to believe it is the ORM. Although I do maintain a completely open mind.
I will do some more testing over the next hour or so. Possibly on some older / single core devices.
Thanks Adrian
Any update for this issue? i also get freeze sometimes at following line SRKObject class
this is called from main thread
Hi,
I looked again recently at this, but in the other instance, the code would stop in the middle of nowhere and in places that could not possibly ever pause. Yours however might be different.
So, a few questions. Does your hang ever continue, or is a transient pause?
Are there other DB operations happening in the background? (they would have to be huge to cause any sort of delay).
Is this call normally within any kind of transaction?
But I will look at that method now to see if there is anything I can spot.
it pause at this line every time, we can see the thread is wait for some thing from the stack trace. it is not a transient pause. i think there is not any background access, since it occur when our app just launch
Is this swift code by any chance? Does it make it to the call to description? Because I think i've just spotted a while that should be an if. I don't suppose you happen to have a screenshot of the trace do you?
my app is built with swift. it is not frequent. i will generate a screenshot when it occur next time
Excellent, I look forward to seeing it. Hopefully, there will be something to go on.
the snapshot
Thank you for the screenshot, i've been investigating the stop point, although on the face of it that call should hardly cause any kind of issue. So i've been looking around and trying to see how other have gone about debugging this particular condition (seems to happen quite a bit with Unity and couple of other big frameworks).
There seems to be something in common with all of them, and that is on the other threads (and mostly network calls) there is a block dispatched which is executed on the main queue, from a background thread. This is the same as the previous poster too, this has the effect of locking that main thread for some reason.
There is some talk about it being when the blocks inadvertently reference a static variable and there could be an issue when that is pushed onto the stack for that specific block.
If you could get it to pause again, would you mind sending the output from a 'bt all' at the debugger, that may prove useful to try and track this down further.
I will also look to see if there are any SQLite issues open where thread calls can destabilize parent thread locking, or anything of that vein.
thread #1: tid = 0x382170, 0x000000010a1d2db6 libsystem_kernel.dylib__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP frame #0: 0x000000010a1d2db6 libsystem_kernel.dylib
__psynch_cvwait + 10
frame #1: 0x000000010a198728 libsystem_pthread.dylib_pthread_cond_wait + 767 frame #2: 0x0000000104e48eed libobjc.A.dylib
monitor_tt_class_initialize + 512 frame #4: 0x0000000104e4ecc5 libobjc.A.dylib
lookUpImpOrForward + 176
frame #5: 0x0000000104e5d8bb libobjc.A.dylibobjc_msgSend + 187 frame #6: 0x0000000104e558b5 libobjc.A.dylib
objc_getProperty_non_gc(objcobject, objcselector, long, bool) + 68
frame #7: 0x00000001035bcf18 EAMInventory-[SRKObject init](self=0x00007feeb1496870, _cmd="init") + 2792 at SRKObject.m:1705 frame #8: 0x00000001036ef081 EAMInventory
SerializableDataEntity.init() -> SerializableDataEntity + 49 at SerializableDataEntity.swift:0
frame #9: 0x00000001037c1da8 EAMInventorySyncEntity.init() -> SyncEntity + 40 at SyncEntity.swift:0 frame #10: 0x00000001037c28d7 EAMInventory
WritableSyncEntity.init() -> WritableSyncEntity + 55 at SyncEntity.swift:0
frame #11: 0x0000000103a5353c EAMInventoryDepartment.init() -> Department + 188 at Department.swift:0 frame #12: 0x0000000103a5357c EAMInventory
Department.__allocating_init() -> Department + 44 at Department.swift:0
MainViewController.loadFilterContentView(self=0x00007feeb2001200) -> () + 4533 at MainViewController.swift:651 frame #14: 0x0000000103955f38 EAMInventory
MainViewController.viewDidFirstAppear(self=0x00007feeb2001200) -> () + 104 at MainViewController.swift:125
frame #15: 0x000000010384ecad EAMInventoryBaseViewController.viewDidAppear(animated=false, self=0x00007feeb2001200) -> () + 173 at BaseViewController.swift:30 frame #16: 0x000000010384ecf1 EAMInventory
@objc BaseViewController.viewDidAppear(Bool) -> () + 49 at BaseViewController.swift:0
frame #17: 0x0000000107055335 UIKit-[UIViewController _setViewAppearState:isAnimating:] + 830 frame #18: 0x000000010708e589 UIKit
-[UINavigationController viewDidAppear:] + 207
frame #19: 0x0000000107055335 UIKit-[UIViewController _setViewAppearState:isAnimating:] + 830 frame #20: 0x0000000107058112 UIKit
64-[UIViewController viewDidMoveToWindow:shouldAppearOrDisappear:]_block_invoke + 42
frame #21: 0x00000001070563be UIKit-[UIViewController _executeAfterAppearanceBlock] + 86 frame #22: 0x0000000106ec109e UIKit
_runAfterCACommitDeferredBlocks + 633
frame #23: 0x0000000106ed4e4c UIKit_cleanUpAfterCAFlushAndRunDeferredBlocks + 95 frame #24: 0x00000001056372ec CoreFoundation
__CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK + 12
frame #25: 0x000000010562cf75 CoreFoundation__CFRunLoopDoBlocks + 341 frame #26: 0x000000010562c6d2 CoreFoundation
__CFRunLoopRun + 850
frame #27: 0x000000010562c0f8 CoreFoundationCFRunLoopRunSpecific + 488 frame #28: 0x000000010d050ad2 GraphicsServices
GSEventRunModal + 161
frame #29: 0x0000000106eb4f09 UIKitUIApplicationMain + 171 frame #30: 0x0000000103aae352 EAMInventory
main + 114 at AppDelegate.swift:13
frame #31: 0x0000000109e7892d libdyld.dylibstart + 1 frame #32: 0x0000000109e7892d libdyld.dylib
start + 1thread #2: tid = 0x3822f1, 0x000000010a1d3ee2 libsystem_kernel.dylibkevent64 + 10, queue = 'com.apple.libdispatch-manager' frame #0: 0x000000010a1d3ee2 libsystem_kernel.dylib
kevent64 + 10
frame #1: 0x0000000109e32408 libdispatch.dylib_dispatch_mgr_invoke + 260 frame #2: 0x0000000109e321a2 libdispatch.dylib
_dispatch_mgr_thread + 54
thread #3: tid = 0x3822f4, 0x000000010a1cd06e libsystem_kernel.dylibsyscall_thread_switch + 10, queue = 'com.demo.eam.databaseAccess' frame #0: 0x000000010a1cd06e libsystem_kernel.dylib
syscall_thread_switch + 10
frame #1: 0x000000010a1ad1cd libsystem_platform.dylib_os_lock_handoff_lock_slow + 111 frame #2: 0x0000000104e558ad libobjc.A.dylib
objc_getProperty_non_gc(objcobject, objcselector, long, bool) + 60
frame #3: 0x00000001036b93e6 EAMInventory+[SharkORM addEntityRelationship:inDatabase:](self=SharkORM, _cmd="addEntityRelationship:inDatabase:", r=0x00007feeb1427140, dbName=@"EAM") + 262 at SharkORM.m:999 frame #4: 0x00000001035b6821 EAMInventory
+[SRKObject setupClass](self=EAMInventory.ProjectAttribute, _cmd="setupClass") + 6129 at SRKObject.m:1598
frame #5: 0x00000001035bc110 EAMInventory+[SRKObject initialize](self=EAMInventory.ProjectAttribute, _cmd="initialize") + 32 at SRKObject.m:1636 frame #6: 0x0000000104e48bff libobjc.A.dylib
_class_initialize + 679
frame #7: 0x0000000104e4ecc5 libobjc.A.dyliblookUpImpOrForward + 176 frame #8: 0x0000000104e5d8bb libobjc.A.dylib
objc_msgSend + 187
frame #9: 0x000000010391ea8c EAMInventorytype metadata accessor for ProjectAttribute + 44 at ProjectAttribute.swift:0 frame #10: 0x000000010392b99b EAMInventory
Project.init() -> Project + 443 at Project.swift:30
frame #11: 0x000000010392bac1 EAMInventory@objc Project.init() -> Project + 17 at Project.swift:0 frame #12: 0x00000001036c7108 EAMInventory
__35-[SharkORM fetchEntitySetForQuery:]_block_invoke(.block_descriptor=-[SharkORM performQuery:rowBlock:](self=0x00007feeb2a019b0, _cmd="performQuery:rowBlock:", query=0x00007feeb2a01b70, rowBlock=0x00007000000934a0) + 12110 at SharkORM.m:2184 frame #14: 0x00000001036c6e56 EAMInventory
-[SharkORM fetchEntitySetForQuery:](self=0x00007feeb2a019b0, _cmd="fetchEntitySetForQuery:", query=0x00007feeb2a01b70) + 198 at SharkORM.m:2276
frame #15: 0x00000001035abb74 EAMInventory-[SRKQuery fetch](self=0x00007feeb2a01b70, _cmd="fetch") + 692 at SRKQuery.m:436 frame #16: 0x0000000103a5e9d9 EAMInventory
ProjectDal.(self=0x00007feeb17a5440, observer=(action = 0x00000001046afc30 ReactiveCocoapartial apply forwarder for ReactiveCocoa.SignalProducer.(startWithSignal ((ReactiveCocoa.Signal<A, B>, ReactiveCocoa.Disposable) -> ()) -> ()).(closure #2) at SignalProducer.swift)) -> SignalProducer<Project?, EAMError>).(closure #1).(closure #1) + 921 at ProjectDal.swift:190 frame #17: 0x0000000103906eee EAMInventory
BaseDal.(self=0x00007feeb17a5440, block=0x0000000103a5af60 EAMInventorypartial apply forwarder for EAMInventory.ProjectDal.(getCurrentProject () -> ReactiveCocoa.SignalProducer<Swift.Optional<EAMInventory.Project>, EAMInventory.EAMError>).(closure #1).(closure #1) at ProjectDal.swift) -> ()) -> ()).(closure #1) + 110 at BaseDal.swift:33 frame #18: 0x00000001036df4b7 EAMInventory
thunk + 39 at PhotosEditViewController.swift:0
frame #19: 0x0000000109e22d9d libdispatch.dylib_dispatch_call_block_and_release + 12 frame #20: 0x0000000109e433eb libdispatch.dylib
_dispatch_client_callout + 8
frame #21: 0x0000000109e2982c libdispatch.dylib_dispatch_queue_drain + 2215 frame #22: 0x0000000109e28d4d libdispatch.dylib
_dispatch_queue_invoke + 601
frame #23: 0x0000000109e2b996 libdispatch.dylib_dispatch_root_queue_drain + 1420 frame #24: 0x0000000109e2b405 libdispatch.dylib
_dispatch_worker_thread3 + 111
frame #25: 0x000000010a1974de libsystem_pthread.dylib_pthread_wqthread + 1129 frame #26: 0x000000010a195341 libsystem_pthread.dylib
start_wqthread + 13
thread #8: tid = 0x38230a, 0x000000010a1ccf72 libsystem_kernel.dylibmach_msg_trap + 10, name = 'com.apple.NSURLConnectionLoader' frame #0: 0x000000010a1ccf72 libsystem_kernel.dylib
mach_msg_trap + 10
frame #1: 0x000000010a1cc3b3 libsystem_kernel.dylibmach_msg + 55 frame #2: 0x000000010562d434 CoreFoundation
CFRunLoopServiceMachPort + 212
frame #3: 0x000000010562c88f CoreFoundation__CFRunLoopRun + 1295 frame #4: 0x000000010562c0f8 CoreFoundation
CFRunLoopRunSpecific + 488
frame #5: 0x000000010978ad24 CFNetwork+[NSURLConnection(Loader) _resourceLoadLoop:] + 412 frame #6: 0x0000000104acf12b Foundation
NSThreadstart + 1198
frame #7: 0x000000010a19799d libsystem_pthread.dylib_pthread_body + 131 frame #8: 0x000000010a19791a libsystem_pthread.dylib
_pthread_start + 168
frame #9: 0x000000010a195351 libsystem_pthread.dylib`thread_start + 13
thread #9: tid = 0x382326, 0x000000010a1d307a libsystem_kernel.dylib__select + 10, name = 'com.apple.CFSocket.private' frame #0: 0x000000010a1d307a libsystem_kernel.dylib
select + 10
frame #1: 0x00000001056725ea CoreFoundation`CFSocketManager + 746
frame #2: 0x000000010a19799d libsystem_pthread.dylib_pthread_body + 131 frame #3: 0x000000010a19791a libsystem_pthread.dylib
_pthread_start + 168
frame #4: 0x000000010a195351 libsystem_pthread.dylib`thread_start + 13
happen again
Thanks, I think I may have found the bug from the information you provided. I will check it out now and post a fix. By rights it should just crash, but it doesn't. It just hangs instead.
so, it seem two thread access systemEntityRelationships at the same time?
yep, when a static is accessed it seems to block the main thread. I'm just removing the statics and putting them into a class. Those items are a bit of a throwback to some original code when the framework was just c & c++
Okay, to do this right is going to take some time. ETA on a fix is tomorrow by the looks of it, subject to passing all the tests.
Thank you Adrian. I will try this fix too. Can you release it on pods too?
Just done it v2.0.9 should be showing, thanks @Parkyprg for being so patient. It was only when the second report came in that I could even see the pattern.
It may not fix it (considering I really struggled to replicate it), but i'd be really surprised if it didn't :)
:) Thank you.
Is this looking good now? It has caused a performance drop in the ORM, but that will be dealt with separately.
i just get the latest code. look good for now. if any issue found, we will let you know. by the way, i have made some changes on my local branch(some changes about printing sql log base on setting, KVO related bug fix and some additional convenient methods), can i send a pull request?
Yes please do.
i think your change may not fix the problem, the problem is two thread access/modify the systemEntityRelationships list, but your change is wrap the getting logic of the member and add lock in the getting method. it won't avoid the access/modification to the list from two threads after they get the list object with a thread safe wrapper.
The problem is still existing. i run bt all when it occur, get the same result as before, not sure if the problem is access [Class description] in different thread.
i guess the problem is on code [r.sourceClass description]
Those calls to get the systemEntityRelationships create a copy of that dictionary, so that should make it thread-safe (from an iteration point of view), now there might be two things accessing the objects within the dictionary which will be the same, but, as they are being used RO there should never have been a problem accessing them as a single static either. I only wrapped a lock around them to ensure there were not two copies being made at the same time. I strongly believe this to be a problem with apple's implementation of their block dispatcher, which seems to lock threads when creating/copying stack objects.
I had other plans for how I was going to deal with all these statics, event registry, backing store & re-implement the caching that would largely do away with large swathes of this code, but I was unsure as to weather it will actually be faster (more performant) in the long run. It would enable good memory savings and some CPU gains on some operations, but retrieving a properties value may well become slower, which would be a more consistent performance concern.
As for description, that is simple string manipulation and work on the heap. And there is a heap per thread anyway, so this should never clash.
Anyhow, I will look again at this. Because those changes made the ORM much slower, so if it is not the static objects, then they should go back or be re-implemented differently as this is one scenario where OO is the worst possible solution for this.
In one of my applications I am accessing/updating db objects from multiple threads and for some reason it simply freeze. I am trying to narrow it down, but being multi-threaded, seems to be difficult.
Are there any special steps to make, in order to work with the objects from multiple threads?